SharePoint – Unable to create any site collection

The Issue:

We are unable to create any site collections. We see error "This page has encountered a critical error. Contact your system administrator if this problem persists".

 

What we see in the logs:

 

• Site collection creation was failing because of the below stack as per ULS logs:

 

[DATE] 09:36:39.25 w3wp.exe (PID) TID Document Management Server Document Sets 55r4 Unexpected

DocumentSet FeatureActivating: throws exception: This page has encountered a critical error. Contact your system administrator if this problem persists.. Stacktrace:
at Microsoft.SharePoint.ApplicationRuntime.SafeControls.IsSafeControl(Boolean isAppWeb, Type type, String& unsafeErrorMessage)
at Microsoft.SharePoint.WebPartPages.TypeCache.CheckIfSafeControl(Type type, Boolean isAppWeb)
at Microsoft.SharePoint.WebPartPages.TypeCache.get_Item(Type type, Boolean isAppWeb, Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.WebPart.WriteXmlGlobal(Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags, BinaryWebPartSerializerWriter writer)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartToStore(WebPart webPart, Int32 viewId, String viewGuid)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset, Boolean throwIfLocked)
at Microsoft.SharePoint.WebPartPages.SPLimitedWebPartManager.AddWebPart(WebPart webPart, String zoneId, Int32 zoneIndex)
at Microsoft.Office.DocumentManagement.Internal.DocumentSetFeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties)
at Microsoft.SharePoint.SPFeature.DoActivationCallout(Boolean fActivate, Boolean fForce)
at Microsoft.SharePoint.SPFeature.Activate(SPSite siteParent, SPWeb webParent, SPFeaturePropertyCollection props, SPFeatureActivateFlags activateFlags, Boolean fForce)
at Microsoft.SharePoint.SPFeatureCollection.AddInternal(SPFeatureDefinition featdef, Version version, SPFeaturePropertyCollection properties, SPFeatureActivateFlags activateFlags, Boolean force, Boolean fMarkOnly)
at Microsoft.SharePoint.SPFeatureCollection.AddInternalWithName(Guid featureId, Int32 compatibilityLevel, String featureName, Version version, SPFeaturePropertyCollection properties, SPFeatureActivateFlags activateFlags, Boolean force, Boolean fMarkOnly, SPFeatureDefinitionScope featdefScope)
at Microsoft.SharePoint.SPFeatureCollection.AddInternal(Guid featureId, Version version, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly, SPFeatureDefinitionScope featdefScope)
at Microsoft.Office.DocumentManagement.VideoAndRichMedia.Internal.VideoAndRichMediaFeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties)
at Microsoft.SharePoint.SPFeature.DoActivationCallout(Boolean fActivate, Boolean fForce)
at Microsoft.SharePoint.SPFeature.Activate(SPSite siteParent, SPWeb webParent, SPFeaturePropertyCollection props, SPFeatureActivateFlags activateFlags, Boolean fForce)
at Microsoft.SharePoint.SPFeatureCollection.AddInternal(SPFeatureDefinition featdef, Version version, SPFeaturePropertyCollection properties, SPFeatureActivateFlags activateFlags, Boolean force, Boolean fMarkOnly)
at Microsoft.SharePoint.SPTemplateAssociationElement.EnsureTemplateAssociatedSiteFeaturesActivated(SPSite site, String sTemplateName)
at Microsoft.SharePoint.SPFeatureManager.<>c__DisplayClass16.b__15()
at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken, Boolean bResetContext, WaitCallback code, Object param)
at Microsoft.SharePoint.SPFeatureManager.EnsureTemplateAssociatedSiteFeaturesActivated(Byte[]& userToken, Guid& tranLockerId, Int32 nZone, Guid databaseid, Guid siteid, String sTemplateName)
at Microsoft.SharePoint.ApplicationRuntime.SafeControls.IsSafeControl(Boolean isAppWeb, Type type, String& unsafeErrorMessage)
at Microsoft.SharePoint.WebPartPages.TypeCache.CheckIfSafeControl(Type type, Boolean isAppWeb)
at Microsoft.SharePoint.WebPartPages.TypeCache.get_Item(Type type, Boolean isAppWeb, Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.WebPart.WriteXmlGlobal(Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags, BinaryWebPartSerializerWriter writer)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartToStore(WebPart webPart, Int32 viewId, String viewGuid)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset, Boolean throwIfLocked)
at Microsoft.SharePoint.WebPartPages.SPLimitedWebPartManager.AddWebPart(WebPart webPart, String zoneId, Int32 zoneIndex)
at Microsoft.Office.DocumentManagement.Internal.DocumentSetFeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties).

 

• We can see that the failure was while activating the feature called "DocumentSet". In this process, it attempts to check if the feature is a SafeControl – for which it would need to look at the web.config file of the web application in IIS.

This is when we found that the affected web applications had no files in the IIS virtual directory on the server. They were missing, which was causing the unexpected error at site collection creation.

 

 

How did we fix this?

 

• We provisioned these web applications to bring back these files:

 

    Add-pssnapin microsoft.sharepoint.powershell
    $contentservice = [Microsoft.sharepoint.Administration.spwebservice]::contentservice
    $webapp = $contentservice.webapplications | where { $_.name -match "WEB_APP_NAME" }
    $webapp.provision()

 

• After this, we were able to create site collections on the re-provisioned web application. We tested creating a new web application, but that was timing out. So we did this on the central admin application pool in IIS:

 

IIS Manager > App Pool > Central Admin App Pool > Shut Down Limit > Changed this from 90 seconds to 600 Seconds

 

• After this we were able to create web applications from central admin site successfully.
• We also noticed that web application creation was taking unusually long to complete. One tip to improve the performance would be to disable Receive Side Scaling and TCP Offloads on the SharePoint and SQL Servers. This is explained in the article below:

 

Information about the TCP Chimney Offload, Receive Side Scaling - https://support.microsoft.com/kb/951037

 

• It is likely that a timed out web application create command is what caused this issue in the first place.
• You would need to disable the settings both at the NIC Card level as well as from command prompt on both SharePoint and SQL servers.
• Please be aware that this would cause a temporary network outage on the servers, so it is best to schedule this for a downtime in the environment.