HTTP error 503 error occurred
The Cause
IIS shows that the following SharePoint application pools were stopped:
- Sharepoint - 80
- SecurityTokenServiceApplicationPool
- SharePoint Central Administration v4
The application pools could be restarted without any issues but when attempting to open Central Administration or a site collection again, the same HTTP error 503 error occurred and the associated application pool and SecurityTokenServiceApplicationPool had once again stopped.
IIS event logs contained numerous occurrences of the three events shown below, all associated to the Windows Process Activation Service (WAS):
The warning with the Event ID 5021 highlighted the issue which was confirmed to be associated to the service accounts defined as the application pool identities NOT having Batch Logon Rights on the Application/WFE server. (Web Front End)
Batch Logon Rights
As part of the initial Farm configuration process, the service accounts associated to various SharePoint application pools (including those listed above) are added to the IIS_IUSRS local group on the Application/WFE server. By default, IIS 7.0 allows the local IIS_IUSRS group batch logon rights through local policy which therefore provides the service accounts defined as the application pool identities the required permissions on the Application/WFE server through membership of the group.
Having looked at the local policy ‘ Log on as a batch job ’ through Server Manager (Administrative Tools > Local Security Policy > Security Settings > Local Policies > User Rights Assignment), I was able to verify that the local IIS_IUSRS group was not listed in the security settings for the policy.
What Happened?
At some point between the Application/WFE server being added to the domain and the SharePoint 2010 installation completing, a domain group policy had taken effect which overwrote the permissions applied by the SharePoint configuration process and removed the IIS_USERS local group from the local policy; ‘Log on as a batch job’. In talking to my client’s IT team, it transpired that the domain group policy had been created to enforce permissions on the local policy to satisfy a requirement of the backup software they’re using.
Why Log on as a batch job is required?
For tasks to be run by the Task Scheduler, Windows requires that the account have "Log on as a batch job" permissions. These permissions are automatically assigned. But our domain group policy had taken effect which overwrote the permissions applied by the SharePoint configuration process and removed the IIS_USERS local group from the local policy.
The Resolution
To resolve the issue, either:
- Add the service accounts defined as the application pool identities to the domain group policy enforced on the Application/WFE server or…
- Create a new policy that overrides the domain policy or…
- Remove the policy in question and give the IIS_IUSRS local group permissions back via local policy
In this case, or we can create a new domain group policy that applied to the Application/WFE server giving the service accounts permissions to the ‘Log on as a batch job’