Kerberos authentication not working after IIS/SharePoint rebuild.

When you test, does the APM log show a log that says "S4U ===> OK!"?

If it does, then that indeed indicates that APM is getting a Kerberos ticket and sending it to the web server. Otherwise you'd also see an Authorization header in the HTTP request sent to SharePoint from APM.

So when a ticket is sent but is rejected with a 401 response, that generally indicates that you simply requested a ticket for the wrong service. In your case APM was likely configured to derive the SPN of the SharePoint service in one of two ways: reverse DNS lookup or static SPN Pattern. So let's say you're using the first method:

• APM takes the selected pool member (1.2.3.4) and does a reverse DNS lookup into the domain to get the hostname (sp1.domain.com)
• APM takes that result and derives a SPN (http/sp1.domain.com) and then issues a Kerberos ticket request to the KDC for this SPN
• APM receives the ticket for this SPN, encodes it and passes it in the Authorization header of the HTTP request to the server

If by some chance the SharePoint server (service) is no longer defined by the SPN, then the server will naturally reject the ticket and send back a 401. So then the first thing you should do is validate what service account the SharePoint server is running under, and what SPN(s) is/are defined for that account.

The static SPN pattern just short circuits the regular reverse DNS lookup process. SharePoint (and IIS) control individual websites with application pools. These are thread and resource management entities, but they also control the identity under which a given service functions. Out of the box and IIS service will run as ApplicationPoolIdentity, which is a service owned by the local machine, therefore the SPN for that service would naturally be http/[local server machine name]. That's why it makes sense to do the reverse DNS lookup with APM. Assuming this is how the server's are configured, the returned PTR record will be the server's machine name, which would be the same as the SPN. But there are definitely times when you don't want to do it this way. Most SharePoint articles recommend creating a separate AD account and making that the owner of the frontend services. So you create that account and then you create an arbitrary servicePrincipalName attribute for that account. And then you can assign that same account as the owner of SharePoint on all of the servers, and you have an entire farm under a single SPN. That's specifically why you'd use the SPN Pattern option in the APM Kerberos SSO profile - to indicate that arbitrary SPN that will be used by all of the servers. What I'm guessing has happened is that you rebuilt the SharePoint servers but did not assign a separate AD account to own the frontend services. The fact that it works without the SPN Pattern now implies that the SPNs are all now machine-based and unique per server.

As for logging, that's a good question. You get the "S4U ===> OK!" message in the APM log because APM was still able to get a ticket for that old SPN. And since APM sent a ticket and SharePoint rejected it, the fault lies almost entirely in the SharePoint config. Had you captured the Kerberos traffic between APM and the KDC with WireShark, you would have seen the requested (incorrect) SPN in APM's request.

What exactly is meant by "owning" a service? Is that being the identity of the application pool?

That's exactly what it is. It's the identity associated with the application pool. That identity can be the local system (ApplicationPoolIdentity, LocalSystem, etc.) or it can be an actual AD account that you create. If you create an account to use as the app pool identity, you need to of course create an arbitrary SPN for that account, and then specify that SPN as the SPN Pattern in the APM Kerberos SSO. This actually creates an interesting artifact. When you do Kerberos with a browser, the browser derives the SPN from the FQDN in the URL and you have no other control over that. If the FQDN is different, then the SPN will be different. But since you're statically (or dynamically) deriving a SPN in the SSO, that SPN can be used for pretty much anything. That same SPN can be used by everything APM Kerberos touches, regardless of the service's actual name.

In the end, we ended up building an SSO Kerberos profile for each IIS application pool account with a static SPN to each. We then used an irule to route the request to the correct SSO Profile:

when HTTP_REQUEST {
    Parse URI to determine target Kerberos SSO Profile and store it in $ssoProfileName
    Ex: example.com/site2 -> $ssoProfileName = "example_site2" (example_site2’s spn would point to applicationPoolSite2Identity)
    WEBSSO::select $ssoProfileName
}

Thanks Kevin for all the help.