I came across this topic troubleshooting a support case where users were getting Access Denied to a site using Trusted Provider (SAML) authentication.
Users were given permission to the site using a group that had other groups nested in it. The users were not direct members of the group being used for permission. For example:
Contoso\Level1 – Users are direct members of this group.
Contoso\Level2 – No users in this group, only the "Level1" group and some other groups. This is the group used in SharePoint site permissions.
So we got a Fiddler trace of the user logging in via SAML auth and found that their SAML assertion only contained Role claims for the "Level1" group, and not the "Level2" groups. If your SAML assertion does not contain a certain group, then SharePoint does not believe you are a member of that group, and you will not get any permission that has been assigned to that group within SharePoint.
In my research, I wasn't finding any definitive answers about whether or not nested groups should typically be part of a users Role claims, so I tested it. Short answer: it depends on which attribute you have mapped to your Role claim in your Relying Party Trust.
I tested this in my lab with Active Directory Federation Services (AD FS) as my Trusted Provider, and found that as long as I was using the standard "Token-Groups" attributes mapped to "Role" in my claim rules, all groups, including nested groups showed in my SAML assertion.
Upon further investigation, I found that in my customers case, they were not using ADFS. They were using a 3rd-party trusted provider that was using the "MemberOf" Active Directory attribute to map to the Role claim. The problem with that is the MemberOf AD attribute only contains groups the user is a direct member of.
Either change the claims mappings for Role to use the Token-Groups attribute, or only use groups in site permissions that users are direct members of.