Connect to SharePoint Online CSOM through ADFS with PowerShell

To manage a SharePoint Online environment I find the CSOM (Client Side Object Model) for SharePoint very usefull. Untill now we used a separate account for this. The UPN of this account was in this form: [account name]@[tenant name].onmicrosoft.com. This was very practical as it even allows access when ADFS is down.

Being one of the admins of the Office 365 enviroment I was able to create such an account. However there may be plenty of situations when one would like to query a site or site collection, but cannot use CSOM because of ADFS authentication. For the latter I found a solution which I will share here.

First of all lets look the answer given to this question by “Brite Shiny” (who also asked the question). This lists the prerequisites needed to authenticate through ADFS.

Summarized these are needed:

  1. Uninstalled the SharePoint Online Management Shell – I found this was not necessary in my case. However, it may be necessary in other cases.
  2. Installed the latest SharePoint Online Client Components SDK (http://www.microsoft.com/en-us/download/details.aspx?id=42038). As “Brite Shiny” explains note the “Online” part, as it is different from the SharePoint 2013 Client Components SDK
  3. Ensured that the Microsoft.SharePoint.Client and Microsoft.SharePoint.Client.Runtime dlls in the program loaded from the 16 version in the Web Server Extensions folder

Besides this you also need at least PowerShell 3.0 (otherwise you can’t use the needed dlls).

PowerShell 3.0 can be downloaded as part of Windows Management Framework 3.0.

For the script I give credit to Michael Blumenthal. On his (old) blog he posted this post to which I made some minor adjustments.

Short and sweet, here is the script:

Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
$webUrl = Read-Host -Prompt "HTTPS URL for your SP Online 2013 site"
$username = Read-Host -Prompt "Email address for logging into that site"
$password = Read-Host -Prompt "Password for $username" -AsSecureString
$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($webUrl)
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)
$web = $ctx.Web
$lists = $web.Lists
$ctx.Load($lists)
$ctx.ExecuteQuery()
$lists | select Title

As you may notice, I specify the full path to each of the dll’s so I am sure that the correct version is loaded.

As you may imagine, in stead of just getting the Title property of all lists there is so much more that can be done. However I leave this to each to decide for their own how far they want to go to script against SharePoint Online.

Collect site collection information from Office 365

Collecting users and groups from Office 365 is a relatively long running script (as mentioned in my previous post). Fortunately collection more general site collection information is easier. There is not very much data that can be collected, this is because a limitation in PowerShell’s access to SharePoint Online (before the upgrade to SharePoint 2013 this was not possible at all for SharePoint Online).

This script is a fairly short one but can be expanded in several ways (of which I will offer a few suggestions).

What it all comes down to can be summarized in this one-liner (don’t forget you will need to be connected to the Office 365 Service and the SharePoint Online service before running the scripts on this post):

$sitecollections = Get-SPOSite -Limit ALL -Detailed | Export-Csv -Path $outputPath -Delimiter ';' -NoTypeInformation;

In this case you may replace the variable $outputPath with the path where you want to have the results written to.

The following example is what I use at a client to monitor site collection storage quota’s (among with a few other things):

function Collect-SiteCollectionInfo
{
    Param(
    [Parameter(Mandatory=$true)]
    [ValidateNotNull()]
    [string]$outputFullFilePath,
    [Parameter(Mandatory=$false)]
    [switch]$selectSites,
    [Paremeter(Mandatory=$false)]
    [char]$csvSeparator = ';'
    )
    Write-Host "Collecting site collections";
    $sitecollections = $null;
    if($selectSites)
    {
        $sitecollections = Get-SPOSite -Limit ALL -Detailed | Out-GridView -Title "Select site collections from which to collect data." -PassThru;
    }
    else
    {
        $sitecollections = Get-SPOSite -Limit ALL -Detailed;
    }
    $itemArray = @();
    foreach($item in $sitecollections)
    {
        [double]$storageUsed = $item.StorageUsageCurrent;
        [double]$storageQuota = $item.StorageQuota;
        [double]$ratio = ($storageUsed / $storageQuota) * 100;
        [string]$percentage = $ratio.ToString("N1") + "%";

        $temp = New-Object System.Object;
        $temp | Add-Member -MemberType NoteProperty -Name "Title" -Value $item.Title;
        $temp | Add-Member -MemberType NoteProperty -Name "Url" -Value $item.Url;
        $temp | Add-Member -MemberType NoteProperty -Name "WebsCount" -Value $item.WebsCount;
        $temp | Add-Member -MemberType NoteProperty -Name "LastContentModifiedDate" -Value $item.LastContentModifiedDate;
        $temp | Add-Member -MemberType NoteProperty -Name "StorageUsageCurrent" -Value $item.StorageUsageCurrent;
        $temp | Add-Member -MemberType NoteProperty -Name "StorageQuota" -Value $item.StorageQuota;
        $temp | Add-Member -MemberType NoteProperty -Name "StoragePercentageUsed" -Value $percentage;
        $itemArray += $temp;

    }
    $itemArray | Export-Csv -Path $outputFullFilePath -Delimiter $csvSeparator -NoTypeInformation;
    Write-Host "Succesfully collected sitecollection information.`r`nFile saved at $outputFullFilePath" -ForegroundColor Green;
}

Now compared to the one-liner it may seem a bit overwhelming, but the output will be more relevant then before (where all values of all sites where returned).

For example if you pass the parameter -selectSites you will be able to select the sites from which you want to collect data (which I personally find the easiest way to choose which sites I want to receive details about), another option would be to replace the Out-GridView by a Where-Object and filter the results in that way.

In lines 25 through 28 I calculate the percentage of the assigned stored that is in use. If you are not interested in this number you can simply remove this from the script (in this casealso remove line 37 which will try to write the percentage), however I think that in most cases this is relevant and useful information. You could also add other calculations (for example a percentage of used Resources or the ammount of sub webs).

The part from line 30 through 38 does nothing more than adding only the data that is relevant for my report to the output. If you want to collect additional information (like Resource usage etc.) you can add this to the $temp object just like the other properties are added.

Get groups with users from SharePoint Online

One of the things PowerShell enables you to do with Office 365 (particularly SharePoint Online) is collecting bulk info. In this post I will be providing a nice little script which can be used to collect groups from site collections including the names of users in those groups.

The main reason you might want to collect this is the information takes quite some time to be collected. By the time the information would be needed It would take a long unnecessarily amount of  time. If the data however is already collected the requested information can be looked up quickly. The only real downside is that your data used will be “old” data. How old depends on how often you execute the function in this post.

Before going into detail about what the script does, let me elaborate about what goes in and what comes out.

There is one mandatory parameter which must be specified: “outputFullFilePath”. This will be the path where the csv will be stored. Providing an invalid or unreachable path will result in the output being lost.

Optional parameters are:

  • csvSeparator: this will be used as separator for the output csv file, by default its value is ‘;’
  • internalSeparator: this will be used as separator inside csv fields (make sure it is different from the csvSeperator), by default its value = ‘,’
  • selectSites: if selected you will be prompted to select of which site collections the groups will be collected (this is a switch it requires no value, if omitted its value is false).

The output will be a csv file with the following headers: SiteCollectionUrl, LoginName, Title, OwnerLoginName, OwnerTitle, GroupUsers, GroupRoles

If the output file is opened in Microsoft Excel the columns can be used for filtering and searching. Making it an east way to find out who is in which group or where a certain person has access over all selected site collections.

Important note: groups can only be collected if the account that runs the script is site collection admin. Tenant admin is not enough! The account has to be specified at each site collection as site collection admin.

Important note: before the following script can be run a connection to the Microsoft Online service and the SharePoint Online service must be established. For more information on how to achieve this, check out this previous post.

Here is the total script (further down I will highlight the main parts of the script):

function Get-SiteCollectionGroups {
    Param(
    [Parameter(Mandatory=$true)]
    [ValidateNotNull()]
    [string]$outputFullFilePath,
    [Parameter(Mandatory=$false)]
    [switch]$selectSites,
    [Parameter(Mandatory=$false)]
    [char]$csvSeparator = ';',
    [Parameter(Mandatory=$false)]
    [char]$internalSeparator = ','
    )
    Write-Host "Collecting site collection groups";
    $SiteCollectionGroups = @();
    $sites = $null;
    if($selectSites)
    {
        $sites = Get-SPOSite -Detailed | Out-GridView -Title "Select site collections to collect groups from" -PassThru;
    }
    else
    {
        $sites = Get-SPOSite -Detailed;
    }
    [int]$counter = 0;
    [int]$total = $sites.Count;
    [string]$counterFormat = "0" * $total.ToString().Length;
    foreach($site in $sites)
    {
        $counter++;
        Write-Host "$($counter.ToString($counterFormat))/$($total.ToString($counterFormat)) [" -ForegroundColor Yellow -NoNewline;
        Write-Host "$($site.Url)" -ForegroundColor Cyan -NoNewline;
        Write-Host "]: " -ForegroundColor Yellow -NoNewline;
        try {
            $groups = Get-SPOSiteGroup -Site $site;
            foreach($group in $groups)
            {
                [string]$groupUsers = "";
                foreach($user in $group.Users)
                {
                    $groupUsers += "$user$($internalSeparator)";
                }
                if($groupUsers -match "$($internalSeparator)$")
                {
                    $groupUsers = $groupUsers.Substring(0, $groupUsers.Length-1);
                }
                [string]$groupRoles = "";
                foreach($role in $group.Roles)
                {
                    $groupRoles += "$role$($internalSeparator)";
                }
                if($groupRoles -match "$($internalSeparator)$")
                {
                    $groupRoles = $groupRoles.Substring(0, $groupRoles.Length-1);
                }
                $group | Add-Member -MemberType NoteProperty -Name "SiteCollectionUrl" -Value $site.Url
                $group | Add-Member -MemberType NoteProperty -Name "GroupUsers" -Value $groupUsers
                $group | Add-Member -MemberType NoteProperty -Name "GroupRoles" -Value $groupRoles
                $SiteCollectionGroups += $group;
            }
            Write-Host "$($groups.Count) groups are successfully collected" -ForegroundColor Green;
        }
        catch
        {
            Write-Host "Groups could not be collected" -ForegroundColor Red;
        }
    }
    $SiteCollectionGroups | Select SiteCollectionUrl,LoginName,Title,OwnerLoginName,OwnerTitle,GroupUsers,GroupRoles | Export-Csv -Path $outputFullFilePath -Delimiter $csvSeparator -NoTypeInformation
    Write-Host "Site collection groups are collected and written to $outputFullFilePath" -ForegroundColor Green
}
# 2 examples, the first is a minimal call, the second is a call with all optional parameters
#Get-SiteCollectionGroups -outputFullFilePath "C:\Backup\AllSiteCollectionGroups$(Get-Date -Format "yyyyMMddhhmmss").csv"
#Get-SiteCollectionGroups -outputFullFilePath "C:\Backup\AllSiteCollectionGroups$(Get-Date -Format "yyyyMMddhhmmss").csv" -csvSeparator ',' -internalSeparator ';' -selectSites

At line 14 we create a generic collection (which can hold any type of object). At line 58 each group is added to this collection. At line 67 this collection is exported to the csv file which is specified at the outputFullFilePath parameter.

If the switch is set to manually select site collections a prompt will be shown. This will be in form of an Out-Gridview (line 18). You can select multiple items with Ctrl or Shift. If manual selection of sites is off (not set) then the groups of all site collections will be collected. Because of the time it takes to collect groups it is advised to only collect the most important site collections. Keep in mind that the collection of the groups is dependant on the permissions of the account that runs them. If the account is not site collection admin of one site no groups will be collected and the host will show a red line where the site collection URL is mentioned (line 64).

Because the process may take a while I added a progress indicator. It does not give an accurate estimation for the remaining time (as it only counts the amount of site collections and not the remaining groups or users). For this three variables are used. They are defined at lines 24 through 26. At line 29 the counter is raised by one for every site collection. At line 30 through 32 the count is written to the host including the URL of the current site collection. Note the switch “NoNewLine” which means that the success or error message (lines 60 and 64) are places behind it in stead of below the counter.

The main loops are quite simple. First there is a loop through all site collections (starts at line 27). Inside this loop there is a loop which loops all groups for each site collection (starts at line 35). Inside each group, all users are added to a string, also all roles of the group (these are only roles on site collection / root site level). After the users and roles are collected the site collection URL, the groups users and the group roles are added to the group object (at lines 55 through 57). Finally the group object is added to the siteCollectionGroups collection.

At the bottom of the script there are three lines commented. The first of the three provides a brief explanation of the two following examples.

The first example (second comment line) is a minimum required use of the function. It only specifies the outputFullFilePath (if this parameter is omitted you will be prompted to enter it before the script is ran.

The second example (third comment line) has all optional parameters, this includes the separators and the manual selection switch.

Save the script someplace, remove the hash (#) before one of the examples, and modify this as it suits your need. Then simply run the file and wait… After completion check the file in the location that is specified in the script and start working the numbers.

Because the file is in CSV format it is easy to load it in PowerShell and use scripting to quickly analyse data.

In my next post I will share a followup script which collects external users over all site collections using the output csv of this script as input.