Boyar Consulting

Empowering Business Potential with Azure and M365

Generating delta imports out of thin air

I have recently been working on a migration project of on-prem and cloud identities. One of the tasks was to migrate exchange online mailboxes to 95% complete in the new tenant ready for when the user is going to complete the migration process. This involved copying the msExchMailboxGuid, msExchArchiveGuid, any x500 addresses and legacyExchangeDN from the mailbox in the current tenant to a shiny new mailbox.

MIM is the tool of choice using the excellent PowerShell MA by Søren and the EXO PowerShell V2 module in particular the Get-EXOMailbox to get the require attributes. One issue with using this cmdlet is that it doesn’t provide any attributes that can be used to generate a list of changes only which would mean a full import every time MIM executes this MA.

So this begs the question how can we generate a list of changes for a delta import? Generate a hash (SHA256) using Get-FileHash for each row of data during an import (full and delta) save it and then during the next delta import pull a full set of data and compare the hashes from the previous import and present only the objects that have changed to MIM. But this retrieving full sets from EXO regardless of whether a full or delta import is run I hear you say. Yes this is true but it does give the benefit that only changes are imported in to the connector space during delta import step saving time and process effort here and when a delta sync is executed.

Schema Script

The PS MA requires a schema script to define the object types, attributes and data types that are required.

Schema.ps1

$obj = New-Object -Type PSCustomObject
$obj | Add-Member -Type NoteProperty -Name "Anchor-objectGuid|String" -Value "08523e0b-e5e6-4b9b-bdf1-576dd90aa1d9"
$obj | Add-Member -Type NoteProperty -Name "ExchangeGuid|Binary" -Value 0x20
$obj | Add-Member -Type NoteProperty -Name "objectClass|String" -Value "UserMailbox"
$obj | Add-Member -Type NoteProperty -Name "UserPrincipalName|String" -Value "bob@a.com"
$obj | Add-Member -Type NoteProperty -Name "LegacyExchangeDN|String" -Value " /o=ExchangeLabs/ou=Exchange Administrative Group(FYDIBOHF23AAAAA)/cn=Recipients/cn=5f1b300000000000000000000-Bob"
$obj | Add-Member -Type NoteProperty -Name "EmailAddresses|String[]" -Value ("smtp:user1@a.com", "smtp:user1@b.com") 
$obj | Add-Member -Type NoteProperty -Name "ArchiveGuid|Binary" -Value 0x20
$obj | Add-Member -Type NoteProperty -Name "MailboxType|String" -Value "EquipmentMailbox"
                                     
$obj

Import Script

The import pulls in mailbox data from EXO and each objects attributes are combined in to a one long string $rowString the hash of this string is then calculated via the Get-RowHash function. This hash is added to an array $CurrentFullHashes which after the import has finished is written out to a file. On the next import this file is load in to another array $LastFullHashes and if a delta import is executed next the hash of the current import object is check to see if it present in the list of hashes from the last import and if not it must be a change and is added to import collection. Regardless of a full or delta import the array of current hashes is always updated and sent to a file, so that it can be compared against for the next delta.

To connect to Azure a service principal with a certificate is used for authentication. Microsoft documentation on how to achieve this can be found here. The required parameters are held in the MA configuration parameters:

AppId=023002fc-0000-0000-000-7da900a216fa
Organization=a.onmicrosoft.com
CertificateThumbprint=D5CE8FB05EF1A83FD33434940292360DB039B280

and these are splatted in the 

Connect-ExchangeOnline @ConfigurationParameter -CommandName Get-Mailbox

line. The certificate with the private key is stored in the personal store of the service account that is used to execute the MA.

param (
    $ConfigurationParameter,
	$OperationType,
    [bool] $usepagedimport,
	$pagesize
    )

function Write-toLog
{
    [CmdletBinding()]

    param
    (
      [Parameter(Mandatory = $true)]
      [string]
      $DebugFilePath,
      [Parameter(Mandatory = $true)]
      [string]
      $Text,
      [Parameter(Mandatory = $false)]
      [bool]
      $DebugOn=$true
    )
    Begin {
		if (! $DebugOn) {Return}
        if(!(Test-Path $DebugFilePath)) {
            $global:DebugFile = New-Item -Path $DebugFilePath -ItemType File
        }
        else {
            $global:DebugFile = Get-Item -Path $DebugFilePath
        }
    }
    Process{
        $Text | Out-File $global:DebugFile -Append
    }
    End{

    }
}
function Get-RowHash
{
  [CmdletBinding()]
  [OutputType([string])]
  param
  (
    [Parameter(Mandatory = $false)]
    [string]
    $RowString
  )
  Begin
  {
    $debugText = "Entering: $('{0}' -f $MyInvocation.MyCommand) " + (Get-Date) 
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
    $returnVal = $null
  }
  Process
  {
    try {
        $stringAsStream = [System.IO.MemoryStream]::new()
        $writer = [System.IO.StreamWriter]::new($stringAsStream)
        $writer.write($RowString)
        $writer.Flush()
        $stringAsStream.Position = 0
        $returnVal = Get-FileHash -InputStream $stringAsStream | Select-Object Hash
    }
    catch {
      $debugText = "Error: $('{0}' -f$_.Exception) "  + (Get-Date) 
      Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
      Throw $_
    }
  }
  End
  {
    $debugText = "Exiting: $('{0}' -f $MyInvocation.MyCommand) " + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
    $returnVal
  }
}

# Debug Logfile
$DebugFilePath = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData\Exchange\debug_$(get-date -Format "ddMMyyyyTHHmmss").txt"
$LastFull = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData\Exchange\LastFull.txt"
$WriteToDebug = $false
$ModuleName = "ExchangeOnlineManagement"
$ReposityName = "PSGallery"
$CurrentFullHashes = @()
$LastFullHashes = @()
$objCount = 0

$debugText = "PS Ver: $($PSVersionTable.PSVersion.Major) "+ " " + (Get-Date)
Write-toLog -DebugFilePath $DebugFilePath -Text $debugText -DebugOn $WriteToDebug 
$debugText = "Starting Import as: " + $OperationType + " " + (Get-Date)
Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug

if(!(Test-Path $LastFull)) {
    $global:LastFullFile = New-Item -Path $LastFull -ItemType File
}
else {
    $global:LastFullFile = Get-Item -Path $LastFull
    $LastFullHashes = Get-Content -Path $global:LastFullFile
    $debugText = "Loading last full import hashes. Contains $($LastFullHashes.Length) hash(es)"+ " " + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug 
}

#Do not import module as this imported when Connect-ExchangeOnline is executed
#Running an import-module causes an error when running Connect-ExchangeOnline 
#Check if the module is installed if not install
if ((Get-InstalledModule -Name $ModuleName 2>$null).Name.Count -eq 0) {
    $debugText = "Starting Module $ModuleName Install : "  + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
    If ((Get-PSRepository -Name $ReposityName).InstallationPolicy -ne "Trusted") {
        $debugText = "Adding PSRepository $ReposityName : "  + (Get-Date) 
        Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
        [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
        Register-PSRepository -Default
        Set-PSRepository -Name $ReposityName -InstallationPolicy Trusted
        $debugText =  "This task has sucessfully executed."
        Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
        $debugText = "PowerShell reposity $ReposityName is now trusted."
        Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
    }
    $successDetails.Clear()
    $debugText = "Installing Module $ModuleName : "  + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
    Install-Module -Name $ModuleName -ErrorAction Stop -Scope CurrentUser -Confirm:$false 3>$null
    $successDetails.Add("Message", "This task has sucessfully executed.")
    $successDetails.Add("Action", "PowerShell module $ModuleName is now installed.")
    $successDetails + " " + (Get-Date) | Out-File $global:DebugFile -Append
}

$debugText = "Connecting to EXOOnline using: "
Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
foreach($k in $ConfigurationParameter.Keys) {
    $debugText = "  $k - $($ConfigurationParameter.Item($k)) : "  + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
}

try {
    Connect-ExchangeOnline @ConfigurationParameter -CommandName Get-Mailbox
}
catch {
    $debugText = "   Failed to connect to Exchange Online " + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
    $debugText = "   Error: $('{0}' -f$_.Exception)" 
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
}
$debugText = "Retreiving mailboxes "  + (Get-Date)
Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
$userMailboxes = Get-EXOMailbox -RecipientTypeDetails 'UserMailbox' -ResultSize 'Unlimited' -Properties recipienttype,exchangeguid,archiveguid,legacyexchangedn,recipienttypedetails | Select-Object recipienttype,userprincipalname,exchangeguid,archiveguid,legacyexchangedn,@{Name="EmailAddresses";Expression={$_.EmailAddresses | Where-Object {$_ -like "x500*"}}},recipienttypedetails
if ($userMailboxes){
    $debugText = "$($userMailboxes.count) mailboxes retreived from EXO "  + (Get-Date)
    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
        Foreach ($userMailbox in $userMailboxes) {
            $mailboxObj = @{}
            $mailboxObj.add("objectGuid",$userMailbox.ExchangeGuid.ToString())
            $mailboxObj.add("objectClass", $userMailbox.RecipientType)  
            $mailboxObj.add("LegacyExchangeDN",$userMailbox.LegacyExchangeDN)
            $mailboxObj.add("EmailAddresses",[string[]]$userMailbox.EmailAddresses)
            $mailboxObj.add("ExchangeGuid",$userMailbox.ExchangeGuid.ToByteArray())
            $mailboxObj.add("UserPrincipalName",$userMailbox.UserPrincipalName)
            $mailboxObj.add("MailboxType",$userMailbox.RecipientTypeDetails)
            $mailboxObj.add("ArchiveGuid",$userMailbox.ArchiveGuid.ToByteArray())
            $rowString =  $userMailbox.ExchangeGuid.ToString() + `
                $userMailbox.RecipientType + `
                $userMailbox.LegacyExchangeDN + `
                $userMailbox.EmailAddresses + `
                $userMailbox.UserPrincipalName + `
                $userMailbox.RecipientTypeDetails + `
                $userMailbox.ArchiveGuid.ToString()
            $hash = Get-RowHash -RowString $rowString
            $CurrentFullHashes = $CurrentFullHashes + $hash.hash
            $debugText = "     String of concat data row - $rowString "  + (Get-Date)
            Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
            $debugText = "     Hash of data row - $($hash.hash) "  + (Get-Date)
            Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
            if((Get-Item $LastFull ).length -gt 0kb -and $OperationType -eq "Delta") {
                #Does the array of hashes for the last import contain the hash of the current row
                if (!($LastFullHashes.Contains($hash.hash))) {
                    $debugText = "     No matching hash $($hash.hash) for user $($userMailbox.UserPrincipalName) added to delta import "  + (Get-Date)
                    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
                    $mailboxObj
                    $objCount = $objCount + 1
                }
                Else {
                    $debugText = "     Existing hash $($hash.hash) not added to delta import "  + (Get-Date)
                    Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
                }
            }
            Else {
                $mailboxObj
                $objCount = $objCount + 1
            }
        }
}
$CurrentFullHashes | Out-File $LastFull -force
$debugText = "$objCount objects imported " + (Get-Date)
Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
$debugText = "Completed Import and disconnecting session " + (Get-Date)
Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug
$debugText = Disconnect-ExchangeOnline -Confirm:$false | Out-String 
$debugText = $debugText + "`n"
Write-toLog -DebugFilePath $DebugFile -Text $debugText -DebugOn $WriteToDebug

Export Script

Nothing is exported in this MA so just a basic script is in place to satisfy the requirement that this script exists.

param (
    $Username,
	$Password,
    $Credentials,
	$OperationType,
    [bool] $usepagedimport,
	$pagesize
    )

# Debug Logfile
$DebugFilePath = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData\API Exchange\debug_export_$(get-date -Format "ddMMyyyyTHHmmss").txt"

Summary

If you can’t generate a set of only changed objects by using last updated date, or watermarks then this gives another method that can be employed. In addition now you can pull mailbox objects in to MIM using the Exchange cmdlets:)

Published by

paulboyarorguk

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.