Saturday, September 9, 2017

Exchange Database Failovers and MSMQ Event ID 2250

My databases had been automatically switching over from one of my three Exchange 2016 CU4 DAG nodes, several times a day. I would move them back and they would fail back over almost instantly.

I started checking the Event Logs, and there were tons of Event ID 2250 and Event ID 2252 in the Application Log that coincided with the failover timestamps in the Exchange/High Availability/Operational Logs. This only happened on the one Mailbox Server though...the other two servers were running like champs.

The full Event IDs are:

MSMQ Event ID: 2250

Message Queuing will not be able to accept messages temporarily because system paged pool is low. During this period, machine quota will be set to 0. No manual intervention is required at this stage. Once memory utilization has normalized, Message Queuing will automatically resume accepting messages.

MSMQ Event ID: 2252

Message Queuing is resuming accepting messages because the system memory usage has normalized.

**Note** The 2252 event would happen directly after the databases moved off of this server.

Googling those events turned up nothing that really helped. As you can see the 2250 states that the page pool was exhausted, which was not the case, and my pagefile is set to preferred architecture (RAM + 10MB in my case 32,778MB).

Since it was only this one server, it had to be some sort of communication error.

So I did the normal Test-ReplicationHealth, Get-MailboxDatabaseCopyStatus and I noticed that PowerShell was throwing tons of errors like "MBX1 does not exist on DC1" and WinRM errors every time I would run a cmdlet.

That led me check the Domain Controllers and sure enough DCDIAG from Exchange-to-DC and DC-to-DC came back nasty with errors for the DC holding the FSMO roles.

The Cause:

Another domain admin was monkeying around with the firewall and GPOs on the DC's causing all kinds of network issues.

DO NOT turn off the Windows Firewall on production servers, and check your GPO scopes before enabling them!!!!!

The Fix:

1. Take away admin rights from everyone else :)

2. Reboot the DC and check DCDIAG to ensure it was clean...it is now.

3. Reboot the crippled Mailbox server
    - The errors were gone before the reboot, but just to make sure, I bounced it anyway.

**Note** Rebooting the FMSO DC will result in mailbox database dismounts. This is because the FMSO DC holds the AD configuration container for the databases.They should remount cleanly after the DC is back online, but you should do this during off hours if you can, since it will affect client connections to Exchange.

Now your databases should stay put according to their activation preference and those MSMQ events will be gone.

Saturday, August 19, 2017

Exchange External Forward 550 5.7.54 Unable To Relay In Non-Accepted Domain

I recently had this problem where an Accepted Domain in my Exchange environment would get a bounce when mailing a Shared Mailbox that was set with an external forward. In case you have the same setup as me, I've found a fix.

Consider this scenario: you have two Exchange Resource Forest us.domain.com and eu.domain.com.
The eu.domain.com is set as an Internal Relay Accepted Domain in the us.domain.com Exchange environment.
The us.domain.com has a Shared Mailbox, which forwards to an external email address with the ForwardingSmtpAddress switch like so:
Get-Mailbox "Shared Mailbox Name" |fl *forw*
DeliverToMailboxAndForward : True
ForwardingAddress          :
ForwardingSmtpAddress      : smtp:externalemailaddress@externaldomain.net

When users from the eu.domain.com Resource Forest send a message to the Shared Mailbox in the US, they get the following error:
The email to the following recipient(s) could not be delivered:
The remote mail server told: 550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

All other external senders (people from outside the company) can send and messages get forwarded successfully.
Setting the forward by using the ForwardingAddress switch and creating a MailContact causes all senders from both the other forest and external to fail; so you must keep the ForwardingSmtpAddress set.
A brief overview of ForwardingSmtpAddress vs ForwardingAddress:
This is a RecipientIdParameter value which has a higher priority than ForwardingSmtpAddress; meaning if you set the parameter ForwardingAddress, other forwarding settings will be overritten. This setting does not require the Set-RemoteDomain -AutoForwardEnabled, it does require an external MailContact in Exchange.
ForwardingSmtpAddress :
This is a msExchGenericForwardingAddress AD attribute. It has lower a priority than ForwardingAddress and is not accessible in in the EAC; you must use PowerShell to set it. You must also use the Set-RemoteDomain -AutoForwardEnabled $True cmdlet to allow forwarding, but it does not require a MailContact in Exchange.

The Fix:

We'll need to create a dedicated Send Connector to the domain for our external forward.

In the EAC, navigate to Mail Flow, Send Connectors, +

In the New Send Connector window, give it a name like "External Forward" and click Next

Create Send Connector

Leave MX record selected and click Next

Send Connector to MX

Click the + and under the FQDN type the domain name for the external contact, click Save, and Next

Send Connector Domain

For the Source Server, click the + and select your Edge server if you have one, or your Mailbox servers (all of them) and click Ok, then Finish

Send Connector Source Server

Enable Verbose Logging on the Connector:

You'll want full logging on the connector so you can check the SMTPSend protocol logs later to verify successful sending.

In the Exchange Management Shell (EMS), run the following:

Set-SendConnector "External Forward" -ProtocolLoggingLevel Verbose


If the external domain requires it, you'll need to enable forced TLS, else messages will be dropped.

In the EMS run the following:

Get-SendConnector "External Forward" | Set-SendConnector -RequireTLS $true

Now test! Have someone from the other Resource Forest send to the Shared Mailbox and have an external sender send a message the Shared Mailbox and verify that it forwards by using the protocol logs.

**Note** Depending on what source server you used (Edge or Mailbox) that's where you'd check the SMTPSend protocol logs.

Sunday, August 6, 2017

Exchange SpamTitan 421 4.3.2 Maximum Connections Exceeded Limit

I run a SpamTitan Private Cloud Antispam Gateway in my Exchange environment, and its awesome...probably one of the best spam filters I've run in a long time. And their support is top-notch!
And if you happen to run a ITAR/DFARS/DoD compliant Exchange environment, this solution will work for you as all data is housed on the AWS private cloud in the US.

One thing I started noticing was we had quite a few incoming messages being deferred when flowing from the gateway to my Edge server. It was pretty sporadic, where if I had messages from one external sender to multiple internal recipients, one recipient would receive it and the others wouldn't.

In the SpamTitan console, go to Reporting > History and you'll see messages with Delivery Status of Deferred and Delivery Response with the following:

host [Your Exchange external IP] refused to talk to me: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (EdgeServer.domain.com)

In my case, I had tons of these going back months. The message will list your Exchange external IP and the server that dropped the connection (this could be your Edge server or your Mailbox servers if they are internet facing).

What was happening was, my Edge server receive connector was still set to default for Max Inbound Connection Per Source, which is 20.

The SpamTitan uses 60 connections according to their support. So, we need to bump up the limit on the Receive Connector.

Fire up the Exchange Management Shell (EMS) on whichever server is internet facing - the Edge in my environment.

Run the following to verify the current settings:

Get-ReceiveConnector | fl *maxin*

You'll get the output of:

MaxInboundConnection                                  : 5000
MaxInboundConnectionPerSource                  : 20
MaxInboundConnectionPercentagePerSource : 2

As you can see, 20 MaxInboundConnectionPerSource is too low, and the SpamTitan filter will defer those messages until connections become available...or worse case, bounce those message if the time limit expires. That's bad because your users won't receive messages, and they won't even know they were supposed to.

We'll bump up the limit to 100, which should suffice for SpamTitan, and give us a little wiggle room without overloading the Exchange server(s).

Run the following:

Get-ReceiveConnector | Set-ReceiveConnector -MaxInboundConnectionPerSource 100

**Note** Some receive connectors on Mailbox servers are set to unlimited, so you'll wanna specify the connector name in the above cmdlet if you're not running this on an Edge server since it will generally only have one receive connector.

Now, you can verify the settings by running the first cmdlet again:

Get-ReceiveConnector | fl *maxin*
You'll get the output of:
MaxInboundConnection                                    : 5000
MaxInboundConnectionPerSource                   : 100
MaxInboundConnectionPercentagePerSource : 2

You're all set! Now new messages won't be deferred.
Be warned: those messages that were being queued as deferred, will start flowing in...so you'll get reports from users that they're receiving messages from days ago :)

**Note** I do not work for SpamTitan, nor did I receive any endorsement from them on this post.

Saturday, August 5, 2017

Exchange Get Mailbox Sizes By Mailbox Type

This post is really just for my own records because I use these all the time; my higher-ups like to know who has giant mailboxes.
And since my Exchange environment is in a Resource Forest, I have every kind of mailbox imaginable - Linked, User, Room, Archives...

So I have these PowerShell cmdlets at the ready to quickly grab that info when I need to send it to the bosses.

Fire up the Exchange Management Shell (EMS) and run any of the following, depending on what info you need.

Shared Mailboxes:

Get-Mailbox -RecipientTypeDetails shared | Get-MailboxStatistics | ft displayname,totalitemsize,itemcount

--To Export a CSV:

Get-Mailbox -RecipientTypeDetails shared | Get-MailboxStatistics | select displayname,totalitemsize,itemcount | Export-Csv C:\Temp\sharedsize.csv -NoTypeInformation

Room Mailboxes:

Get-Mailbox -RecipientTypeDetails roommailbox | Get-MailboxStatistics | ft displayname,totalitemsize,itemcount

--To Export a CSV:

Get-Mailbox -RecipientTypeDetails roommailbox | Get-MailboxStatistics | select displayname,totalitemsize,itemcount | Export-Csv C:\Temp\roommailbox size.csv -NoTypeInformation

Equipment Mailboxes:

Get-Mailbox -RecipientTypeDetails equipment | Get-MailboxStatistics | ft displayname,totalitemsize,itemcount

--To Export a CSV:

Get-Mailbox -RecipientTypeDetails equipment | Get-MailboxStatistics | select displayname,totalitemsize,itemcount | Export-Csv C:\Temp\equipment size.csv -NoTypeInformation

User Mailboxes:

Get-Mailbox -RecipientTypeDetails usermailbox | Get-MailboxStatistics | ft displayname,totalitemsize,itemcount

--To Export a CSV:

Get-Mailbox -RecipientTypeDetails usermailbox | Get-MailboxStatistics | select displayname,totalitemsize,itemcount | Export-Csv C:\Temp\usermailbox size.csv -NoTypeInformation

Linked Mailboxes:

Get-Mailbox -RecipientTypeDetails linkedmailbox | Get-MailboxStatistics | ft displayname,totalitemsize,itemcount

--To Export a CSV:

Get-Mailbox -RecipientTypeDetails linkedmailbox | Get-MailboxStatistics | select displayname,totalitemsize,itemcount | Export-Csv C:\Temp\linkedmailbox size.csv -NoTypeInformation

Archive Mailboxes:

Get-Mailbox | Get-MailboxStatistics -Archive | ft displayname,totalitemsize

--To Export a CSV:

Get-Mailbox | Get-MailboxStatistics -Archive | select displayname,totalitemsize,itemcount | Export-Csv C:\Temp\Archive size.csv -NoTypeInformation

Happy reporting!

Sunday, July 23, 2017

Exchange Cleaning Up Meetings From Terminated Employee

You have a user who has left the company and they have a bunch of meetings scheduled on Room Mailboxes; this is probably something every exchange admin has to deal with at least once.
If you have a small organization or a limited number of rooms, those meetings can be taking up valuable timeslots that other users could be using.
Why there's not an automated mechanism to remove those meetings when you disable a mailbox? I dunno :(

A lot of times, you might have also granted Full Access to that user's mailbox for someone that has taken over their responsibilities, and that person gets annoying alerts for meetings that no longer need to be scheduled.

I'll show you how to delete the meetings from the Room Mailboxes in one shot, and how to remove all meetings from the user's mailbox calendar by using PowerShell.

Deleting Meetings From the User's Calendar:

First, we'll get a count of how many meetings the user has. This will be useful to ensure that the cleanup cmdlet works later.

Fire up the Exchange Management Shell (EMS) and run the following:

Search-Mailbox -identity disableduseraccountname -SearchQuery kind:meetings -EstimateResultOnly | Out-File C:\Temp\DeletedUserMeetings.txt

**Note** Change disableduseraccountname to name of the user's mailbox, and change the TXT file path.

The above cmdlet will spit out a TXT file with all the meetings that the departed user had scheduled.

The reason I pipe it out to a TXT file, is because there might be tons of meetings...I've seen a user with 6000 meetings scheduled. They thought they'd be with the company until the year 2045 :)

Next, we'll delete all those meetings, by running:

Search-Mailbox -identity disableduseraccountname -SearchQuery kind:meetings -DeleteContent

Once that cmdlet completes, you can run the first one with the -EstimateResultOnly switch to verify all meetings are gone.

Deleting Meetings From the Room Mailboxes:

Now, we'll clean up all meetings scheduled by the terminated user from every room mailbox, in bulk.

Again, we're gonna check and see how many meetings exist for this user, by running:

Get-Mailbox -RecipientTypeDetails roommailbox | Search-Mailbox -searchquery "kind:meetings from:disableduseraccountname" -EstimateResultOnly | Out-File C:\Temp\DeletedUserRoomMailboxMeetings.txt

**Note** Change disableduseraccountname to the mailbox name and the file path.

The cmdlet will search all Room Mailboxes for any meeting scheduled by the terminated user and output a TXT file.

Now, we'll delete all those meetings from the Room Mailboxes:

Get-Mailbox -RecipientTypeDetails roommailbox | Search-Mailbox -searchquery "kind:meetings from:disableduseraccountname" -deletecontent

This cmdlet will search those Room Mailboxes, deleting any meeting scheduled by the supplied user.

You can run the estimate results cmdlet again to verify they were deleted, or open up one of the Room Mailboxes in Outlook and see if any meeting still exists on the calendar.

Saturday, July 22, 2017

Exchange Find AD Users With TargetAddress Set

I recently wrapped up a Lotus Notes to Exchange migration and during co-existence we needed to have the TargetAddress attribute set on Exchange mailboxes so mail would forward back to Notes until their mailboxes were moved over.
The issue I had was after migrating the last of the Shared Mailboxes, some users complained that the mailboxes weren't receiving mail.
What happened was after we migrated, we overlooked a few of those that still had TargetAddress attributes set. So I needed to find all mailboxes that still had them.
But the catch is, we have two Resource Forests that are synced with MIM, so I have MailUsers from another domain that do have TargetAddresses, and I don't want to touch those.
With PowerShell, it's real quick to find those Users that match my specific domain.
First, what is a TargetAddress?
The TargetAddress attribute is an Active Directory User property that forwards mail to a mailbox located somewhere other than your Exchange. This can be external, another Exchange environment, or in my case a Notes system.
To find all users that have the TargetAddress set, for a specific domain and export to a TXT file, run the following cmdlet in the Exchange Management Shell or the Active Directory PowerShell:
Get-aduser -filter {targetAddress -like "*.domain.com"} -properties * | Select-Object Name, targetaddress | Out-File C:\Temp\TargetAddresses.txt
**Note** Change "*.domain.com" to your domain name and change the out-file path.

Now you can go remove those in bulk by following my previous post here.

Saturday, July 1, 2017

Exchange Set Retention Policy Scheduled Task

My organization uses the same mailbox creation provisioning script for multiple Exchange environments through our ServiceNow application and we can't set the Retention Policy during mailbox creation because it would affect those other environments.

So I set up a scheduled task to grab all UserMailboxes without any policy applied, and set our custom policy.

Exchange on-prem doesn't have a the option to run Set-RetentionPolicy -IsDefault $true like Office365 does, so you either have to set the retention policy during mailbox provisioning or manually later on.

So, what we're gonna do is create a Task on our Exchange Management Tools server; you can set it directly on an Exchange server if you choose.

The task will run a PowerShell cmdlet that finds all UserMailboxes with no Retention Policy applied (newly created mailboxes) and it will then set our organization's Retention Policy.

Create a Service Account:

First, you'll want to create a Service Account in your domain, which will be used to run the scheduled task. It's best practice to use service accounts rather than your own account to run scheduled tasks, so if you ever leave your position and they deactivate your account, it won't break the task!

In your domain, create a new user called something like exchscriptaccount and set a super-strong password.

This account will need to be a member of the Recipient Management Role Group, otherwise it won't have permissions to make changes to mailboxes.

Next, add the newly created user to the Local Administrators Group on your Exchange Management Tools server or Exchange server if your running it from there. The scheduled task will need local admin rights to run PowerShell things, and since you have a super strong password, it's not an issue.

Creating The Task:

Create the scheduled task on the Exchange Management Server (or one of your Exchange Servers):

Open the Task Scheduler Control Panel, click Action > Create Task...

On the General tab:

Give it a name like Set Retention Policy

Click "Change User or Group..." hit "Locations" and switch to your domain, then search for your exchscriptaccount service account.

Check the box for "Run with highest privileges"

On the Triggers Tab:

Click "New..."

Set it for how often you need it to run. I run mine Daily at 12AM - no specific reason, but you do want it to run Daily.

On the Actions Tab:

Set the "Action" dropdown to "Start a program"

Under Program/Script, copy/paste the following:


In the "Add arguments" field, copy/paste the following:

-NonInteractive -WindowStyle Hidden -command ". 'C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto; Get-Mailbox -Filter {retentionpolicy -eq $null -and recipienttypedetails -eq 'usermailbox' } | Set-Mailbox -RetentionPolicy 'My Retention Policy'"

**Note** Change 'My Retention Policy' to the name of your policy. You can also change recipienttypedetails -eq 'usermailbox' to other types like LinkedMailbox if you have those.

In the Settings Tab:

Checkmark the following boxes:

- Allow task to be run on demand

- Stop the task if it runs longer than: 1 hour (if it runs longer than an hour, you got something wrong!)

- If the running task does not end when requested, force it to stop

Click OK when you have everything set.

Testing the Task:

In the main task window, right-click your new "Set Retention Policy" task, and click Run.

When it finishes running, you should have a (0x1) Last Run Result.

Check the properties on a newly created mailbox that you know didn't have the Retention Policy set, and it should now have your policy applied.

Now, Exchange will do the boring job of applying the policy for you :)

Sunday, June 4, 2017

Exchange Set Retention Policy In Bulk For Specific Mailbox Type

We recently rolled out a new Retention Policy in my Exchange 2016 environment but I only wanted it to apply to Linked and User Mailboxes; not Shared or Resource Mailboxes.
We'll set a different policy for those. So I needed a way to apply this new Policy to a certain mailbox type. We'll get this done quickly with PowerShell!
Generally, when you apply a Retention Policy in bulk you would run Get-Mailbox | Set-Mailbox without any filters, and that would apply it to all Mailboxes in the organization.
We don't want that, so we'll use a filter to grab the Mailbox Type.
Fire up the Exchange Management Shell and run the following cmdlet:
Get-Mailbox -Filter {(RecipientTypeDetails -eq 'UserMailbox')} | Set-Mailbox –RetentionPolicy "UserMailbox Retention Policy"
The above cmdlet will apply your policy named "UserMailbox Retention Policy" to User Mailboxes only.
Since this is a Resource Forest, I also have Linked Mailboxes in my environment. So we'll need to apply the Policy to those as well:
Get-Mailbox -Filter {(RecipientTypeDetails -eq 'LinkedMailbox')} | Set-Mailbox –RetentionPolicy "UserMailbox Retention Policy"

This cmdlet will filter only Linked Mailboxes and apply the "UserMailbox Retention Policy"

**Note** Change "UserMailbox Retention Policy" to the name of your policy.

If you have multiple policies for different mailbox types, change the RecipientTypeDetails -eq 'mailboxtype' to one of the following:






For instance, setting a Room Mailbox Policy would look like so:

Get-Mailbox -Filter {(RecipientTypeDetails -eq 'RoomMailbox')} | Set-Mailbox –RetentionPolicy "RoomMailbox Retention Policy"

Now your different mailbox types will have the proper policies applied!

Saturday, June 3, 2017

Exchange Export List of Mailboxes Created After a Certain Date

In my organization, we have a billing structure for each site, where they charge back resources like mailboxes to the company. This is done monthly, so I needed a way to get a list of mailboxes to send to site managers.

We'll use our old buddy PowerShell to export a CSV of mailboxes that were created after a certain date.

Fire up the Exchange Management Shell (EMS) and run the following:

Get-Mailbox -ResultSize unlimited | where {$_.whenmailboxcreated -gt (get-date "4/18/2017")} | select displayname,whenmailboxcreated | Export-CSV C:\Temp\Mailboxes-4-18-17.csv

What the cmdlet does is grabs all mailboxes created after 18APR17 with "-gt" (greater than) operator and then exports the CSV with the mailbox display names and when they were created.

**Note** Change the date to how far you need to go back and change "C:\Temp\Mailboxes-4-18-17.csv" to the path and filename of your choosing.

Now you have a when were these mailboxes created report to send out!

Saturday, May 27, 2017

Exchange Messages Missing From The MessageTrackingLog

In my Exchange 2016 environment, we run an Edge server, which is where I generally do message tracking; this gives a better picture of message flow, since all external mail goes in or out of the Edge.

If you don't run an Edge server, you'll need to do message tracking on your Mailbox Servers.

The issue,is, just running the Get-MessageTrackingLog on one Mailbox server, searches only on that server, which means some messages that you know went out, are missing from the log.

So, we need a way to search the log on every server we have. PowerShell makes it easy!

Fire up an Exchange Management Shell on one of your Mailbox Servers, or on your Exchange Management Tools machine and run the following:

$Servers = Get-ExchangeServer;  $Servers | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer -eq $true} | Get-MessageTrackingLog -recipients email@address.com -start 05/23/2017

**Note** Change email@address.com to the email addy you're working with and change the -start date

Now, your message tracking searches will show all messages not just the ones local to a server.

Exchange Count the Number of Messages In The MessageTrackingLog

I had to diagnose a mailbox forwarding to external issue recently, and while I could tell that messages were going out by using Message Tracking on the Edge server, the user that had the forward wanted a count of how many messages were being sent.

So this is quick post on using PowerShell to get a count of messages by sender in the MessageTrackingLog.

Fire up the Exchange Management Shell on the Edge server if you have one, or on one of your Mailbox Servers.

And run the following commands:

$count = get-messagetrackinglog -Recipients "email@address.com" -Start 5/23/2017

$count | Group-Object -Property Sender | Select-Object name,count | sort

**Note** Change "email@address.com" and -start date

You can also do -sender instead of -recipient in the first variable, like so:

$count = get-messagetrackinglog -Sender "email@address.com" -Start 5/23/2017

And if you want to narrow down the results further, you can also add an -EventID like so:

$count = get-messagetrackinglog -Recipients "email@address.com" -EventID SEND -Start 5/23/2017

Happy message counting!

Sunday, May 21, 2017

Exchange/AD Export List of Users With Empty Email Address Fields

In my Exchange 2016 environment, I run a Resource Forest, which means we have to use MIM to sync users from the Accounts Forest. The way we have it configured, the attribute that triggers a sync is the Email Address field. If that field is empty, the account won't get created in the Resource Forest.

Not every account needs an email address, or sometimes the provisioning script might not populate the email address.

My bosses wanted an "empty email address" report, so I needed a way to export a list of users with empty email address field, and their respective OU's.
Since this is an Accounts Forest (with no Exchange) we'll have to use Active Directory PowerShell cmdlets...which aren't as easy to use to grab recipient information.

So, here's a quick one-liner that will grab all ADUsers with no email address populated; the full path of the Organizational Unit they live in; and export that list to a CSV file.

Fire up Active Directory Module for Windows PowerShell and run the following cmdlet:

Get-ADUser -Filter {EmailAddress -notlike "*"} -Properties EmailAddress | Select-Object Name,@{n='OU';e={$_.canonicalname -replace "/$($_.cn)",""}} | Export-Csv "C:\Temp\EmptyEmailAddresses.csv"

**Note** Change the "C:\Temp\EmptyEmailAddresses.csv" path to wherever want to save the csv.

Your csv output will look like so:

Name OU
Stacey Branham exchangeitup.com/Mailboxes
User1 exchangeitup.com/NonMailUsers
User2 exchangeitup.com/NonMailUsers
User3 exchangeitup.com/DisabledUsers
User4 exchangeitup.com/DisabledUsers

Happy reporting :)

Sunday, May 7, 2017

Exchange Count Total Number Of Items In All Mailboxes

After completing the user mailbox migration from Lotus Notes to my Exchange 2016 environment, our higher-ups wanted a "how much email did we move?" report.
While it does seem kind of arbitrary, they wanted a full count to go along with their return on investment report for the board.

We used the Quest migration tool to move the mailbox items, and while it is a pretty decent tool, it didn't really give a full picture of items moved, without digging through each migration batch log - which would take forever.

So I created two quick scripts to count the number of all items in each mailbox in the Exchange organization, and then present the total.

Get Item Count By Database

The first script will count items in each Mailbox Database, which can be useful for a more granular breakdown.

You can download the Get-TotalItemCountInDBs.ps1 file on my gDrive.

Or copy and paste the following text into Notepad, and save as a .ps1 file:

 ##Change "DBNAME" to the Database you're working with
$Mailboxes = Get-MailboxDatabase "DBNAME" | Get-Mailbox

 $MailboxTotalItemCount = 0
 foreach ($Mailbox in $Mailboxes)
 $MailboxStats = Get-MailboxStatistics -Identity $Mailbox
 $MailboxItemCount = $MailboxStats.ItemCount
 $MailboxTotalItemCount = $MailboxTotalItemCount + $MailboxItemCount
 Write-Host "Total Item Count:      $MailboxTotalItemCount"

**Note** Since this is just a quick script, I didn't have to time to make it pretty and prompt for the DB name, so you'll need to replace the "DBNAME" with the database in the ps1 itself, before running it.

When the script is done running, you'll get an output like so - this is for one of my databases:

[PS] C:\Users\stacey\scripts>.\Get-TotalItemCountInDBs.ps1 
Total Item Count:      2983548

Get Item Count In All Databases

The second script will crawl through every database, and present the total count for all mailboxes on all databases.

You can download the Get-TotalItemCount.ps1 on my gDrive

Or, once again, copy the text below and save as a .ps1:

 $Mailboxes = Get-MailboxDatabase | Get-Mailbox
 $MailboxTotalItemCount = 0
 foreach ($Mailbox in $Mailboxes)
 $MailboxStats = Get-MailboxStatistics -Identity $Mailbox
 $MailboxItemCount = $MailboxStats.ItemCount
 $MailboxTotalItemCount = $MailboxTotalItemCount + $MailboxItemCount
 Write-Host "Total Item Count:      $mailboxTotalItemCount"

**Note** You don't have to specify any database in this script, just run it as is.

When this script is done (and yes it will take a while) you'll get the results like so:

[PS] C:\Users\stacey\scripts>.\Get-TotalItemCount.ps1
Total Item Count:      15797733

Now, you'll have a number to present to your "handlers" showing that: yes, we moved over 15 million items (in my case) over to Exchange; and yes that was overkill since most users won't ever even know how to find most of those items :)

Sunday, April 23, 2017

Exchange Internal Mail - 400 4.4.7 Message Delayed

In my Exchange 2016 environment I have three multirole servers in a DAG, running on VMWare. Everything had been humming along nicely until everyone started getting the following bounce message:

4/17/2017 9:09:53 PM - Server at MBX1 ( returned '451 4.4.0 Target host responded with: 4.4.7 Message delayed'

This was for internal messages, between the three Exchange servers. Which is pretty odd, since they're set up to transfer mail out of the box.

I also found the following Events on the servers:

Log Name:      Application
Source:        MSExchangeTransport
Date:          4/17/2017 5:20:52 PM
Event ID:      1035
Task Category: SmtpReceive
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      MBX1
Inbound authentication failed with error UnexpectedExchangeAuthBlobCheckForClockSkew for Receive connector Default MBX1. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [].

As you can see, the "ClockSkew" part clues us in what's going on.
The problem was that the Time sync on the VMware hosts wasn't set up correctly, so the system clocks on the Exchange guests were off...way off.

Anything over 5 minutes will generally cause issues, with everything from mail routing to DAG replication.

To fix it, follow my previous post here on fixing the VMWare host time issues.

Saturday, April 8, 2017

Exchange Resource Forest Selective Trust - Part 2

Continuing from Part 1, we'll now set up our authentication permissions in the Resource Forest to allow users in the Accounts Forest to connect to the Exchange servers.

Setting Authentication Permissions

1.Open Active Directory Users and Computers (ADUC).

2.Under View, ensure that Advanced Features is selected.

3.In the console tree, click the OU where your Exchange Servers live.

4.Right-click the Exchange server object that you want users in the Accounts Forest to access, and then click Properties.

5.On the Security tab, do the following:

Click Add.

In "Enter the object names to select", type the "Resource Forest Exchange Auth" group name, and then click OK.

Select the Allow check box next to the following permissions:

Allowed to Authenticate
Read Account Restrictions
Read DNS Hostname Attributes
Read MS-TS-GatewayAccess
Read Personal Information
Read Public Information permission

Then click OK.

6. Do the above steps on each of your Exchange servers.

Depending on the size of your environment, allow time for the permissions to replicate across forests.

**Note** In my environment, we have some "screwy" ACLs, so we had to add those perms on the Resource Forest DC's as well, just in case you hit a wall and no one can authenticate even after applying the perms.

Now choose one of the users in the "Accounts Forest Exchange Auth" Group and have them try to connect or open Outlook - they should be allowed.

Then choose a user not in the group and do the same - they should be blocked from access.

Now, you should be ready to go and secure with your Selective Trust all set!

Exchange Resource Forest Selective Trust - Part 1

My environment utilizes two Resource Forests (each holding an Exchange 2016 organization) one in the US and one in Europe; both Resource Forests share one Accounts Forest.

We needed to lock down authentication to the US forest because of business regulations...meaning no user in Europe would be able to connect to the US Exchange servers.

In order to fulfil this requirement, I implemented a Selective Trust which is:

"Selective authentication over a forest trust restricts access to only those users in a trusted forest who have been explicitly given authentication permissions to computer objects (resource computers) that reside in the trusting forest."

In this two part series, I'll show you how to enable and configure the Selective Trust and how to apply authentication permissions on the Exchange Servers in the Resource Forest.

Create Security Groups

In order to keep user authentication clean and manageable, we'll use security groups in both forests.

In the Accounts Forest create a Security Group called something like "Accounts Forest Exchange Auth" and add the users who will be allowed to authenticate with the Exchange Forest.

**Note** The users in this group are the only ones will be able to connect to their Linked Mailboxes; users not in this group will get errors in Outlook that it can't find their mailbox.

In the Resource Forest create a Security Group called "Resource Forest Exchange Auth" and add the "Accounts Forest Exchange Auth" Security Group from the Accounts forest into that group.

**Note** This will basically be a nested group where the Accounts Forest Group is a member of the Resource Forest Group. This way, we only have to add members to one group, one time.

We're going to use this group in the Resource Forest to apply authentication permissions on the Exchange servers later.

Enabling Selective Authentication

Using the GUI:

1. Open Active Directory Domains and Trusts.

2. In the console tree, right-click the domain node for the forest root domain, and then click Properties.

3. On the Trusts tab, under either Domains trusted by this domain (outgoing trusts) or Domains that trust this domain (incoming trusts), click the forest trust that you want to administer, and then click Properties.

4. On the Authentication tab, click Selective authentication, and then click OK.

Using the command line:

Open an elevated CMD, and run the following (all on one line):

Netdom trust TrustingDomainName /domain:TrustedDomainName /SelectiveAUTH:Yes /userD:DomainAdministratorAcct

The DNS name of the trusting forest root domain in the trust that is being managed.

The DNS name of the forest root domain that is trusted in the trust that is being managed.

Your domain admin account

Your domain admin password

Now your Selective Trust has been created.

In the next article, we'll set the permissions on the Exchange servers, to allow users to authenticate.

Saturday, April 1, 2017

Exchange Create Address List for Equipment Mailboxes

In my Exchange 2016 environment, users weren't happy with having to search or scroll down the GAL to find resources such as cars and projectors. They were used to the way Lotus Notes would present these. The easiest way to deal with this is to create separate lists for Resource Mailboxes.

I'll show you how to use PowerShell to create a Resources/Equipment Address List.

Fire up the Exchange Management Shell (EMS) and run the following cmdlet:

New-AddressList -Name "All Resources" -RecipientFilter {((RecipientType -eq 'UserMailbox') -and (RecipientTypeDetails -eq 'EquipmentMailbox'))}

**Note** You can name the list whatever you like. I chose "All Resources" because it falls in line with default lists like All Rooms, and that's what my bosses wanted ;)

What this cmdlet does is, create a new Address List (which is dynamic so any new Resource Mailbox is added automagically) that pulls in mailboxes that match the "Equipment" recipient type details.

Next, go ahead and force an Address List update, by running:

Get-AddressList | Update-AddressList

**Note** Depending on how many lists you have and their size, it might take some time to complete.

After the update finishes, Outlook in online mode will pick it up immediately; cached mode will pick it up after it syncs.

Outlook Address List