Back in the days when our hair was on fire with security issues, we brought in a number of industry experts to educate us on developing secure software.  This resulted in the Security Development Lifecycle which all our products now use.  One of the best talks was a from a guy from the NSA who had a talk that was something along the lines of, “The zen of secure software”.  In this talk he argued against the all-or-nothing mindset in favor of a model of constantly expanding the envelope.  Any given moment, you evaluate what you need to do (in this case – to address security issues),  you do it and then … you do a bit more.  Then do it again.  And again.  And again.  And with constant vigilance and time, you would achieve excellence.

And so it is with using PowerShell.  I’ve recently had a number of engagements with people just getting started with PowerShell and getting frustrated with the syntax or some of the semantics and then they look at these thick PowerShell books and feel a bit overwhelmed. 

Don’t be overwhelmed.

PowerShell is meant to be an fun environment to experiment with what is possible.  That means  you’ll try some things and they’ll fail – don’t let that freak you out (it is one of the reasons we added –WHATIF to everything!).  I’ve been trying to learn to do yoga for a while now.  In yoga there are lots of positions that require balance and I’m not particularly good at that.  I used to get super frustrated with these poses until one day my teacher quoted someone saying,  “falling is learning – just focus on having fun”.  That phrase transformed my experience and while I wish I didn’t fall, now when I do, I smile, laugh and get up and try it again.

All you need to do is to learn what you need to do to accomplish the task at hand … and a bit more. Then do it again.  And again.  And again.  Have fun with it and push the envelope.

Now I’m late for yoga – gotta run.   Experiment!  Enjoy!  Engage!

Jeffrey Snover [MSFT]
Distinguished Engineer
Visit the Windows PowerShell Team blog at:    http://blogs.msdn.com/PowerShell
Visit the Windows PowerShell ScriptCenter at:  http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx

So I just installed RU1 on my brand new Exchange 2010 server and then I issue a Get-Exchangeserver -Identity MyExchangeServer and get the following output for AdminsDisplayVersion and ExchangeVersion:

Ok that looks a little familiar for some reason. I go to my Exchange 2010 RTM server and issue the same CMDlet and get:

...The same result! But one server has RU1 installed and the other is RTM. Shouldn't I get a different version number back?

Well... no. Exchange 2007 and forward do not reflect the version number either in the value for AdminDisplayVersion, ExchangeVersion, or at this registry key HKLM\SOFTWARE\Microsoft\v8.0\<Role>\ConfiguredVersion as influenced by roll ups. This is a common misconception.

The most conclusive way to get the version of your exchange server, rollup and all, is to check the file version of ExSetup.exe in the BIN folder.

Here is Exchange 2010 RU1 version:

And here is Exchange 2010 RTM:

Another way of getting this information is to run the following PowerShell one-liner:

GCM exsetup |%{$_.Fileversioninfo}

The below output is from an exchange 2010 server running RU1:

Here is an exchange 2010 RTM server:

You can then correlate the version number you find with those listed here, here or on the actual rollup update download pages.

Hope this post reduces some confusion out there!

- Tom Kern

Sean Kearney (aka Ye110beard of Highway to PowersHell fame ) sent me a pointer to a fun video which posits that maybe W7 is TOO good.  Just maybe it is so good that it will put IT Pros out of business.  I think the exact opposite is true - I think PowerShell and the ability script the OS and products allows IT Pros to generate a ton of value and increase their integrated lifetime earnings.  Still it is a fun video.  You can catch it HERE.

Enjoy!

Jeffrey Snover [MSFT]
Distinguished Engineer
Visit the Windows PowerShell Team blog at:    http://blogs.msdn.com/PowerShell
Visit the Windows PowerShell ScriptCenter at:  http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx

No matter how many lines of code, every software application has scalability limits. Some limits can be caused by limits that are external to the application itself, while others can be inherent within the application. As a software engineering team, it is important for us to understand the scalability limits of the solution as a whole; this includes the limits of both external dependencies and internal code. Not only does this enable us to document and communicate this information to our customers, but it also helps us make better software. It provides us with a deeper understanding of our application within the most demanding environments.

Exchange 2010 is new and different from its predecessors in that, unlike all previous versions of Exchange, Exchange 2010 is the first version designed specifically to run as an on-premise application, as a hosted service in the cloud, and as a combination of both (cross-premise). At 20 million mailboxes and growing, it's more important than ever that our understanding of the limits of Exchange be crisp.

A while ago, we began the process of documenting the scalability limits of both Exchange 2007 SP2 and Exchange 2010. We've looked at all areas: things like organization sizing, rules sizing, IRM messages, secure channels, named properties, load balancing, network latency, and more.

Here is our initial draft (version 0.2) of an Excel spreadsheet documenting our findings thus far. There's a lot of great information in there, including issue descriptions, mitigations, and links where appropriate.

We hope you enjoy this first draft version. Please feel free to send us your comments and feedback.

-- Erin Bookey

Today I am pleased to announce the release of the Exchange Server Pre-Deployment Analyzer (ExPDA).

You can use the Exchange Pre-Deployment Analyzer to perform an overall topology readiness scan of your environment. When you run the Exchange Pre-Deployment Analyzer, it provides a detailed report that will alert you if there are any issues within your organization, which could prevent you from deploying Exchange 2010. For example, the Exchange Pre-Deployment Analyzer will notify you if you haven't deployed the minimum required Exchange service pack on all your existing Exchange servers.

The checks performed by ExPDA are similar to the pre-requisite checks implemented (via Exchange Best Practices Analyzer) in the Exchange 2010 Setup program; in fact ExPDA is based off the Exchange Best Practices Analyzer (ExBPA) engine. However, unlike Exchange 2010 setup, this tool focuses only on overall topology readiness and not the ability to run Exchange 2010 on the local computer. The scan also performs a deep analysis of each existing Exchange 2003/2007 server to verify that it has the necessary updates and configuration in-place to support Exchange 2010. The end report is structured as follows:

• Critical - A configuration problem that will prevent Exchange 2010 from being deployed in the organization. For example, the Active Directory Forest is not operating in Windows Server 2003 Forest Functional Mode or higher.
• Warning - A configuration item that may prevent customers having the best possible experience with Exchange 2010. A warning may also reflect some functionality that is not available in Exchange 2010.

ExPDA is another component in our vision to provide a seamless upgrade experience that reduces the complexities in deploying Exchange 2010. To start planning your upgrade, please utilize the Exchange Deployment Assistant. The Deployment Assistant allows a customer to create Exchange 2010 on-premises deployment instructions that are customized to their environment. The Assistant asks a small set of questions, and based on the answers, it provides a finite set of instructions that are designed to get a customer up and running on Exchange 2010.  Running the Exchange Server Pre-Deployment Analyzer is now a recommended step within the pre-requisites section of the Deployment Assistant.

You can download ExPDA at http://www.microsoft.com/downloads/details.aspx?FamilyID=88b304e7-9912-4cb0-8ead-7479dab1abf2&displaylang=en.  ExPDA is supported on Windows 7, Windows Vista with Service Pack 2, Windows Server 2008 with Service Pack 2, Windows Server 2008 R2, and Windows Server 2003 with Service Pack 2.

Q&A

1. Does this tool replace the Exchange Deployment Assistant?

   No, ExPDA is merely an additional tool that can be used as a step within the upgrade experience. The Exchange Deployment Assistant will walk you through all aspects of the upgrade, namely how to coexist properly with Exchange 2010 and legacy versions of Exchange, whereas, ExPDA is one step within that process and ensures that the environment is ready to have the first Exchange 2010 server deployed.

2. I ran the scan and have questions about the results. What should I do?

   If you'd like to read more about the requirements of Exchange 2010, please see the Planning for Exchange 2010 section on TechNet.

3. I ran the scan and received unexpected results. If I think there's a bug, who can I contact?

   If you need assistance, please visit the Exchange Server Deployment Forum or you can send mail to exbpafb AT Microsoft DOT com.

4. Is this new functionality available in all languages?

   No. ExPDA is only available as a U.S. English version.

5. I'd like to know if my organization is capable of running Exchange 2007. Can I use ExPDA to check this?

   No, ExPDA only verifies if an organization is ready to have the first Exchange 2010 server installed. If you need to determine whether your organization is ready to have Exchange 2007 deployed, you can utilize ExBPA v2.8 and the latest Exchange 2007 ExBPA.Readiness.xml:

   1. Install ExBPA 2.8 and the last ExBPA 2.8 Update.
   2. Download the latest Exchange 2007 service pack rollup. At the time of this writing this is SP2 RU2.
   3. Extract the rollup binaries using this command: msiexec /a filepath to MSI file /qb TARGETDIR=filepath to target folder
   4. Copy the ExBPA.Readiness.xml to the \en folder.
   5. Launch ExBPA.

6. Where can I look at the list of checks made in this new scan?

   If you're familiar with the BPA XML files, the new check is wholly contained within the ExBPA.Readiness.xml file located within %Program Files%\Microsoft\Exchange Server\V14\ExPDA\en folder.

   • Reporting the number of Active Directory trees, domains, sites, admin groups, routing groups, Exchange 5.5 servers, Exchange 2000 servers, Exchange 2003 servers, total mailboxes, Windows 2000 Active Directory servers, Windows Server Active Directory servers, Windows Server Active Directory servers. Report how many Active Directory domain/sites have Exchange servers installed.
   • Verifying that the Schema Master is Windows 2003 SP1 or later.
   • Identifying Active Directory domains that are not in native mode.
   • Identifying Active Directory sites that do not have a global catalog server running Windows 2003 SP1 or later.
   • Verifying that there are zero Active Directory Connector servers in existence.
   • Identifying any SMTP site links in existence.
   • Verifying that the Exchange organization is in native mode.
   • Identifying any non-standard proxy address generators.
   • Identifying whether you have any ambiguously defined email addresses in your recipient policies.
   • Identifying any non-MAPI public folder hierarchies (a.k.a. AppTLH's) in use.
   • Identifying Routing Groups that span Active Directory sites.
   • Identifying any Active Directory sites that span Routing Groups.
   • Identifying any Routing Group Connectors that have specialized settings (activation, max size, accept/reject lists, restrict message type/priority).
   • Identifying any SMTP Connectors that support non-SMTP address spaces.
   • Identifying any SMTP Connectors that use inline domain-wildcarding for address spaces (e.g. *foo.com instead of *.foo.com).
   • Identifying any X.400 Connectors in the topology.
   • Identifying any EDK-based Connectors in the topology (excluding Notes).
   • Verifying that any servers running Exchange 2003 have SP2 or later.
   • Verifying that any servers running Exchange 2007 have SP2 or later.
   • Identifying any SMTP virtual servers that are not using port 25 for incoming/outgoing.
   • Verifying that all Exchange 2003 servers have SuppressStateChanges set.
   • Identifying any Exchange 2003 servers that have active NNTP newsfeeds.
   • Identifying any Exchange 2003 servers that use the Event Scripting service.
   • Identifying any Exchange 2003 servers that have the ExIFS (a.k.a. M:) drive enabled.
   • Identifying any parts of Active Directory that have Access Control Entry inheritance disabled.

Thanks for your continued support. Feel free to post feedback in comments of this blog post, or visit the Exchange Server Deployment Forum.

- Ross Smith IV

In Exchange Server 2010, there is no more single instance storage (SIS). To help understand why SIS is gone, let's review a brief history of Exchange.

During the development of Exchange 4.0, we had two primary goals in mind, and SIS was borne out of these goals:

1. Ensure that messages were delivered as fast and as efficient as possible.
2. Reduce the amount of disk space required to store messages, as disk capacity was premium.

Exchange 4.0 (and, to a certain extent, Exchange 5.0 and Exchange 5.5) was really designed as a departmental solution. Back then, users were typically placed on an Exchange server based on their organization structure (often, the entire company was on the same server).  Since there was only one mailbox database, we maximized our use of SIS for both message delivery (only store the body and attachments once) and space efficiency. The only time we created another copy within the store was when the user modified their individual instance.

For almost 19 years, the internal Exchange database table structure has remained relatively the same:

Then came Exchange 2000.  In Exchange 2000, we evolved considerably - we moved to SMTP for server-to-server connectivity, we added storage groups, and we increased the maximum number of databases per server.  The result was a shift away from a departmental usage of Exchange to enterprise usage of Exchange.  Moreover, the move to 20 databases reduced SIS effects on space efficiency, as the likelihood that multiple recipients were on the same database decreased.  Similarly, message delivery was improved by our optimizations in transport, so transport no longer benefited as much from SIS either.

With Exchange 2003, consolidation of servers took off in earnest due to features like Cached Exchange Mode.  Again the move away from departmental usage continued.  Many customers moved away from distributing mailboxes based on their organization structure to randomization of the user population across all databases in the organization.  Once again, the space efficiency effects of SIS were further reduced.

In Exchange 2007, we increased the number of databases you could deploy, which again reduced the space efficiency of SIS. We further optimized transport delivery and completely removed the need for SIS from a transport perspective.  Finally, we made changes to the information store that removed the ability to single instance message bodies (but allowed single instancing of attachments). The result was that SIS no longer provided any real space savings - typically only about 0-20%.

One of our main goals for Exchange 2010 was to provide very large mailboxes at a low cost. Disk capacity is no longer a premium; disk space is very inexpensive and IT shops can take advantage of larger, cheaper disks to reduce their overall cost. In order to leverage those larger capacity disks, you also need to increase mailbox sizes (and remove PSTs and leverage the personal archive and records management capabilities) so that you can ensure that you are designing your storage to be both IO efficient and capacity efficient.

During the development of Exchange 2010, we realized that having a table structure optimized for SIS was holding us back from making the storage innovations that were necessary to achieve our goals. In order to improve the store and ESE, to change our IO profile (from many, small, random IOs to larger, fewer, more sequential IOs), and to resolve our inefficiencies around item count, we had to change the store schema. Specifically, we moved away from a per-database table structure to a per-mailbox table structure:

This architecture, along with other changes to the ESE and store engines (lazy view updates, space hints, page size increase, b+ tree defrag, etc.), netted us not only a 70% reduction in IO over Exchange 2007, but also substantially increased our ability to store more items in critical path folders.

As a result of the new architecture and the other changes to the store and ESE, we had to deal with an unintended side effect.  While these changes greatly improved our IO efficiency, they made our space efficiency worse.  In fact, on average they increased the size of the Exchange database by about 20% over Exchange 2007. To overcome this bloating effect, we implemented a targeted compression mechanism (using either 7-bit or XPRESS, which is the Microsoft implementation of the LZ77 algorithm) that specifically compresses message headers and bodies that are either text or HTML-based (attachments are not compressed as typically they exist in their most compressed state already).  The result of this work is that we see database sizes on par with Exchange 2007.

The below graph shows a comparison of database sizes for Exchange 2007 and Exchange 2010 with different types of message data:

As you can see, Exchange 2007 databases that contained 100% Rich Text Format (RTF) content was our baseline goal when implementing database compression in Exchange 2010. What we found is that with a mix of messaging data (77% HTML, 15% RTF, 8% Text, with an average message size of 50KB) that our compression algorithms are on par with Exchange 2007 database sizes. In other words, we mitigated most of the bloat caused by the lack of SIS.

Is compression the answer to replacing single instancing all together? The answer to that question is that it really does depend. There are certain scenarios where SIS may be viable:

• Environments that only send Rich-Text Format messages. The compression algorithms in Exchange 2010 do not compress RTF message blobs because they already exist in their most compressible form.
• Sending large attachments to many users. For example, sending a large (30 MB+) attachment to 20 users.  Even if there were only 5 recipients out of the 20 on the same database, in Exchange 2003 that meant the 30MB attachment was stored once instead of 5 times on that database. In Exchange 2010, that attachment is stored 5 times (150 MB for that database) and isn't compressed. But depending on your storage architecture, the capacity to handle this should be there. Also, your retention requirements will help here, by forcing the removal of the data after a certain period of time.
• Business or organizational archives that are used to maintain immutable copies of messaging data benefit from single instancing because the system only has to keep one copy of the data, which is useful when you need to maintain that data indefinitely for compliance purposes.

If you go back through our guidance over the past 10 years, you will never find a single reference to using SIS around capacity planning.  We might mention it has an impact in terms of the database size, but that's it.  All of our guidance has always dictated designing the storage without SIS in mind.  And for those that are thinking about thin provisioning, SIS isn't a reason to do thin provisioning, nor is SIS a means to calculate your space requirements.  Thin provisioning requires an operational maturity that can react quickly to changes in the messaging environment, as well as, a deep understanding of the how the user population behaves and grows over time to sufficiently allocate the right amount of storage upfront.

In summary, Exchange 2010 changes the messaging landscape.  The architectural changes we have implemented enable the commoditization of email - providing very large mailboxes at a low cost.  Disk capacity is no longer a premium.   Disk space is cheap and IT shops can take advantage of larger, cheaper disks to reduce their overall cost.  With Exchange 2010 you can deploy a highly available system with a degree of storage efficiency without SIS at a fraction of the cost that was required with previous versions of Exchange.

So, there you have it. SIS is gone.

- Ross Smith IV

Today I got the question

I’ve looked into all of my.format.ps1xml files but I can’t find out where Format-List is getting its default view for, say, System.Diagnostics.Process:

PS> ps |select -first 1 |fl

Id      : 7616
Handles : 543
CPU     : 10.6860685
Name    : CmdletDesigner

How does it know it should display just these four properties?
It seems there isn’t any ListControl view selected by this type within those formatting files...

Excellent question!  Let’s pick behind the curtain and see how it works.  When an object gets to one of our formatting commands, we use its type to look and see if there is a VIEW defined for it.  These views are defined in our formatting files in $PSHOME directory.  They have filenames like xxx.Format.ps1xml .  If we find it, we use it.  If we don’t find it, we look to see whether this type has a DefaultDisplayPropertySet defined.  If it does, we use it.  If it doesn’t we emit all the properties either as a table or a list depending upon how many properties there are.

But wait a minute – let’s go back and look at this DefaultDisplayPropertySet – what’s that all about?  If you’ve been using PowerShell a while, you’ll realize that PowerShell uses a type mashup system.  Mashups are really just an identity space and a mechanism for 3rd parties to add addition information into an identity space.  For map mashups, the identity space is latitude/longitude and a mechanism for someone to provide the lat/long for all the Starbucks in the world.  For PowerShell, we use TYPEs as an identity space and then have various mechanisms for 3rd parties to add extra information.  One of those mechanisms is TYPE files (e.g. $pshome/types.ps1xml ).  If you take a look at the file, you’ll see that we have a set of entries which are keyed off of a typename and can add extra information.  If you search for System.Diagnostics.Process – you’ll see a bunch of stuff.   One of the things you can add to a type is a MemberSet

One of the things you’ll see is a super double secret MemberSet called PSStandardMembers.   This is a hidden memberset that PowerShell uses for it’s own purposes.  In that memberset, you (we) can define something called DefaultDisplayPropertertySet which provides a set of properties that should be displayed if there isn’t a view defined.  Let me show you that it is really there (you can follow along at home):

PS> $x = Get-Process |Select -First 1
PS> $x |Format-List

Id      : 7616
Handles : 543
CPU     : 10.7172687
Name    : CmdletDesigner

PS> # Let me prove to you that it is hidden
PS> $x |Get-Member PSStandardMembers

PS> # It's really there, just hidden.  You can show hidden things using -Force
PS> $x |Get-Member PSStandardMembers -Force

   TypeName: System.Diagnostics.Process

Name              MemberType Definition
----              ---------- ----------
PSStandardMembers MemberSet  PSStandardMembers {DefaultDisplayPropertySet}

PS> $x.PSStandardMembers
PSStandardMembers {DefaultDisplayPropertySet}

PS> $x.PSStandardMembers.DefaultDisplayPropertySet

ReferencedPropertyNames : {Id, Handles, CPU, Name}
MemberType              : PropertySet
Value                   : DefaultDisplayPropertySet {Id, Handles, CPU, Name}
TypeNameOfValue         : System.Management.Automation.PSPropertySet
Name                    : DefaultDisplayPropertySet
IsInstance              : False

PS> $x.PSStandardMembers.DefaultDisplayPropertySet.ReferencedPropertyNames
Id
Handles
CPU
Name
PS>

Now it really isn’t secret we just haven’t talked about it much.  It is documented @ MSDN HERE .

PSStandardMembers is used for a number of things in addition to DefaultDisplayProperties.  In particular, it is used to control object serialization.  You can find out more about that in my blog entry HERE.

Enjoy!

Jeffrey Snover [MSFT]
Distinguished Engineer
Visit the Windows PowerShell Team blog at:    http://blogs.msdn.com/PowerShell
Visit the Windows PowerShell ScriptCenter at:  http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx