Measure vs Count vs $i++

Suppose you need to count the number of certain objects (e.g. users in your domain) and you know that the number (and object size) is big, what’s the best way of doing that?

The first one that pops to mind is: just get the collection and check the Count property:

(Get-QADUser -SizeLimit 0).Count

The problem with that is that you are getting (and keeping) all the objects until the enumeration is over, and only check the count after that. This means that in large environments you can easily run out of memory. In my domain (which is not huge at all), this command uses 153 MB of memory. And if I go for a bigger collection by retrieving all AD objects (running Get-QADObject) I run out of memory before the command finishes.

So, let’s look for alternatives. Why don’t we just pipe the output to the Measure-Object cmdlet which by default simply counts the number:

Get-QADUser -SizeLimit 0 | Measure-Object

The cmdlet will be getting the objects through the pipeline and thus will not need to keep them all, so it should use less memory, right?

Wrong. It looks like due to lack of optimization Measure-Object is actually keeping the whole collection of real objects instead of just counting them and letting them go. Thus, it produces the exact same hit on your system and uses awful amounts of memory (hope that this gets fixed in v2! anyone from the PowerShell team reading this?)

So the right way of counting objects is actually to do that yourself by simply incrementing a counter variable:

Get-QADUser -SizeLimit 0 | ForEach-Object {$count++}
$count

On my system this used only 27 MB instead of 153 MB used by Count and Measure-Object – a 6 time reduction!

And, the ratio is going to be even bigger if the number of object being counted grows.

Summary: standard solutions ain’t necessarily the best ones. In this particular case, manual $count++ works more optimally than standard Measure-Object. PowerShell provides multiple ways to solve the same tasks and the performance/resource-intensiveness differences between them can be huge.

For other optimization tips see also: Optimizing PowerShell Performance and Memory Consumption

Tags: , , , , , , ,

20 Responses to “Measure vs Count vs $i++”


  1. 1 Jeffrey Snover January 18, 2008 at 3:08 pm

    > anyone from the PowerShell team reading this?)

    Of course we read your blog Dmitry!
    I’ll go beat the appropriate engineer. 🙂

    Seriously though, thanks for bring that to our attention, we’ll look into why that is the case.

    Keep up the great work.

    Jeffrey Snover [MSFT]
    Windows Management Partner Architect
    Visit the Windows PowerShell Team blog at: http://blogs.msdn.com/PowerShell
    Visit the Windows PowerShell ScriptCenter at: http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx

  2. 2 Shay January 18, 2008 at 3:27 pm

    In case of QAD cmdlets, this might reduce memory consumption as well🙂

    Get-QADUser -SizeLimit 0 -DontUseDefaultIncludedProperties | ForEach-Object {$count++}

    —–
    Shay Levi
    $cript Fanatic
    http://scriptolog.blogspot.com

  3. 3 dmitrysotnikov January 18, 2008 at 4:26 pm

    Jeffrey, don’t beet the engineer too much. The cmdlet is great, and when it becomes optimized it will become absolutely indispensable.

    Shay, thanks for the comment. This makes the call even faster and consumes even less resources.

  4. 4 Ron Bertino July 24, 2008 at 10:27 pm

    Hi Dmitry

    I was wondering if you could give me a hand with a bit of code….I’m sure you’ve never heard that request before! 🙂

    First up, I’ve succeeded on my first task of searching through all Active Directory OUs, and listing OUs with no users in them:

    Get-QADObject -Type ‘organizationalUnit’ | ForEach-Object {
    $count = 0
    get-QADUser -SizeLimit 0 -SearchRoot $_.DN | ForEach-Object {$count++}
    if ($count -eq 0) { $_.DN }
    }

    But I would like to now extend the script to print a table, showing the total number of users per Active Directory OU, sorted from highest to lowest user count.

    The first column should contain the user count per OU, and the second column should contain the OU distinguished name, and should ideally be sorted by user count.

    Any tips you can suggest would be greatly appreciated.

    Thanks for this….and thanks for this very useful blog.

  5. 5 Ron Bertino July 25, 2008 at 12:07 am

    Ok, I got it figured out….at least here’s one solution.

    Get-QADObject -Type ‘organizationalUnit’ | ForEach-Object {
    $UserCount = (get-QADUser -SizeLimit 0 -SearchRoot $_.DN).count
    @{$_.DN = $UserCount}
    } | Format-Table

    I now need to tweak the sort order and table format.
    Almost there…

  6. 6 dmitrysotnikov July 25, 2008 at 6:57 am

    Ron,

    Just modify the last line to:

    } | Sort Count -descending | Format-Table

    And you should be done.

    Also, here’s the AD PowerShell forum which is normally very helpful for questions like this: http://powergui.org/forum.jspa?forumID=173

    Dmitry

  7. 7 Ron Bertino July 25, 2008 at 4:18 pm

    See, that’s my problem…

    I’m creating a custom hash table and passing this through the pipe.

    Therefore the sort-object cmdlet can’t reference a “count” property, as in your example, since that property name doesn’t exist.

    But I’ll take your advice and post this issue through the forum.

    Thanks.

  8. 8 Ron Bertino July 25, 2008 at 4:25 pm

    Ah, found the magic parameter.
    All working now

    Get-QADObject -Type ‘organizationalUnit’ -SearchRoot “OU=test,OU=Corporate Office,DC=MHSINC,DC=CORP” | ForEach-Object {
    $UserCount = (get-QADUser -SizeLimit 0 -SearchRoot $_.DN).count
    @{$_.DN = $UserCount}
    } | Sort-Object values -descending | Format-Table

  9. 9 Ron Bertino July 25, 2008 at 4:31 pm

    And here’s the final code, including a cleaned up table format.
    Note the weird discrepancy between using “$_.value” (plural) in the table format hash table vs having to use “sort-object value” (singular).
    Weird.

    $TableFormat = @{Expression={$_.Name};Label=”OU”}, @{Expression={$_.value};Label=”User Count”}

    Get-QADObject -Type ‘organizationalUnit’ -SearchRoot “OU=test,OU=Corporate Office,DC=MHSINC,DC=CORP” | ForEach-Object {
    $UserCount = (get-QADUser -SizeLimit 0 -SearchRoot $_.DN).count
    @{$_.DN = $UserCount}
    } | Sort-Object values | Format-Table $TableFormat -AutoSize

  10. 10 dmitrysotnikov July 25, 2008 at 4:39 pm

    Very cool!

    You can actually tweak the performance a bit by using:

    Get-QADUser -DontUseDefaultIncludedProperties

    Instead of just Get-QADUser. After all, you don’t need all the account properties – just their count.

  11. 11 Max September 18, 2008 at 7:02 pm

    On Ron’s last post I get an error “Unexpected token ‘@{‘ in expression or statement.”

    Just does not like the ).count@{

    Sorry I am a beginner with PS, this has me puzzled.

  12. 12 Dmitry Sotnikov September 18, 2008 at 8:11 pm

    Max, what specifically are you trying to do?

  13. 13 Max September 18, 2008 at 8:37 pm

    Exactly what Ron was, I want to query how many user objects per OU (bonus would be not to count disabled user objects) and get back the results (Count & OU DN).

    Then i would also like to learn how to do this for only top level OUs (read each top level OU automated in the domain), so count every user object starting from the top level OU on down, return total user object count from top OU down but show complete total not for each sub OU.

  14. 14 Max September 18, 2008 at 8:42 pm

    Just incase, this is the code it is complaining about (below)

    My understanding is this line is to setup the variable for the result format, etc…

    $TableFormat = @{Expression={$_.Name};Label=”OU”}, @{Expression={$_.value};Label=”User Count”}

    The following is a separate line for the actual code below, yes?

    Get-QADObject -DontUseDefaultIncludedProperties -Type ‘organizationalUnit’ | ForEach-Object { $UserCount = (get-QADUser -SizeLimit 0 -SearchRoot $_.DN).count @{$_.DN = $UserCount} } | Sort-Object values | Format-Table $TableFormat -AutoSize

    This is the code last listed by Ron on your blog here, I just added the -DontUseDefaultIncludedProperties as you recommended.

    But I get an error “Unexpected token ‘@{‘ in expression or statement.”

  15. 15 Dmitry Sotnikov September 23, 2008 at 3:06 pm

    Max, I believe this got resolved in this discussion thread: http://www.powergui.org/thread.jspa?messageID=22036

  16. 16 Nick January 5, 2011 at 8:05 pm

    A statement and a question:

    1) don’t forget to set: $count = $null
    as I discovered that ‘somthing’ had placed a value in their throwing off my results (as ++ is incrementing $count)

    2) Were the performance concerns addressed in PowerShell 2.0? This might be a good follow-up blog post.

  17. 17 thommck June 26, 2013 at 10:20 am

    Thanks for clearing this up, I’m still at newbie stage and counting was something that confused me!

    I was wondering if you know if the memory optimization ever happened in v2 or v3 (or the upcoming v4!)

    • 18 Dmitry Sotnikov July 24, 2013 at 7:28 am

      There definitely was some optimization in v3 compared to v2 – I saw a lot of improvements. I have not experimented with these particular examples though. You are welcome to do that yourself!🙂


  1. 1 Reservoir sampling and performance considerations Trackback on January 21, 2008 at 4:26 am
  2. 2 Huddled Masses » Blog Archive » Reservoir sampling and performance considerations Trackback on February 3, 2008 at 7:38 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




My Recent Tweets

Legal

The posts on this blog are provided “as is” with no warranties and confer no rights. The opinions expressed on this site are mine and mine alone, and do not necessarily represent those of my employer - WSO2 or anyone else for that matter. All trademarks acknowledged.

© 2007-2014 Dmitry Sotnikov

January 2008
M T W T F S S
« Dec   Feb »
 123456
78910111213
14151617181920
21222324252627
28293031  

%d bloggers like this: