Category Archives: PowerShell

AWS Gotcha: IAmazonS3.CopyObject() Method On S3 Objects Changes ACLs

There are many gotchas with Amazon’s AWS (I suppose that is true with all technologies).  Here is one that plagued me just the other day:

I had a bunch of publicly readable files in an S3 bucket.  I needed to update some headers for all items in the bucket, so I wrote a PowerShell script that calls the AWS .Net SDK CopyObject() method on an instance of the Amazon.S3.AmazonS3Client class. (Of course, I could have just done this in C# but PowerShell seemed easier for my purposes.)

Advanced aside: CopyObject() actually comes from the IAmazonS3 interface, which is implenented in AmazonS3Client. An AmazonS3Client object is returned from the Amazon.AWSClientFactory.CreateAmazonS3Client() factory method as show below.

This is an excerpt from my script, the variables should be self explanatory if you are familiar with AWS:

$s3Config = New-Object Amazon.S3.AmazonS3Config
$s3Config.RegionEndpoint = [Amazon.RegionEndpoint]::USEast1
$s3client = [Amazon.AWSClientFactory]::CreateAmazonS3Client($AccessKey, $SecretKey, $s3Config)

$s3Request = New-Object Amazon.S3.Model.CopyObjectRequest 
$s3Request.MetadataDirective = "REPLACE"
$s3Request.SourceBucket = $s3Bucket
$s3Request.SourceKey = $s3object.Key;
$s3Request.DestinationBucket = $s3Bucket
$s3Request.DestinationKey = $s3object.Key;
$s3Request.Headers.Item('Content-Disposition') =  <new-content-disposition>
$s3Request.Headers.Item('Content-Type') = <new content type>
$s3client.CopyObject($s3Request)

What I learned too late was that CopyObject() removes the public read ACL from my $s3Object.  I ended up needing to clean up my S3 objects to re-enable public read, using another PowerShell script similar to the following:

$s3Config = New-Object Amazon.S3.AmazonS3Config
$s3Config.RegionEndpoint = [Amazon.RegionEndpoint]::USEast1
$s3client = [Amazon.AWSClientFactory]::CreateAmazonS3Client($AccessKey, $SecretKey, $s3Config)

$s3Request = New-Object Amazon.S3.Model.PutACLRequest
            
$s3Request.BucketName = $s3Bucket
$s3Request.Key = $s3object.Key
$s3Request.CannedACL = [Amazon.S3.S3CannedACL]::PublicRead
$s3client.PutACL($s3Request) > $null

I guess the moral of the story is 1) AWS is fraught with best practices that can only be learned by making mistakes, and 2) Test. Test. Test.

Talk tonight in Newport Beach, CA: Introduction to PowerShell

It has been a long time since I’ve written… time has gotten the better of me, I suppose.

At any rate, I will be giving a brief talk tonight (April 9) on PowerShell. I will discuss what it is, why you should care as a technology professional, and how to get started learning it.

If you are interested, here is a link with more information: http://www.meetup.com/vNext-OrangeCounty/events/143003342/

PowerShell ii

I try and work as much as possible from the PowerShell host, as opposed to using the mouse to point and click. One common use case is for me to open different file types.

As with everything in PowerShell, there are several ways to do this.

I could enter the file path as a parameter to the program that opens the file type. A relatively easy example would be to open a text file in Windows Notepad. If I have a file called “hello.txt” in my PWD (Present Working Directory) I could type notepad '.\hello.txt'.

This would launch that file in Notepad.

But what if I want to open an Excel file. By default, I CANNOT type excel ‘.\hello.xls’.

Alas, there is an easy way to open this file. Just type ii '.\hello.xls'

By the way, I don’t type out the whole file name or even the single quotes around the file name. I just type the first couple letters of the file name and press tab…. PowerShell tabs through all matching files for me. So I would just type ii hel<tab><enter>.

ii is an alias for Invoke-Item. It does not require knowing the name or location of the program that opens any given file type.

Advanced Aside: ii opens the file based on the Windows file extension mappings. When you install most programs on your machine, they register with Windows to open specific file extensions by default. It usually is not necessary, but you can change these mappings in the Control Panel.

PowerShell %

In a previous post, I demonstrated the Where-Object cmdlet (whose alias is ?).

Another important cmdlet is ForEach-Object.  In fact, it is so important that it also has a simple alias: %

Say for example that you want to calculate the length in kilobytes of every file in the current directory. It’s this simple: Get-ChildItem -File | % {$_.Length / 1KB}

So what’s going on here?

The left side of the pipe (see my previous blog) is enumerating all children (e.g. files) in the current directory. I use the -File switch to limit the results to files. On the right side of the pipe, we use ForEach-Object to loop through each file and calculate the file length, in KB. We wrap the calculation in brackets { and } because this is an expression we need PowerShell to evaluate. Note that because it is such a common unit of file system measurement, PowerShell also understands the notation KB.

Advanced Aside: As I’ve said before, there are many ways to do the same thing in PowerShell. For instance, you could also have done this: Get-ChildItem -File | Select-Object {$_.Length / 1KB}. In this case, the Select-Object mechanism also offers the ability to pretty this up. Try this for instance: Get-ChildItem -File | Select-Object Name, @{Name="Kbytes";Expression={$_.Length / 1Kb}} | ft -AutoSize. But for a quick way to enumerate values of a collection on your PowerShell host, what could be simpler then % ?

Here’s another example:

Let’s say you wanted to enumerate a list of all odd numbers from 1 to 10,000…. This PowerShell host syntax looks a bit convoluted, but you would just type 1..10000 | % { if ($_ % 2 -eq 0) {return} write-host $_ }. To make this easier to read, just imagine the syntax were broken out on multiple lines, like this:

NOTE: this code will not run due to the position of the line breaks:

1..10000 | %         # for each number from 1 through 10,000
{
   if ($_ % 2 -eq 0)    # does current object mod 2 have a remainder of zero
   {
      return               # if its even then do a noop
   }
   ## else
   Write-Host$_         # output current object (i.e. the odd numbers)
}
Advanced Aside: It is possible to divide up code over multiple lines on the PowerShell host, as long as your code follows certain rules regarding the placement of curly braces and such.

PowerShell $_

In my previous post — my first blog ever! — I gave a high level overview of PowerShell. I tried to provide enough material to make it immediately useful.

But in order to see how typing could possibly be more efficient then clicking around with a mouse, I need to introduce a few more basic constructs.

Here is a (contrived) hypothetical: I log onto a production machine that far too many people have access to. And I see something strange. iTunes is installed. Why would someone install iTunes on a production machine? I need to find out who installed it and decide whether I can safely remove it.

The graphical way:

  1. For starters, in Windows 8.1, it just took me almost five minutes just to find the Event Log Viewer Application…. but that’s a rant for another day.
  2. I click on the “Application” log, but see lots of useless metadata (such as a bunch of event entries with a “Task Category” of “None.”)
  3. So I randomly click on a log entry. The bottom half of my screen shows details. Do I have to read all the details for each entry? Should I have chosen another career path?
  4. I move down to the next event log entry. More of the same. The text just blurs together.
  5. Now I notice an option on the right side of my screen to “Filter Current Log…”
  6. But I click on this and am already lost. What could the correct “Event level” be? What about the “Event sources” or “keywords?” This did not help.
  7. This is where I am thinking it’s time to hire an intern.
Windows Event Viewer

Windows Event Viewer Chaos – Now What?

And now the PowerShell way:

  1. I launch PowerShell (I have it pinned to my Windows desktop).
  2. I type: Get-EventLog Application | ? {$_.Message -like "*itunes*"} and get this:
    PowerShell: list of iTunes install events

    PowerShell: list of iTunes install events

  3. Hmmm. Id 152 looks interesting. I type: (Get-EventLog Application | ? {$_.Index -like "152"}).UserName PowerShell thinks for a second and then replies back with this: Peter
  4. So I go talk with Peter and find out he installed it by accident. He promises to be more careful next time.
  5. Life is good.

Advanced Aside: in PowerShell, there are MANY ways to do the same thing. For example, starting with PowerShell V3, I could have typed Get-EventLog Application | ? Message -like "*itunes*" instead of Get-EventLog Application | ? {$_.Message -like "*itunes*"}. Or I could have listed all properties of Index 152, instead of just the UserName. Or I could have formatted the results of my first query as a list instead of a table.

Now to Explain What I am doing in step 2

  • Get-EventLog Application: The word “Application” is called a parameter. With an important exception (called a switch), parameters are name/value pairs, that look like this: Verb-Noun -ParamaterName -ParamaterValue. So in my case I would type Get-EventLog -LogName Application. But when I type help Get-EventLog -Full (note help is short (an alias) for Get-Help) I see a detailed explanation of the parameter -LogName including this information: Position? 1. This means -LogName is what is called a positional paramamter, meaning that if the value appears in a certain position after the cmdlet (in this case first), then the name -LogName is not required. This helps shorten the amount I have to type into the PowerShell host, since PowerShell is all about user efficiency. So instead of typing Get-EventLog -LogName Application, I just type Get-EventLog Application.

    Advanced Aside: -Full is an example of a switch, i.e. it is like a parameter with a name/value pair but it only has a name, no value. In other words, it is a Boolean indicator where it’s absence is like “false” and its presence is like “true.” So when I type help Get-EventLog -Full I am turning on a property (more information) for the cmdlet.

  • | (pronounced pipe): Indicates that the output to the left of the pipe becomes the input to the right of the pipe. So in my case, the result of Get-EventLog Application becomes the input for ? {$_.Message -like "*itunes*"}.
  • ? — The “?” is NOT a typo. It is short (an “alias” for) Where-Object, another cmdlet. In other words, I am taking the results on the left side of the pipe and searching only for results WHERE
  • {$_.Message -like “*itunes*”}: OK, this is the hardest part for me to explain. For starters, the stuff between { and } is called a script-block. It is basically an expression that we are asking PowerShell to evaluate. Buried in the middle of our script block is -like. This means our script block expression is a Boolean expression; within the script block, we are comparing something on the left side of -like to something on the right hand side of -like. In other words, if the something on the left side is ALIKE to what is on the right side then the expression is true. In our case, we are comparing $_.Message to "*itunes*".

    Hopefully the "*itunes*" part is pretty self-explanatory. It’s just looking for any strings (case-insensitive) that contain the text “iTunes.” $_.Message requires some discussion, however. Remember that everything in PowerShell is an object. The syntax “$_” is equivalent to the keyword this in C#. In other words, $_ means the instance of this object (where each object is one row/entry from Get-EventLog Application. The . (period) after $_ tells us we are looking at the property of an object. This is just like many other programming languages. In C#, for instance, we could type string x = "hi"; int length = x.Length where x is an instance of a string (an object) and x.Length is the property Length on the object x. So in summary, $_.Message means we are looking at the property Message on the current instance of whatever object came in from the left side of the pipe (an entry in the EventLog).

    Now things get more fun…. If we run the left hand side of our pipe statement (Get-EventLog Application) by itself, we would find a lot of results. This makes sense. Surely the computer has more than one entry in the event log. So we are actually piping to the right hand side ? {$_.Message -like "*itunes*"} each EventLog entry individually. And the right hand side of our pipe is outputting to us every EventLog entry in which the -like expression is true. So PowerShell is showing us EVERY Application EventLog entry in which the “Message” contains “iTunes” (case-insensitive).

  • Finally, notice in step 3 I wrapped the entire command in parenthesis ( and ) and then added the .UserName. That enabled me to hone in on just the specified property I was looking for, the user who installed iTunes. I could just have easily typed: Get-EventLog Application | ? {$_.Index -like "152"} and gotten back all of the properties, including UserName.
Advanced Aside: Although I ran this example against the event log on my local machine, I could just have easily used PowerShell on my local machine to check event logs on remote machines. It is not necessary to remote desktop into a remote machine to check event logs using PowerShell. If you are interested, check-out these two PowerShell commands: Invoke-Command and New-PSSession.

PowerShell

Introduction

PowerShell has been around for over seven years now. While the basics are easy, the more useful functionality is not easy to learn. Therefore, I came to my own PowerShell self-discovery several months back and since then have used it for all kinds of tasks: storing my passwords, managing my personal computers to reduce my use of the mouse, configuring remote computers, provisioning cloud infrastructures (on AWS and Azure), writing deployment scripts, and coding my own PowerShell command line tools.

So what is PowerShell?

It is a scripting language that runs inside a shell on the Microsoft platform. Significantly, it differs from DOS and scripting languages on other platforms in that it is .Net based, NOT text-based. In other words, in other popular shells, all input and output is in the form of text. This means the text must be manipulated in order to be of use as the input to commands or the output from commands. In PowerShell, while text (strings) are certainly supported, input and output is represented as .Net objects. This means that input and output does not require text manipulation and the user inherits all of the power of objects.

A couple more definitions:

In general, a scripting language is best suited for writing small blocks of code. It is high level, meaning it greatly abstracts the underlying hardware, making scripting good for rapidly writing functional programs to perform very specific tasks, sometimes at the expense of performance or flexibility.

Advanced Aside: Note that PowerShell is Turing Compete.

A shell, or command line interface, is an interpreter that hosts a scripting language. In other words, rather than compiling code into binary, a shell accepts scripts on the fly and executes them into meaningful actions.

Microsoft decided to call PowerShell commands cmdlets (pronounced command-lets).  Importantly, by convention, cmdlets are verb-noun pairs.  Actual examples include:

  • Get-ChildItem
  • Set-Item
  • Invoke-Command
  • Out-Host

Starting PowerShell

Modern Windows ships with two PowerShell shells, which Microsoft calls “Hosts.”  However, vendors are able to provide additional shells:

  • PowerShell: The default host
  • PowerISE (Integrated Scripting environments): A multi-tab environment for writing scripts and performing basic debugging with break points (NOTE: scripts can actually be written in any text editor as well as the standard PowerShell host).

The easiest way to launch PowerShell is just to start it from your Windows Programs menu.

The Three Key Commands

The following three cmdlets are all you need to know to get started using PowerShell:

  1. Get-Help (or just type ‘help’): Provides basic and extended help and syntax for any given PowerShell cmdlet.  For example, at the PowerShell prompt, type: Get-Help Get-Help

    Now trying typing this: Get-Help Get-Help -Full Notice this time PowerShell includes examples in addition to the basic usage.

    Advanced Aside: adding -Full is an example of something PowerShell calls a switch.

    NOTE: the first time you run help you may be promoted to update the help content.  This is because PowerShell is able to reach out over the Internet to download help content.  In fact, the clean install may not even ship with the help files installed until you run this download for the first time.

  2. Get-Member: Lists the methods and properties available on an object. Remember, PowerShell input and output are .Net objects, not text. Therefore, the input and output have methods and properties, just like all .Net objects do. For example, type: 7 | Get-Member. PowerShell tells you that “7” is an int (System.Int32) and lists the methods a .Net developer would expect to see on ints, such as ToString and GetType.

    Advanced Aside: the first part of this command, “7 | ” is called a pipe and without getting to detailed tells PowerShell to apply the Get-Member command to the number 7.

  3. Get-Command: Lists every cmdlet and function currently available to call in the PowerShell host (for now, ignore the difference between a function and cmdlet and think of them as the same). You can also narrow down the available cmdlets to list. For example, if you think the verb in the verb-noun cmdlet pair starts with “Get” you can type Get-Command get-*

    Note that PowerShell is case insensitive, “get-*” is the same as “Get-*” or “GET-*”

    Advanced Aside: I was very careful to state that Get-Command by itself lists the cmdlets PowerShell currently knows about. That is because it is possible to add more cmdlets into the PowerShell host’s scope.

Wildcards

As I demonstrated above, PowerShell also accepts wildcards for parameter values and as part of its Intellisense, when you are selecting cmdlets in the host or ISE. Type for example Get-H* and then press <tab>. Watch how PowerShell cycles through the list of available cmdlets each time you press <tab>.

Advanced Aside: There are other ways of doing this too. My first instinct was to pipe the result of Get-Process to a where clause. But I haven’t shown where clauses in this post. Fortunately, there are many ways to do the same thing in PowerShell, though they may not all have the same performance characteristics if you are running long running scripts, especially on remote computers.

Pipes and More

By way of demonstrating another important feature, try typing this: Get-Service | More into your PowerShell host (note it may not work in the ISE). This lists all Windows services in your computer but only displays enough to fill the current screen (so you don’t need the scroll bar). To continue through the list, hit the
<space> bar. The | is called the pipe and applies the output of the stuff to the left of the pipe as input to the stuff to the right of the pipe. I showed another example of this earlier when I did this 7 | Get-Member.

Cmdlets vs Scripting

In this post I have only demonstrated typing simple cmdlets or functions on the host. But PowerShell provides a rich syntax for writing scripts with standard programming constructs like loops, decisions, Boolean logic, etc.

Basic Cmdlet Example

I want to end this post by showing how you might use your newly acquired skills to list all running processes on your computer that start with the letter “S.” Let’s say you suspect their might be an out of the box PowerShell cmdlet for this. You already know that all cmdlets follow a verb-noun pair. So you might think the cmdlet contains the word “process.” Therefore, type this in your PowerShell host: *process* and press <tab>. Keep tabbing through…. ah, Get-Process looks promising. Let’s see what it does. Type Get-Help Get-Process. Looks good! The resulting help even shows a syntax with the parameter – Name. And you should be thinking to yourself, “Adam said I can include wildcards in my parameter values.” So now you type Get-Process -Name "S*" and watch the magic happen!

Stay Tuned:

So I’ve come to the end of this, the first blog post I’ve ever written. As time rolls on, I expect to expound on more features of PowerShell. If you’re really desperate to learn it in depth, now, however, then I strongly recommend this book: Windows PowerShell Cookbook.