PowerShell Bits And Pieces

I recently came across a few interesting things in PowerShell and I’m writing it down for future me or anybody else that might find this useful.

Importing CSV files
If you have a CSV file with proper headings in the first row you can import it using the Import-CSV command and you will end up with an array of objects that contain properties matching the headings. For example:

Name, IP, Role
 Srvr1,10.0.0.1,Data
 Srvr2,10.0.0.2,Web

Will give you an array with the following:

CsvArray

Using Visual Studio Code as your PowerShell IDE
Turns out VS code is actually very handy when editing PowerShell, you get intellisense, debug, validation, watches and key bindings that closely match Visual Studio to name but a few features. I used Visual Studio before VS Code as my IDE and although it worked very well, VS Code is lightweight and less cluttered and only takes few seconds to load even on a low end Windows tablet. You have to download the PowerShell plugin and enable but luckily that can now be done from the GUI instead of the command line.

InstallPS

You have to do a bit of work to get debugging to work, click the debug icon, then the gear icon to edit the launch.json file. For every script you want to debug you have to create a new entry and specify the name of the PowerShell file and a meaningful name to display in the drop down list.

VSDebug

Break and ForEach loops
Just like C# you can use Continue to go to the next iteration of a loop and Break to exit the loop but the behaviour is different depending on where you use it. Break will exit the current loop if used in a Foreach, For, While, or Do loop and it will exit the current code block if used in a switch statement. Using Break outside a loop or switch statement will exit the script even if it is used inside a code block not the main script. When you use Break inside a ForEach command Break behaves as expected and will exit the immediate loop but if you are using the ForEach-Object cmdlet Break will not exit the loop it will exit the script.

PowerShell without remoting
There are a few PowerShell commands that you can run remotely without using PowerShell remoting, PowerShell must be installed on the remote computer but PowerShell Remoting doesn’t have to be enabled or configured. You can find the complete list is here. I found this feature very useful to execute WMI queries for inventory purposes.

Working With Regex
Regex behaves like it did in .NET you use the Select-String cmdlet and it returns a MatchInfo object for each match. For example to find lines in a file that start with a UTC date.

$match = Get-Content c:\temp\logfile.txt | Select-string -Pattern '^(\d\d\d\d)-(\d\d)-(\d\d) (\d\d):(\d\d):(\d\d).(\d\d\d)Z' | Select-Object Line

The Line property will show the whole line in the file that matched while the  Matches property shows more information around the pattern that was matched. You can also use Select-String to process multiple files in a directory by specifying a filter and the MatchInfo object for each match will contain information about the file, line no etc where it was matched.

Direct Output In Two Directions
You can use Tee-Object command to redirect output to a file or variable and still pass it to the pipeline as well. For example to keep the output of a process in a log file but still see what is happening in the console and even process the output further down the pipeline:

Start-Process ... | Tee-Object C:\Output.log | ... more processing ...

Francois Delport

How To Assign A Public Static IP Address In Azure Using Azure Resource Manager

In this post I’m going to show you how to assign a public static IP address in Azure using Azure Resource Manager, if you are using the classic deployment model use this link.

Contrasting Classic Deployment With ARM
In ARM you don’t have a cloud service that is the container for your VM, load balancer, public IP and other functionality any more. The different components that make up your environment is clearly separated and you compose your environment by stitching together the pieces you need. For example, you can create a network interface card on it’s own, it doesn’t have to be part of a VM and you can move it between VMs or attach it to a load balancer etc. Same goes for your public IP addresses, load balancers, network groups etc. That said you can create everything you need for your environment in the same ARM template giving you that feeling of cohesion.

Creating A Public IP address Using The Portal
When you create a new VM in the portal the default setting is to create a new dynamic public IP address along with the VM. You can change it to be static or use an existing public IP address if need be or you can choose None if you don’t want your VM to be directly accessible from the internet.

Nic

Note that even if you choose to have a public IP address for your VM it won’t have a DNS name, at the time of writing you can’t assign a DNS name when you create the VM in the portal, you have to do it  afterwards, instructions here.

Creating a public IP address using ARM templates

{
  "apiVersion": "2015-06-15",
  "type": "Microsoft.Network/publicIPAddresses",
  "name": "mynewpublicip",
  "location": "[resourceGroup().location]",
  "properties": {
  "publicIPAllocationMethod": "static", //or dynamic
    "dnsSettings": {
    "domainNameLabel": "testing8423"
    }
  }
}

The DNS settings are optional and will create a DNS entry with your label and the name of the Azure region for example: testing8423.westus.cloudapp.azure.com. On your NIC template you reference the public IP address.

...
 "name": "defaultNICName",
  "dependsOn": [
 "Microsoft.Network/publicIPAddresses/mynewpublicip",
 "Microsoft.Network/virtualNetworks/virtualNetworkName"],
 "properties": {
 "ipConfigurations": [
 {
  "name": "DefaultNicIpconfig",
  "properties": {
  "privateIPAllocationMethod": "Dynamic",
  "publicIPAddress": {
   "id":"[resourceId(Microsoft.Network/publicIPAddresses       /mynewpublicip)]"
 },...

Note: I removed some of the JSON elements for sake of readability.

Creating a public IP address using PowerShell
I didn’t try it myself since I’ve been using ARM Templates of late but it looks easy enough. You create a new public IP address with New-AzureRmPublicIpAddress and then you create a new NIC and assign the IP address to it using New-AzureRmNetworkInterface and the -PublicIpAddress or -PublicIpAddressID parameter.

New-AzureRmPublicIpAddress -Name IPName -AllocationMethod Static/Dynamic -ResourceGroupName ... 

New-AzureRmNetworkInterface-PublicIpAddress<PSPublicIpAddress> -ResourceGroupName... -IpConfiguration ... -Location ... -Name ...

White Listing Public IPs
When you create a new public IP address you get one from the block of IP addresses Microsoft owns, so you can’t choose a specific one. If you have a requirement to white list specific public IP addresses for the long term you’ll have to create the public static IP upfront and retrieve the allocated IP address.  You can assign the public IPs to NICs as required. You can white list a range of Azure IP addresses downloaded from here but that isn’t always practical.

Francois Delport

Continuous Deployment Of Databases : Part 4 – Flyway

This is part 4 in my series on continuous deployment of databases. In this post I’ll be having a quick look at FlyWay.

Part 1 covers the logic behind the deployment options.
Part 2 covers SqlPackage.exe.
Part 3 covers ReadyRoll.
VSTS Sql Server Database Deployment Task can be found here.

In my two previous posts I briefly had a look at two tools for SQL Server Continuous Deployment, SqlPackage.exe and ReadyRoll. I wanted to see what is out there for Oracle, MySQL and other databases especially if you have a situation where your application supports multiple database and you want to do Continuous Deployment against all of them. There are plenty commercial applications for SQL and Oracle that integrate directly with build systems and IDEs but not that much for other databases based on my quick google search.

Flyway
I’m going to take a quick look at Flyway, a migration based tool, bearing in mind I work in .NET and some of the features are Java centric. It is an open source project and looking at the commit history on GitHub it has been around for a long time and is supported by an active development community. At the time of writing it supported 18 databases.

You create migrations using SQL script files or Java code based migrations which comes in handy for very complex data manipulation and handling blobs. It is based on convention over configuration so naming folders and files correctly is very important since the version number which controls the deployment sequence is part of the name. There are various ways to execute the migrations for example using the command line tool or calling the Java API directly, this opens up scenarios like including Flyway as part of your application to keep your database and application in sync. There are also plugins for Maven, Gradle and Ant but not much in terms of .NET.

I’ll quickly touch on the main operations used by the application:
Migrate: The meat of the application, it migrates a schema to the required version by determining the migrations that should be run.
Clean: Wipes the destination database clean, used during development and testing.
Info: Shows the migrations applied with execution time and state and which ones are pending.
Validate: Verify the applied migrations match the ones you have locally.
Baseline: Integrate Flyway with an existing database so only newer migrations will be applied.
Repair: Flyway uses a metadata table to keep track of applied migrations, this will the repair the metadata table.

Tooling
It doesn’t have direct integration with Visual Studio or VSTS but it doesn’t look to difficult to roll your own automated deployments with some scripting to call the command line tool. Similar to the SqlPackage.exe you can call the Flyway command line tool from your release management tool to update the target database directly or generate a report of the changes that will be applied in case a DBA has to scrutinise the changes first.

Oracle Side Note
If you want to use Flyway against Oracle or just want to use Visual Studio in general for Oracle database development have a look at Oracle Developer Tools for Visual Studio. Oracle tools is very similar to Sql Server Data Tools and makes is easy to add your Oracle database to source control. You can also use it to compare Oracle schemas and generate change scripts which can be the base for your Flyway migration scripts.

Francois Delport