Tuesday, April 24, 2012

Building a Build Process, Part 6: Creating a Custom MSBuild File

This is part of a larger series in building a proper build process. For more information and additional series links, check out the introductory post.


Ed. Note: I cannot give enough praise to the Pluralsight.com Continuous Integration course; it helped me put a lot of these pieces together, and this part in the series could not have happened without it. A lot of the content for this post ended up being a pretty direct port of what is talked about there, but there are only so many ways to personalize a best practice while keeping it simple, and so I hope you’ll sign up for a Pluralsight subscription, and that their lawyers will take kindly to this series. :)


So, now that we have our initial project structure created and under source control, we have to find a way to build it. “Oh, I know!” you say. “We’ll hit F5!” well, while that might work for local development, your build server can’t hit F5.

Lucky for us, the process of hitting F5 and turning your code into a DLL isn’t all magic. In fact, it’s a highly documented tool called MSBuild, by the Microsoft team, that ships with the .NET Framework.

In today’s installment, we’re going to do the following:

  • Add MSBuild to the Path line and test it using Powershell
  • Create an XML file that will build our application using MSBuild Tasks
  • Examine variables in MSBuild XML files
  • Create tasks for cleaning the build, initializing the build, and compiling the build
  • Put it all together to create build targets.
  • Use PowerShell to execute those build targets.

Find the Location of MSBuild on Your Machine

The good news is, if you have the .NET Framework, you have it. If you have the .NET 4.0 Framework (and really, you should), you should be able to find it in %WinDir%\Microsoft.NET\Framework\v4.0.30319\.

Add MSBuild to the System Path

  • Click Start
  • Right-click “Computer” and select Properties
  • Click “Advanced System Settings”
  • Click “Environment Variables”
  • In the second section, System Variables, look for the PATH variable, and click edit.
  • Ensure that a semi-colon is after the last entry, and then past in the path to your MSBuild.exe, leaving off a trailing slash (e.g. mine was C:\Windows\Microsoft.NET\Framework\v4.0.30319” without quotes.)

Run PowerShell and Test Access to MSBuild

  • Click the start menu, begin to type “PowerShell”, and bring up the PowerShell console by right-clicking and choosing “Run as Administrator”. You must do this in order to run MSBuild the way it is necessary to run it. (you may also want to pin the icon to your taskbar at this point).
  • When the console is open, type MSBuild.
  • You should see an error about not specifying a project or solution file. If you see any red text, such as “command not found”, etc., then something went wrong and powershell can’t see your MSBuild location. (try running as admin or checking your path variable setup).

Create a New, Empty .build XML File

  • Open the TestProject solution in Visual Studio
  • Right-click on the solution and select Add –> New Item.
  • Select XML File as the type, and name it TestProject.build
Now you’ll have a .build file at the root of your solution. This is what we want.

Adding the Schema for an MSBuild XML file

After the first line of the XML file, you’ll need to add the root node and give it the schema reference for its XML namespace (xmlns). After you do this, it should look like the following:

Note that I also added a ToolsVersion attribute, with “4.0” denoting the version of the .NET Framework we’re using.

Add Some Properties

Instead of going back and refactoring after showing you a full examples, I’m going to save some precious keystrokes and tell you that at some point, it will be easier and more flexible to use variables. To do this:
  • Create an <ItemGroup> </ItemGroup> node within the Project section.
  • The format under this will essentially be: <PropertyName Include=”PropertyTextToInclude”/>
  • Add references to your BuildArtifacts folder, and also to your solution file, like so:


The Basics: Cleaning and Initializing Our Directory

Next, we create two targets. One, “Clean”, will delete the .\buildartifacts folder. The second, “Init”, will recreate it. Pretty simple. The XML to accomplish this is below:

You’ll notice that I have the “Init” target depend on the “Clean” target via DependsOnTargets. This means that any time we create the folder, we’ll first delete it to make sure we’re starting fresh.
Also notice that instead of putting a path to the buildartifacts folder directly, I’m using @(BuildArtifacts), which tells MSBuild to refer to the ItemGroup variable we created earlier.

Trying our “Clean” and “Init” Targets in PowerShell

  • Open PowerShell in Admin mode
  • Navigate to the Solution folder (e.g. cd \Users\Sean\Projects\TestProject for me)
  • Also open this folder in Windows Explorer. See the buildartifacts folder there?
  • In PowerShell, run: MSBuild TestProject.build /Target:Clean
  • Note that in Windows Explorer, the buildartifacts folder is gone.
  • In PowerShell, run: MSBuild TestProject.build /Target:Init
  • Note that the folder has reappeared. Add a file to the folder – a text file or something small.
  • In PowerShell, run: MSBuild TestProject.build /Target:Init
  • Note that the folder has been deleted and recreated, and thus no longer contains the item you put there.

Getting to the Good Stuff: Compiling our App

Up until this point, we haven’t compiled our code. Since that’s what gets us paid, in a manner of speaking, we should create an MSBuild task to compile the code. We can do this via the following additional target:

This calls the MSBuild exe from within MSBuild (I know…whoa, dude), passes it the solution, and compiles the program by passing it in our solution file variable. It also specifies that the Output Directory should be buildartifacts. Tip: See the .FullPath? Our variables are also objects, and so MSBuild will know that %(BuildArtifacts) is an object and thus will pick up on the FullPath property of it.
Note that we’ve made “Compile” dependent on “Init”, so that everytime we compile, the buildartifacts folder will be blown away and re-created.
Try it out: In PowerShell, run “MSBuild TestProject.build /Target:Compile” and watch our solution be compiled to the buildartifacts directory. Pretty sweet, huh?

Telling MSBuild What to do by Default

Passing a Target every time is pretty lame, especially when we usually just want to compile the code.
Luckily, by adding the DefaultTarget attribute to the Project node, we can tell MSBuild what to compile by default. Let’s try that now. Modify the Project node XML to make the default target “Compile”, like so:

Save the file, and run "MSBuild TestProject.build” (without a target attribute). The project should compile.

Next Time…

In the next article, we’ll explore how to start and stop the Cassini Web Server asynchronously, and how to run Visual Studio builds through common output directories.

 

Feedback Welcome!

I'd love to hear any comments on this series. Find it useful? Think there's a better way to implement the technique or something I should have mentioned? Please drop a line in the comments to help me improve the series!

References


<— Part 5: TortoiseSVN Client Connection and Repository Layout

Monday, April 23, 2012

Quick Tip: Need a User to Take Screenshots of a Problem? Try Using Win7's Built-in Tool [Tips & Tricks]


I can't believe I hadn't heard of this before. There’s a neat tool that ships with Windows 7 called the “Problem Steps Recorder” tool (psr.exe)


The Process Works Like This:
  • A user clicks the record button
  • The user performs all the steps to recreate the problem (and can add comments at each step)
  • The user clicks stop recording, and is prompted to save a ZIP file, which they can then mail to a technician.
  • The ZIP file contains an MHT file.This file contains a screenshot of every step the user performs, as well as system information regarding those steps. You can look through them one-by-one or play them as a slide show. 

To Access the Tool:
  • From a windows 7 machine, in the Start menu, type “psr” and click PSR.exe. NOTE: if you have applications running in Administrator mode, you will need to run PSR.exe in administrator mode in order to be allowed to capture those.


Quick Tip: Maximo 7 -- Location of Workorder Status Information [Field Notes]

A quick note for myself an anyone else who might be interested:

Maximo 7 has a few internal status codes, but custom statuses can easily be created. However, these have to map to one of the internal status codes.

I wanted to give our customers an overview of the internal status codes and their mappings. After some searching, I found the following method successful:

  • Login to Maximo 7 
  • Click Go To --> System Configuration --> Platform Configuration --> Domains
  • Search for the "WOSTATUS" domain and expand it.
Here, you'll have a list of all statuses with internal and external values.

Hope this helps!

Sunday, April 22, 2012

Building a Build Process, Part 5: TortoiseSVN Client Connection and Repository Layout

This is part of a larger series in building a proper build process. For more information and additional series links, check out the introductory post.
Welcome back! Now that we have a TLS-encrypted Apache setup with SVN, we’re going to take a look at creating the repository layout.

NOTE: Before I begin, I should mention that while I will speak about them authoritatively, the methods for laying out repository structures are by no means set in stone, and are in fact debated quite vigorously at times in the tech community. This is the general flavor that I’ve picked up in some places, including an excellent Pluralsight course on Continuous Integration by James Kovacs. It’s the style that I believe I’ll adopt going forward, though don’t hold me to it.

Obtaining TortoiseSVN

TortoiseSVN is pretty much the de-facto standard for Subversion clients on Windows. The latest version as of this writing is 1.7.6. You can download the 32-bit version or 64-bit version from their downloads page.
Installing Tortoise is about as standard as it gets. Run the installer, accept the license agreement, and select the options for install.
During this time, I usually right-click “command line client tools” and install it by choosing will be installed on local hard drive. You never know when they’ll come in handy (though I don’t intend to refer to them in this series).
After this, the app installs. NOTE: If you’ve got some cash, I highly recommend you donate to the TortoiseSVN project. It’s a fundamental piece of software for developers and good software deserves our support.

Linking a Local Folder to the Repository

Our first step is to pull down the repository we created (which is currently blank, but nevermind that):
  • Create a folder somewhere on your hard drive where you would store repository data (for example, mine is in C:\Users\Sean\Repositories, and I have a Win7 Library configured to point there).
  • Navigate into that folder.
  • Create a new folder called “TestProject”, named after the repository we created on the Subversion CentOS VM.
  • Right-Click on the TestProject Folder. Notice that there’s some new options now; this is where TortoiseSVN lives – in your context menu.
  • Select “SVN Checkout”. This tells Tortoise to attempt to pull down a repository into the folder you have selected.
  • For the URL of the repository, type https://[ip or hostname of your svn server]/svn/TestProjectRepo. Note the https; we’re going to pull this repository down over a TLS-encrypted connection.
  • Double-check that the checkout directory is the new folder that you created on your local machine called TestProject, and then Click OK.
  • At this point, you will likely receive a message about the fact that certificate failed. Click “Accept the certificate permanently”, since we know the certificate is trustworth (we created it, after all)
  • Next, you’ll be prompted for username and password. Enter one of the two logins we created when we initially set up Subversion. You may want to click “save authentication”; otherwise you’ll be prompted for it whenever you communicate with the repository.
You will see through the TortoiseSVN dialog box that the process is completed. The folder may also have a green check-mark overlaid. This is TortoiseSVN’s handiwork; it lets you know that a repository is up to date.

Creating the Initial Repository Layout

Enter the TestProject folder on your local machine. We’re going to create three empty folders under this directory that have a very specific meaning. Create the following folders:
  • trunk
  • branches
  • tags
This is one of the standard layouts for a repository in Subversion, and the one I tend to use. The subversion concepts related to these folders are something like the following:
  • Trunk: This where the main portion of development takes place. It is sort of the “master timeline” of real-time development of your application source code. More often than not, your working copy (more on this soon) will be the trunk.
  • Branches: With subversion, you can create branches. Think of it like creating an alternate reality of your code at a specific point in time. You can take your whole system as it is, and “branch it” to either fix bugs for a specific point in time, or branch it to implement a certain feature based on the code at that point. Branching can be kind of a pain in Subversion, as you then have to merge your changes back into the trunk, so you tend to do a lot of branching and merging to ensure things stay in sync. For this reason, I avoid it unless it’s necessary. I do however, like the idea of branching when code is released, so that you can fix bugs for that release and then merge it back into production.
  • Tags: Think of tags like a read-only branch, or a copy of your code frozen at a specific point in time. This is actually pretty cool. It allows you to do things like tag every major release, so that at any point your developers can pull it back up and compile it to check it out. Imagine a big client comes to you and they have a bug in a legacy version of the application you put out. You can’t force them to upgrade. How do you figure out what the code was at that point in time? Simple; just pull down the tag folder and it’s there.
Normally, you won’t be looking at all three of these at once. You’ll want to create what’s called a working copy of the source that is connected either to the trunk, or a specific branch. Think of your working copy as your lens into the source code. We’ll be creating the working copy soon, but first we have to commit this initial structure.

A Word on the Update / Commit Cycle

A big idea behind subversion is that instead of locking files so only one person can use them, it allows all people to edit all files in the repository (assuming they don’t have read-only access). This is usually vastly more efficient, but it does come with some caveats. If two people edit the same file, it will create a conflict, and that conflict will have to be resolved by a differential view and resolution (“diff”) of the two versions of a file with another developer. This isn’t usually a bad thing, but it becomes cumbersome if the files haven’t been checked in for a long time.
To avoid this, I usually stick to the following principles:
  • Before you begin coding, update the repository by right-clicking on the project folder and choosing “SVN Update”. This will get the latest copy of the source code.
  • As you work on the code, update once in a while in case someone has made changes that need to be merged with your file. Little changes made more often means smaller, more manageable conflicts.
  • Update once before you check your code in to ensure that no conflicts will arise.
  • Check in your code several times a day (if you’ve added something complete and useful that works.) this will ensure that your changes are smaller for other developers (more on the check-in process below.)
  • Always add a message when you check in and describe your changes in enough detail so that others can understand them. No novels are necessary, but it’s important to know in human-readable form who changed what, when, and why.

Committing Our First Changes

So, we added three folders to the repository. As small of a change as it is, it is a change, and so we need to check those changes in so Subversion can pass them along to anyone else. To do this:
  • Right-click on the TestProject folder and select “SVN Commit…”
  • Enter a message into the message box that makes sense (e.g. “Added initial folder structure”.)
  • Note that since we added new items, they’re not currently under source control. Subversion is smart enough to not add new items unless you tell it to.
    • If you don’t see any files, make sure the “Show unversioned files” checkbox is checked.
  • Click the “All” button and subversion will select all items to commit them to the repository.
  • Click OK and the folders will be committed to the repository. You see that it added each folder, and that the repository is now at revision 1.

Creating the Working Copy

Now that we have the files committed to the repository, we can delete the local folder because we know the files are committed to the repository and we can pull them down again later. That’s exactly what we’re going to do, but this time, we’re going to create our working copy by pulling down only the trunk.
  • Delete the TestProject folder in windows explorer.
  • Create a new folder called TestProject.
  • Right-click the TestProject folder and select “SVN Checkout…” from the menu.
  • Make the URL of the checkout path “https://[ip or hostname of svn server]/svn/TestProjectRepo/trunk” and click “OK”.
The trunk will be pulled down, and you’ll see it’s at revision one. Now, even though “branches” and “tags” exist in the universe, the world we know will consist of the trunk folder. If we ever need to switch to a branch, we can do this through the “Switch” statement (but we have no use for that now).

Creating the Project Folder Structure

After watching the Pluralsight course on Continuous Integration, I really like the layout that was chosen for the project. We’re going to create a folder structure, and then I’ll explain why we created it like we did.
Create some new directories so the folder structure looks like the following:
  • TestProject (already exists)
    • \buildartifacts
    • \src
      • \app
      • \test
    • \thirdparty
      • \libs
      • \tools
    • \doc
These folders have the following purposes:
  • TestProject (root folder): This is where your solution file will be. Later in this series, it will also be where we place the MSBuild XML file that applies to our software.
  • buildartifacts: This is the place where we’re eventually going to put all of the code that we’ve compiled, automatically, via the MSBuild script. This folder should be left out of source control because everyone’s binaries will be different and not of much use except to the developer (more on that later).
  • src: as you probably guessed, this is where all our actual source code is going to go.
  • app: this will store all the application source code (your individual projects will eventually be in this directory)
  • test: this is where all your unit testing and integration testing projects will live. This will let us do some handy things later as far as packaging up the project for release, etc.
  • thirdparty: this where code goes that isn’t yours. Could be something you referenced, or an open-source project that you make use of, etc.
  • libs: this is specifically for third-party libraries that you use within your source code.
  • tools: this is for DLLs or applications that help you with the build process or things external to your source code. Think NUnit for running unit tests, or special add-ons to the MSBuild DLLs.
  • doc: this is where all your documentation for your projects should live, if you need to reference it. It could be internal documentation if you choose to keep it this way, a user manual that evolves along with the project, a dictionary of acronyms or documentation on processes for your developers or business use cases or marketing materials. Any documentation that is based on the code in the src folder that evolves along with it should be included here.
At this point, let’s update the solution by right-clicking TestProject and selecting SVN Update. Since no other changes to our files were detected, the update completes successfully. We then commit the changes by right-clicking on TestProject and choosing SVN Commit, following the same process as we did earlier. (did you remember to add a commit message?)

Creating the Solution in our Project’s Root Directory

After all our hard work to set up the solution right, we’re finally ready to create the solution within Visual Studio.
First, we’ll likely want to enable the Always Show Solution option within Visual Studio so that it’s easier to create a blank solution – which is what we’ll be doing first.
Next, complete the following steps:
  • Open Visual Studio 2010 (or version of your choice)
  • Select “New Project”.
  • In the New Project folder, expand the “Other Project Types” folder and click the “Visual Studio Solutions” category.
  • Name the solution TestProject.
  • Since a folder is always created for a solution, but we have a folder already, we’re going to choose the location as the folder above the TestProject folder. (for example, my TestProject folder is in C:\Users\Sean\Projects\TestProject, so I select C:\Users\Sean\Projects.) This keeps us from having to copy/paste the solution to the right spot manually later.
  • The new solution is created and you see it within Visual Studio.

Adding Other Projects Under the src Directory

  • Right-click on the newly-created solution in Visual Studio and select Add –> New Project.
  • Add a C# class library called TestProject.Core, in the location of TestProject\src\app (the folder structure that we’d created previously). This will be the project that holds all of your application’s shared core logic (in case you need to share code common to both a web site and a WPF app, etc).
  • Delete the class1.cs file from the Core Project (we won’t need it).
  • Right-click on the solution and add another C# class project, TestProject.Core.Tests (this will hold the tests for the core business code). Give the location of this project TestProject\src\test. Delete its class1.cs file as well.
  • Repeat this process for two more projects, An MVC3 Web App called “TestProject.Web” and a class library called “TestProject.Web.Tests”. Can you figure out the right directories to put them in?
At this point, we’ve got a full project setup.

Committing our Chan0ges – and Ignoring a Directory

We’re ready to commit our changes – or are we?
I’d mentioned earlier that the “buildartifacts” directory doesn’t really make sense to commit, as everyone will have their own, and the files are highly likely to cause conflicts as they are binary files. It would be useless to attempt to try to do much with them outside the contest of our personal development, so we’re going to have subversion ignore the buildartifacts folder, ensuring that it won’t add that information to the repository ever.

  • In Windows Explorer, right-click on the buildartifacts folder.
  • Select TortoiseSVN –> Unversion and add to ignore list –> buildartifacts. Subversion will now ignore the whole folder.
With this done, we’re ready to commit all our changes.
  • Right-click on the TestProject folder and select …if you guessed commit, you’re wrong! Remember, we should update the solution first (it’s a good habit to get into). Select Update.
  • With no conflicts detected, select “Commit” and click “all” to select all the new unversioned files.
  • Hit OK and the files will be committed to the repository.
Congratulations! You’ve fully completed a solution setup with Visual Studio and Subversion.

Feedback Welcome!

I'd love to hear any comments on this series. Find it useful? Think there's a better way to implement the technique or something I should have mentioned? Please drop a line in the comments to help me improve the series!
 

References


<— Part 4b: Securing Subversion's Connection via TLS

Building a Build Process, Part 4b: Securing Subversion's Connection via TLS

This is part of a larger series in building a proper build process. For more information and additional series links, check out the introductory post.

Alright, I just couldn’t take the guilt. I can’t abide Subversion and Apache being set up without defaulting to https and a TLS encrypted connection. We’ll remedy this below.

Install The Required Packages

On the CentOS server, login as root and type the following in the terminal to install packages:
  • yum install mod_ssl openssl
These packages may already be installed; don’t worry if yum tells you that it has nothing to do.

Use OpenSSL to Generate a Self-Signed Certificate

Navigate to the certificate store by heading to:
  • cd /etc/pki/ca
Next, we’ll need to create a strong private key (2048-bit encryption):
  • openssl genrsa -out ca.key 2048
Then, we generate a CSR (Certificate Signing Request):
  • openssl req -new -key ca.key -out ca.csr
You’ll then have to enter the following fields:
  • 2-digit country code (e.g. US)
  • State or Province full name (e.g. Maryland)
  • Locality / City (e.g. Baltimore)
  • Organization Name (e.g. Microsoft or Sean Killeen)
  • Org Unit Name / Section (e.g. Subversion Test)
  • Common Name (your new hostname, eg scm.seankilleen)
  • Email Address (e.g. SeanKilleen@gmail.com)
  • A challenge password
  • An optional company name
The next step is to use the CSR to generate the actual certificate:
  • openssl x509 –req –days 365 –in ca.csr –signkey ca.key –out ca.crt
Then, copy the certificate files into the TLS certs location that we’ll use later:
  • cp ca.crt /etc/pki/tls/certs
  • cp ca.key /etc/pki/tls/private/ca.key
  • cp ca.cr /etc/pli/tls/private/ca.csr

Modify Apache’s Configuration to offer SSL

Open the file for editing:
  • gedit +/SSLCertificateFile /etc/httpd/conf.d/ssl.conf
Find the line referencing “SSLCertificateFile” and change it to the location of your keyfile. It should look like this when you’re done:
SSLCertificateFile /etc/pki/tls/certs/ca.crt
A few lines after that, you’ll edit the SSLCertificateKeyFile:
SSLCertificateKeyFile /etc/pki/tls/private/ca.key
Save the file and close it.
Next, restart apache – from the console:
  • /etc/init.d/httpd restart
Redirect Non-SSL connections to the SSL Connection
To do this, we need to edit the apache configuration file.
  • gedit /etc/httpd/conf/httpd.conf
Towards the bottom of the file, the <VirtualHosts> configuration can be found.
Paste the following lines at the bottom of the file:
NameVirtualHost *:80
NameVirtualHost *:443

<VirtualHost *:80>
        <Directory /var/www>
        AllowOverride All
        </Directory>
        DocumentRoot /var/www
        ServerName localhost
</VirtualHost>
<VirtualHost *:443>
        SSLEngine on
        SSLCertificateFile /etc/pki/tls/certs/ca.crt
        SSLCertificateKeyFile /etc/pki/tls/private/ca.key
        <Directory /var/www>
        AllowOverride All
        </Directory>
        DocumentRoot /var/www
        ServerName localhost
</VirtualHost>
Now point your browser to https://[your ip or host name] and you’ll see that the site loads under an http connection. NOTE: you may get an error about the certificate, but this is because it does not come from a CA and thus is not “trusted” by your computer. The encryption is still TLS 1.0 256-bit encryption.

Enforce SSL/TLS Only (Redirect http to https)

I put this in a separate section because I wanted the additions to be compartmentalized.
To use mod_rewrite to redirect any http requests to https, change the VirtualHost *:80 to the following:
<VirtualHost *:80>
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
    <Directory /var/www>
        AllowOverride All
        </Directory>
        DocumentRoot /var/www
        ServerName localhost
</VirtualHost>

Configure the Firewall to Allow Port 443 Connections

On the console:
  • iptables –A INPUT –p tcp –dport 443 –j ACCEPT
  • /sbin/service iptables save
  • iptables –L –v
Now, try heading to http:[your ip or hostname]/svn, and watch it redirect to https.
Success!


Feedback Welcome!

I'd love to hear any comments on this series. Find it useful? Think there's a better way to implement the technique or something I should have mentioned? Please drop a line in the comments to help me improve the series!


References