Tuesday, February 12, 2008

PowerShell has been a topic of interest for me for the last couple of weeks. I have been squeezing in some reading on it here and there for about a little over a week. Out of the blue pops up one of those issues at work that does not involve making an elegant object model, but rather digging through log files. How cool is that? For more years than I care to mention, it seems like every time I start looking into a new topic, something comes up where that research saves a ton of time. So there I am whipping up this script, when SharpReader pops up toast that says Using an IDE to write PowerShell Scripts. If you are doing any PowerShell work at all, go ahead, stop reading this post and go get that tool. The post will be here when you get back. The PowerShell Suite has been discounted for 2 weeks associated with Hanselman's post and there is a free version for non-commercial use.

The issue was that users in Germany were getting access denied during a particular time frame. Since this time frame overlapped with with the nightly import of user information, there was some speculation that there was a collision with the import. We wanted to get a list of users that got the access denied and correlate that with the users that were imported. That leaves us with 500MB for 2 hours on each server. 8 web servers and a 4 hour period in question yields 16GB worth of log files to mine. Definitely an automation task.

The log file looks like:

... cut
[8364/18416][Tue Feb 07 04:30:27 2008][..\..\..\CSmHttpPlugin.cpp:402][INFO:1] PLUGIN: ProcessResource - Resolved Url '/security/accessdenied.aspx?ReturnUrl=index.aspx'.
[8364/18416][Tue Feb 07 04:30:27 2008][..\..\..\CSmHttpPlugin.cpp:515][INFO:1] PLUGIN: ProcessResource - Resolved Method 'GET'.
[8364/18416][Tue Feb 07 04:30:27 2008][..\..\..\CSmHttpPlugin.cpp:566][INFO:2] PLUGIN: ProcessResource - Resolved cookie domain '.SITENAMEWASHERE.com'.
[8364/18416][Tue Feb 07 04:30:27 2008][..\..\..\CSmHttpPlugin.cpp:3727][INFO:1] PLUGIN: EstablishSession - Decoded SMSESSION cookie -  - User = 'uid=USERIDHERE,dcxdealercode=DEALERCODEWASHERE,ou=dealerships,ou=dealers,o=DOMAINWASHERE.com', IP address = 'IPADDRESSWASHERE'.
... cut

What we needed was the occurrence of access denied where the source was the home page. A few lines later in the log, the SiteMinder user information was present. The script then needs two modes: looking for access denied and looking for the user info. Get-Content provides reader like functionality kicking out a line of text at a time from the provided file. Piping to a ForEach calls the associated script block for each line of text in the file. Use some Regex magic to find the correct string and extract information from the user information line. The result of the Regex is where I ran into the biggest hang up. PowerShell has an automatic variable $matches that is populated when a match is found (when a match is found not when a match is attempted). So I had to clear out the $matches variable after processing a match. The only output was the parsed user information that could be passed off to Brian, who deals with the imports. It is a pretty simple piece of script that looks like:

param ( $infile )

$lookingForSiteMinderCookie = $FALSE
$matches = $null
$Get-Content $infile | ForEach-Object {
  if( $lookingForSiteMinderCookie )
  {
    if ( $_ -match '^.*\[.*\](\[.*\])\[.*\]\[.*\].*uid=([a-zA-Z0-9]*).*dcxdealercode=([0-9]*).*$' )
    {
      "TimeStamp: {0}`tUser: {1}`tDealer: {2}" -f $matches[1],$matches[2],$matches[3]
      $matches = $null
      $lookingForSiteMinderCookie = $FALSE
    }
  }
  else
  {
    if ( $_ -match '^.*accessdenied\.aspx\?ReturnUrl=\%2fhome\%2fmain\.aspx.*$' )
    { 
      $matches = $null
      $lookingForSiteMinderCookie = $TRUE 
    }
  }
}

 

This was saved as LogParse.ps1.

From the PowerShell command line:

  • Use Get-ChildItem with a filter that matches the log file names to get a list of log files
  • Iterate through each file using ForEach calling LogParse.
  • Output to Out-File to save the results

Get-ChildItem logfilepattern | ForEach-Object { .\LogParse $_ } | Out-File deniedUsers.results

I have found PowerShell to be a very useful tool for doing some quick and dirty tasks. I can certainly see that with a bit more experience with it, a command line .net interpreter could be very handy. Check it out. You can rarely go wrong by having another tool in the toolbox. I do want to point out that this was primarily an exercise in using Powershell because I wanted to learn it better. The power of PowerShell is not going to come in text processing where there are many tools that have been around for a very long time that do that job very well. Many shells use strings when piping between commands. Passing a string object is not that much better. The real power comes when you start using objects and even more when you use objects in the pipeline. That is the feature unique to PowerShell.

Tuesday, February 12, 2008 8:29:01 AM (Eastern Standard Time, UTC-05:00)  #    Comments [0]  |  Trackback
Saturday, January 12, 2008

Equality comparisons in .net behave differently by default for reference types and value types. Value types are considered equal when the state of the object is equal. Reference types are considered equal when the reference points to the same location in memory. Therefore two different instances of a reference type would not be considered equal even if the internal state is the same. A quick unit test can be used to show this.

The class we will be using for comparison:

using System;

namespace Equality
{
  class MyClass
  {
    public MyClass(int id)
    { _id = id; }

    private int _id;
    public int Id
    {
      get { return _id; }
      set { _id = value; }
    }
  }
}

The test fixture and the first test:

using System;
using MbUnit.Framework;

namespace Equality.Test
{
  [TestFixture]
  class EqualityTest
  {
    const int equalId = 10;
    const int notEqualId = 12;

    MyClass testClass = new MyClass(equalId);
    MyClass equalClass = new MyClass(equalId);
    MyClass notEqualClass = new MyClass(notEqualId);

    [Test]
    public void EqualsMethod_Test()
    {
      Assert.IsTrue(testClass.Equals(equalClass),"Equal by identity but not by reference");
      Assert.IsFalse(testClass.Equals(notEqualClass), "Not equal by either identity or reference");
    }
  }
}

Test Results:

------ Test started: Assembly: Equality.exe ------
Starting the MbUnit Test Execution
Exploring Equality, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MbUnit 1.0.2700.29885 Addin
Found 1 tests
[failure] EqualityTest.EqualsMethod_Test
TestCase 'EqualityTest.EqualsMethod_Test' failed: Equal by identity but not by reference
MbUnit.Core.Exceptions.AssertionException
Message: Equal by identity but not by reference
...
0 passed, 1 failed, 0 skipped, took 1.63 seconds.

Since all reference types derive from System.Object, we can change this behavior by overriding the Equals method to do a comparison that means something to the object. In this case we will use the Id property. In this case we will first test to make sure that the object being compared is of the correct type and then we will compare the Id property. As an interesting excercise, take a look at the override of the Equals method in the System.ValueType class using Reflector. This implementation uses reflection to determine what items can hold state. It then does a comparison for each item (using .Equals in the end so that reference types work correctly).

  public override bool Equals(object obj)
  {
    MyClass testClass = obj as MyClass;
    return Id == testClass.Id;
  }

Results:

------ Test started: Assembly: Equality.exe ------
Starting the MbUnit Test Execution
Exploring Equality, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MbUnit 1.0.2700.29885 Addin
Found 1 tests
[success] EqualityTest.EqualsMethod_Test
1 passed, 0 failed, 0 skipped, took 1.59 seconds.

Now we have the Equals method expressing a meaning for equality that matches our understanding of the object. This is a nice advantage when using framework classes such as collections. Collection methods like Contains will call the object's Equals method. Simply overriding the Equals method gives us many more tools to work with without much effort. One thing that should be mentioned here is the warning that occurs when you override Equals. You will be greeted with a compiler warning about overriding Equals without overriding GetHashCode. GetHashCode is defined as a method that should return a number unique for a given instance of the object. We are using the Id property for that purpose.

public override int GetHashCode()
{ return Id; }

Now it would be really nice to have some consitancy with other mechanims used to compare equality. Another common mechanism for equality testing is the == operator. Lets put together a test that will see if this complies with our objects definition of equality.

[Test]
public void EqualOperator_Test()
{
  Assert.IsTrue(testClass == equalClass, "Equal by identity but not by reference");
  Assert.IsFalse(testClass == notEqualClass, "Not equal by either identity or reference");
}

Results:

------ Test started: Assembly: Equality.exe ------
Starting the MbUnit Test Execution
Exploring Equality, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MbUnit 1.0.2700.29885 Addin
Found 2 tests
[success] EqualityTest.EqualsMethod_Test
[failure] EqualityTest.EqualOperator_Test
TestCase 'EqualityTest.EqualOperator_Test' failed: Equal by identity but not by reference
Message: Equal by identity but not by reference
...
1 passed, 1 failed, 0 skipped, took 1.78 seconds.


 

Looks like we have a bit more work to do. What we need to do now is provide the operator == method. We will take advantage of the fact that we have already defined equality in the Equals method on our object. (Note that this is a static method that takes two objects as parameters. One side effect of this is that you do not get to make static members virtual and override them.)

public static bool operator==(MyClass lhs, MyClass rhs)
{
return lhs.Equals(rhs); }

Now we fire up the test to see where we stand and we are greeted with an error message. The compiler is telling us that when == is implemented that != is required as well. After you finish wondering why someone by default would not implement a != method as an inverse of the == method, implement the != method using our Equals method or the == operator. Of course we need to add some tests to ensure that != is doing what we expect.

[Test]
public void NotEqualOperator_EqualTest()
{
  Assert.IsTrue(testClass != notEqualClass, "Not equal by either identity or reference");
  Assert.IsFalse(testClass != equalClass, "Equal by identity but not reference");
}

public static bool operator !=(MyClass lhs, MyClass rhs)
{ return !lhs.Equals(rhs); }

It can be handy for users of your classes to be able to count on basic functionality such as equality. It also will help leverage the framework for collection operations as well. Another thing very similar in nature to this exercise that will help leverage the framework is implementing the IComparable or IComparable<T> interface.

Other Considerations

In Bill Wagner's book Effective C#, Item 10 mentions some issues with overriding GetHashCode. In short, the framework has a very efficient way to come up with the hash code, in many cases you will not come up with something better. If you do not have something as simple as an identity column that is assigned from the database, consider this advice carefully.

 

Saturday, January 12, 2008 4:19:33 PM (Eastern Standard Time, UTC-05:00)  #    Comments [0]  |  Trackback

Theme design by Jelle Druyts