WCF, Duplex Channels, firewalls and NATs

Introduction

WCF supports duplex channels; this means you can have a server fire an event back to the client. This works with Silverlight and also with full .NET apps. This blog does not go into details on how to create a duplex channel but it does show the results of testing it behind a firewall and on a NATed connection.

Creating a Duplex Channel

To create a duplex channel you need to define a contract that says what you need the client to implement. On your main contract you need to include a reference to this interface . This interface is known as the callback contract. Your service interface will look like

[ServiceContract(SessionMode = SessionMode.Required,CallbackContract = typeof(ICallBackChannel))]
public interface IService1
{
[OperationContract(IsOneWay = true)] // IsOneWay means the caller will not expect a reply, this allows us to use call back channel without causing a deadlock.
void PingServer();
}
ICallBackChannel is the interface that defines the contract that the client will implement. Eg
public interface ICallBackChannel
{
[OperationContract(IsOneWay = true)]
void Ping();
}

Other ways to get events

You don't need to use duplex channels, you can manually implement the interface in the client and manage this yourself. You will find that this approach won't work when you are NATed or firewalled.

Testing it from a NATed connection (and firewalled)

If you use netTcpBinding then the client will keep the socket open and receive call-back messages. To test I used a Virtual PC and configured the connection to use a NAT. I turned the firewall on and tested it. It worked fine. If you were to use a wsDualHttpBinding behind a NAT it would not work. Let's examine the traffic that goes on with the http binding.
When I open the Client up, you see the following traffic on the server. (This has been captured using fiddler)

The client has managed to connect to the server but the server cannot talk back to the client. This is due to the address virtualxp-28481 not existing on our network as the client is NATed. So, Duplex Channels will not work if your client is NATed. A firewall may also block the port.
If we change the binding to use netTcpBinding then it works fine. We can see the open sockets on the client.

If we kill the connection the client channel goes into a faulted state.

Some other tricks

When the connection goes into a faulted state WCF raises an event. The client and the server can hook into this event: for example, the client could try to re-open the connection.
When I create the callback channel, we can attach to the Faulted event .
wisdomCallbackServer.InnerChannel.Faulted += new EventHandler(InnerChannel_Faulted);
The code that gets fired just reconnects.
Also, as a default your server will only allow 10 connections, this can be increased by changing the settings in the config file :



If I get time, I may post the test project on codeplex. I will re-blog if I do this.

Don't use .RenderBeginTag("pre")

Even in our modern XHTML/DHTML/AJAX world we still sometimes need to use a good old fashioned <pre>

tag. In our Records Management application, Wisdom, we needed to use one to display the contents of a plain text email that had been stored in Wisdom. So I used the following code fragment:

writer.RenderBeginTag("pre");
writer.Write(HttpUtility.HtmlEncode(textContent));
writer.RenderEndTag();

but this didn't work very well! The first line of the email was indented as shown below:

         This is a
multi line
plain text email.

The reason is that RenderBeginTag uses some (normally helpful) logic internally to indent HTML tags and content so that it is more readable when you view the source. BUT you don't want this inside a <pre> tag because it preserves the whitespace! So you need to use WriteFullBeginTag and WriteEndTag methods on HtmlTextWriter instead:

writer.WriteFullBeginTag("pre");
writer.Write(HttpUtility.HtmlEncode(textContent));
writer.WriteEndTag("pre");

Reflection, byref and _COM

When using Refection against .NET codebyref just works as you would expect it to. When you are reflecting against COM byref does not work as you may expect.


If the target is .NET the following code would just work. The fact that we are creating the object from a class ID tells us it’s COM, so it won’t work.


var addin = GetCOMObject();

object[] objparams = new object[] { intVal,stringVal,etc};


var addintype = Type.GetTypeFromCLSID(new Guid("ADADAF30-E012-45db-95BE-E7544E918EBD")); // An Outlook Addin


var rval = addintype.InvokeMember(methodname, BindingFlags.InvokeMethod, null,addin, objparams);


Assume the method that we are invoking passes something back byref to param 2. The following code will not work show you the value :


stringval = (string) objparams[1];


To make this work with COM, you need to use ParameterModifers.


So, your code will look like


var addin = = GetCOMObject();

object[] objparams = new object[] { intVal,stringVal,etc};

ParameterModifier mods = new ParameterModifier(_params.Length);

mods[1] = true; // param 2 passes back by ref

ParameterModifier[] modstopass = { mods }; // we need an array
var addintype = Type.GetTypeFromCLSID(new Guid("ADADAF30-E012-45db-95BE-E7544E918EBD"));


var rval = addintype.InvokeMember(methodname, BindingFlags.InvokeMethod, null, addin, objparams, modstopass, null, null);


We had this issue with a C# Add-in for Outlook.


Let’s hope this “just works” with C#4 and the dynamic type. ..

Errors Enumerating DataRows in a DataTable in a strongly typed DataSet that has been upgraded from .NET 2.0 to .NET 3.5

In Visual Studio 2008 the version number of MSDataSetGenerator, the tool which generates strongly typed DataSets based upon a *.xsd file, increased from 2.0.50727.1433 to 2.0.50727.3074. The only significant difference in the output files is that when targeting framework .NET 3.5, the new version of the tool uses System.Data.TypedTableBase (from System.Data.DataSetExtensions) as the base class rather than System.Data.DataTable.
This sounds like a small difference but it can trip you up! Let’s say I have the following program:
class Program
{
   static void Main(string[] args)
   {
      SearchResultsDataSet ds = new SearchResultsDataSet();
      ds.SearchResults.AddSearchResultsRow(Guid.NewGuid(), "Ref1", "Title1");
      ds.SearchResults.AddSearchResultsRow(Guid.NewGuid(), "Ref2", "Title2");
      ds.AcceptChanges();
      foreach (SearchResultsDataSet.SearchResultsRow row in ds.SearchResults)
      {
         Console.WriteLine(row.Ref);
      }
      Console.ReadLine();
   }
}

SearchResultsDataSet is a strongly typed DataSet in a separate assembly called X. Assembly X and my test program are both built targeting .NET 2.0, and deployed to a customer site. I subsequently upgrade both my test program and X to .NET 3.5 and rebuild them, but only deploy the test program to the customer site; X has not changed, so I leave the existing copy in place. When my test program attempts to enumerate the DataRows in the “SearchResults” table I will get the following error:

C:\Users\alexr\Documents\Visual Studio 2008\Projects\TypedDataSetTest\bin\Debug> TypedDataSetTest.exe

Unhandled Exception: System.MissingMethodException: Method not found: 'System.Collections.IEnumerator SearchResultsDataTable.GetEnumerator()'.
at TypedDataSetTest.Program.Main(String[] args)

C:\Users\alexr\Documents\Visual Studio 2008\Projects\TypedDataSetTest\bin\Debug>

Or let’s say I do this the other way around: I upgrade both projects to .NET 3.5, rebuild both but deploy the new version of X alongside the old version of the test program. When I try to enumerate the “SearchResults” table I will get the following error:

C:\Users\alexr\Documents\Visual Studio 2008\Projects\TypedDataSetTest\bin\Debug>
TypedDataSetTest.exe

Unhandled Exception: System.NullReferenceException: Object reference not set to
an instance of an object.
at System.Data.TypedTableBase`1.GetEnumerator()
at TypedDataSetTest.Program.Main(String[] args) in C:\Users\alexr\Documents\V
isual Studio 2008\Projects\TypedDataSetTest\Program.cs:line 17

C:\Users\alexr\Documents\Visual Studio 2008\Projects\TypedDataSetTest\bin\Debug>

What’s really going on AND what can make this problem difficult to spot
You may be thinking “What kind of fool would change the .NET framework an assembly was targeting and then not deploy it to the live environment”. And you’re right. But consider the following scenario:

1. Create a strongly typed DataSet in an assembly that targets .NET 2.0.
2. MSDataSetGenerator generates a .Designer.cs file containing the code.
3. Upgrade the assembly to target .NET 3.5.
4. The .Designer.cs file is already in place and it doesn’t get regenerated. It doesn’t get regenerated UNTIL you make a change to the *.xsd file.

This caught us out with our product, because we deploy Hotfixes by keeping the Assembly Version Number the same and deploying only assemblies which have been modified (using other file metadata to distinguish them). We were caught unawares by our .NET 3.5 strongly typed DataSets suddenly morphing from being based upon System.Data.DataTable to being based upon System.Data.TypedTableBase and some of our integration components broke.

Another thing that makes this problem hard to identify is that a lot of us (quite rightly) try to diagnose problems like this by using Reflector to decompile the assembly. In this case though, the source code of the test program is the same in both cases; it’s the underlying IL that is different. The convenient C# construct of foreach (SearchResultsDataSet.SearchResultsRow row in ds.SearchResults) has this going on behind the scenes:

First, for the .NET 2.0 version:

L_0046: callvirt instance class [Wisdom.Common.DataSets]Diagonal.Wisdom.Common.DataSets.SearchResultsDataSet/SearchResultsDataTable [Wisdom.Common.DataSets]Diagonal.Wisdom.Common.DataSets.SearchResultsDataSet::get_SearchResults()
L_004b: callvirt instance class [mscorlib]System.Collections.IEnumerator [Wisdom.Common.DataSets]Diagonal.Wisdom.Common.DataSets.SearchResultsDataSet/SearchResultsDataTable::GetEnumerator()
L_0050: stloc.2
L_0051: br.s L_006d
L_0053: ldloc.2
L_0054: callvirt instance object [mscorlib]System.Collections.IEnumerator::get_Current()
L_0059: castclass [Wisdom.Common.DataSets]Diagonal.Wisdom.Common.DataSets.SearchResultsDataSet/SearchResultsRow
L_005e: stloc.1

And then the .NET 3.5 version:

L_0046: callvirt instance class [Wisdom.Common.DataSets]Diagonal.Wisdom.Common.DataSets.SearchResultsDataSet/SearchResultsDataTable [Wisdom.Common.DataSets]Diagonal.Wisdom.Common.DataSets.SearchResultsDataSet::get_SearchResults()
L_004b: callvirt instance class [mscorlib]System.Collections.Generic.IEnumerator`1 [System.Data.DataSetExtensions]System.Data.TypedTableBase`1::GetEnumerator()
L_0050: stloc.2
L_0051: br.s L_0068
L_0053: ldloc.2
L_0054: callvirt instance !0 [mscorlib]System.Collections.Generic.IEnumerator`1::get_Current()

(I might have missed a bit off the start and the end, I don’t understand IL that well). As you can see, in the second example the compiled assembly uses a generic enumerator rather than a standard System.Collections.IEnumerator. But this difference is not visible in C#.

Solving the Double Hop Issue

With a Batch File

I have come across many blogs and websites that go into great detail on how to solve the double hop issue. I have shared these with my colleagues and customers they all think that they seem too complex, and are therefore do not want to use Kerberos.

Solving the Double Hop Issue with Kerberos is not complex.
This diagram illustrates the problem.

This is taken from a very good blog on the same subject which goes into way more detail…

What you need to fix it

• I would recommend running services as a DOMAIN USERs rather than network service or local users.
(“services” meaning Application Pools for web sites and SQL services)
• Service Principal Names.
• Local Machine Rights granted to the Service Accounts.
• Act as part of operating system
• Impersonate user
• Settings in AD for the Machines to allow delegation.
• Settings in AD for the Service Accounts to allow delegation.

 

What does this mean

A Service Principal Name (SPN) is the name by which a client uniquely identifies an instance of a service. For example HTTP/Servername is the SPN for a web site at Servername. MSSQLSvc/ServerName:1234 is the SPN for a SQL Service running on Servername on Port 1234 (see http://msdn.microsoft.com/en-us/library/ms677949(VS.85).aspx for more information on service principal names).

Local Machine rights are managed with Local Security Policy (found under Administrative Tools) .

Delegation is to pass someone’s user credentials to another process or server.

Both AD settings are edited with the MMC Snap-in “Active Directory Users and Computers”. You need to allow both the service users and the machine to delegate.

Some tools from the web

You need :

• Found on the windows resource kit
• NTRIGHTS.EXE (Set local machine rights)
• SETSPN.exe
• Installed with IIS
• adsutil.vbs
• On codeplex
• machinedelegation.vbs
• userdelegation.vbs
• Below and on codeplex
• Enable.bat

The Batch File

@REM ==================================
@REM Setup VARS
@REM ==================================

set MACHINE1=DBLHOP
set MACHINE2=DBLHOPSVR

set FQDN=domain.local

set USERACCOUNT=DOMAIN\ServiceAccount
@REM WEBID is the WEBSITE ID, THE Default website is 1
set WEBID=1

@REM ==================================
@REM Use adsutil to config IIS to use kerberos
@REM ==================================

cscript C:\inetput\adminscripts\adsutil.vbs set w3svc/WebSite/root/NTAuthenticationProviders Negotiate

@REM ==================================
@REM Use SETSPN to create the SPNs
@REM ==================================

setspn -A HTTP/%MACHINE1% %USERACCOUNT%
setspn -A HTTP/%MACHINE1%.%FQDN% %USERACCOUNT%

setspn -A HTTP/%MACHINE2% %USERACCOUNT%
setspn -A HTTP/%MACHINE2%.%FQDN% %USERACCOUNT%

@REM ==================================
@REM Use NTRIGHTS to enable Act as part of operating system and
@REM Impersonate a client after authentication
@REM ==================================

ntrights +r SeTcbPrivilege -u %USERACCOUNT% -m \\%MACHINE1%
ntrights +r SeTcbPrivilege -u %USERACCOUNT% -m \\%MACHINE2%

ntrights +r SeImpersonatePrivilege -u %USERACCOUNT% -m \\%MACHINE1%
ntrights +r SeImpersonatePrivilege -u %USERACCOUNT% -m \\%MACHINE2%


@REM ==================================
@REM Use VB Scripts to give the user the delegate right
@REM ==================================

cscript userdelegation.vbs %USERACCOUNT% enable

@REM ==================================
@REM Use VB Scripts to give the machine the delegate right
@REM ==================================

cscript machinedelegation.vbs %MACHINE1%
cscript machinedelegation.vbs %MACHINE2%

@REM ==================================
@REM Use IISRESET to restart IIS on both servers
@REM ==================================

iisreset %MACHINE1%
iisreset %MACHINE2%

Really good external websites

blog foo
ASP.NET Applicaiton to help

More Info

Diagonal

Wisdom and Microsoft Dynamics CRM

The wisdom team have been busy integrating Wisdom into Microsoft Dynamics CRM. Cases created in CRM can now be linked to a Wisdom case area. This gives Dynamics CRM Document and Records management capabilities and much more.

We believe that we have taken an interesting approach to this problem that will allow CRM administrators to link any CRM entity type to Wisdom thus giving any CRM entity EDRM capabilities.
Useful links

Microsoft Dynamics http://www.microsoft.com/dynamics/
Wisdom http://www.diagonal-consulting.com/Technology/Wisdom/Pages/Default.aspx

Enumerating Files in a Directory from C#

Enumerating files in a directory with C# seems straight forward, there's Directory.GetFiles that gives strings and DirectoryInfo.GetFiles that gets FileInfo objects. However both these methods return arrays, when there are very large numbers of files this can be an expensive call.

In .net 4.0 Microsoft are addressing this issue with methods like DirectoryInfo.EnumerateFiles.

We can get this functionality in .net 2.0/3.x by making native calls to the methods that GetFiles use, namely FindFirstFile, FindNextFile and FindClose. Pinvoke.net is an excellent resource for using native windows API calls from .net code, and looking at their entry on FindFirstFile gives a good example of computing the total size of a directory.

The sample code shows how native handles should be wrapped with classes deriving from SafeHandle which makes sure unmanaged resources get cleaned up, in this case calling FindClose.

Using the DllImports and associated structures provided there we can then easily write a method that uses the yield keyword to enumerate the files in a directory, without having to add them all to a collection and without the consuming code knowing about native structures like WIN32_FIND_DATA:

public static IEnumerable<string> EnumerateFiles(string directory, string filePattern)
{
 string pattern = directory + @"\" + filePattern;
 WIN32_FIND_DATA findData;
 using (SafeFindHandle findHandle = FindFirstFile(pattern, out findData))
 {
   if (!findHandle.IsInvalid) // was the input valid, e.g. valid directory passed
   {
     do
     {
       if ((findData.dwFileAttributes & FileAttributes.Directory) == 0) // if not a directory
       {
         yield return Path.Combine(directory, findData.cFileName);
       }
     }
     while (FindNextFile(findHandle, out findData));
   }
 }
}

Using the yield keywords means that at no point are the strings all loaded in to memory. If consuming code breaks the enumeration it means the code after the yield statement will not be executed, so in this case no further calls to FindNextFile would be made. Finally blocks will however be executed when the enumeration gets broken, which means that the SafeFindHandle will get disposed (a using statement implicitly uses a finally block), releasing the find handle.

This just gets file paths, but it could easily be extended to return other information from WIN32_FIND_DATA and return a set of objects. Dates can be converted from the FILETIME structure by using bit shifts and DateTime.FromFileTime, e.g.

static DateTime GetFileTime(FILETIME fileTime)
{
  long fileTimeValue = (long)fileTime.dwLowDateTime
    | ((long)fileTime.dwHighDateTime <<
  return DateTime.FromFileTime(fileTimeValue);
}

This approach should only be used when the directories you're working with are likely to contain large numbers of files, but do allow operations to be performed on the files as the list is being retrieved from the file system.

Testing Console Applications

I did some work this week on a command line installer for RBS. I needed to make some modifications to the install process. I wanted to-do this the TDD way. I did not want to refactor the entire project just to make it testable, I thought I would try to come up with a nice way to test my console application. I needed to capture the output of the console application to check that my new features were working correctly.

RBS is remote blob storage, see the RBS blog http://blogs.msdn.com/sqlrbs/

Firstly I tried to launch the exe and capture the output, this worked well but I could not debug my new code. I know true TTDers don't use debuggers, but I needed to :)

I managed to get this working by referencing my Console Project from my test project (yes you can reference an EXE). I could then add code like :

InstallProviderSetup.Main(new string[] { "-CLIENTCONFIG" });

As you can see, you can just call the static Main method from code. To see if my test worked I need to check the output of the console application. I did this with the following code :

var stdout = GetStdOut();
Assert.AreEqual(true, stdout.Contains("The required switch CONFIGURATIONFILE"),"The required switch CONFIGURATIONFILE not in console out");
Assert.AreEqual(true, stdout.Contains("The required switch NAME"),"The required switch NAME not in console out");

The rest of the code is :

MemoryStream memoryStreamConsole;
StreamWriter streamWriterConsole;

[TestInitialize()]
public void TestInitialize()
{
memoryStreamConsole = new MemoryStream();
streamWriterConsole = new StreamWriter(memoryStreamConsole);
Console.SetOut(streamWriterConsole);
}

protected string GetStdOut()
{
streamWriterConsole.Flush();
var rval = Encoding.Default.GetString(memoryStreamConsole.ToArray());
System.Diagnostics.Trace.WriteLine(rval);
return rval;
}

Feel free to view the entire project at codeplex

http://sqlrbs.codeplex.com/

 

Diagonal are using Microsoft RBS to give Wisdom (an Electronic Records Management System (EDRM) ) Content Addressable storage (CAS). So far we have integrated with EMC. As more CAS vendors create RBS providers. Wisdom will support more CAS systems.
You can find out more about Diagonal and Wisdom by visting the wisdom website

Private Members in Unit Tests

I was running a technical session on TDD today. Whilst I was doing my research into ways to work with private members I discovered a new way. The C# language has evolved and the tools in Visual Studio have improved providing new ways to solve old problems. You can now solve the private member problem the following ways:

I won't go into the first few too much as I believe the last one supersedes them.

#if DEBUG or TEST

You can surround your public accesses to your private members inside compiler directives so they only get built when you build you unit tests.

Internal members and internal visible to

Make your private members internal rather than private and then use the InternalsVisibleTo attribute to make them visible to your tests. (new to .net 2)

Partial classes

You can put your unit tests in a partial class and exclude this class when you build the release code (new to .net 2)

Create a private accessor

This is a feature which is new in VS2008. You can see this working as follows:

• Create a new Class Library project
• Add a private member to Class1 (the generated one)
  string nottelling = "my tests cannot access me";
• Add a test project
• 
  
  Open class1.cs and right click on the text Class1.
  
  
  

• Select Create Private Accessor -> TestProject1
• This will create a Private Accessor in test project one.
• Add a new Unit test and enter the following code

[TestMethod()]

public void Test()

{

  var class1 = new Class1_Accessor();

  Assert.AreEqual(class1.nottelling, "my tests cannot access me");

}

Old Problems may have different solutions using newer tools, you should always keep looking for new ways to solve old problems

Trees & Hierarchies in SQL

Hierarchies and trees are a challenge in SQL.
This is an example of a very simple tree

Parent ID	ID	Name
	0001	Earth
0001	0002	EU
0001	0003	USA
0002	0004	UK
0002	0005	France
0004	0006	England
0006	0007	Yorkshire

With our application the Parent ID and ID columns are both GUIDs. To help with readability I have used numbers in the examples.


This looks like
Earth
  EU
    France
    UK
      England
        Yorkshire
  USA


The table structure above will work very well in most cases. It's very easy to select the parent of an item and select the children of an item. You can easily add new items or delete items. You get problems when you need to find things like all the decedents of an item.

You can use CTE (common table expressions) but under heavy load and with lots of data it just does not work.

We looked at the nested model. This does not work with simple deletes and inserts. (http://www.developersdex.com/gurus/articles/112.asp )

The final approach we took was to add an incremental number to our table and a row path column. As follows :

Row bitint
RowPath varbinary(160)
When we generate the rowpath we pad it with zeros so each row number starts at the same place. We do not use delimiters we used fixed spaces.

This is a simple representation of how the table will look. This is simple as I am just padding the rowpath with 2 zeros (not 16).

Row	Parent ID	ID	Name	RowPath
1		0001	Earth	01
2	0001	0002	EU	0102
3	0001	0003	USA	0103
4	0002	0004	UK	010204
5	0002	0005	France	010205
6	0004	0006	England	01020406
7	0006	0007	Yorkshire	0102040607

This is an example of how the data looks in SQL with it padded with 16 zeros.

0x0000000000000795
0x00000000000007950000000000000796
0x0000000000000797
0x00000000000007970000000000000798
0x000000000000079700000000000007980000000000000799
0x00000000000007970000000000000798000000000000079A
0x00000000000007970000000000000798000000000000079B
To get the child items, then we need to run the following SQL :

@path is the path that we need to find the child items from
@pathplusone is our path plus one, see the function below for how to increment this.
Select Name from Table where RowPath > @path and RowPath < @pathplusone


We increment the path with the following SQL

CREATE FUNCTION [dbo].[IncrementRowPath]
(
    @VarBinary VARBINARY(160) = 0x0000000000000000
)
RETURNS VARBINARY(160)
AS
BEGIN
    DECLARE @Result    VARBINARY(160)
    -- If the data is the wrong size.
    IF(@VarBinary IS NULL OR DATALENGTH(@VarBinary) % 8 != 0)
        SET @Result = 0x0000000000000000
    -- Increment the last 8 byte segment.
    ELSE
    BEGIN
        DECLARE @LastSegment BIGINT
        SET @LastSegment = CAST(SUBSTRING(@VarBinary, DATALENGTH(@VarBinary) - 7, 8) AS BIGINT)
        SET @Result = SUBSTRING(@VarBinary, 1, DATALENGTH(@VarBinary) - 8) + CONVERT(VARBINARY(8), @LastSegment + 1)
    END
    RETURN @Result
END

I hope this helps you out with trees.

The only issue is that you need and index on rowpath, this index is going to be large, but in our testing and our live implementations it's not caused an issue.