Tuesday, August 14, 2012

Are you Monitoring the rights stats on your database server?

The blog is based on real world issues that I have seen over the past few years.
SQL server is so easy to setup and cluster that anyone with good knowledge of general infrastructure can do this. This is good but can lead to incorrect configuration. How many people tweak MAXOP ? do you lock pages in memory? do you create one tempdb per CPU core? A good DBA will know what all these questions mean. Also quite rightly most people put LOGS and DATA on their own drive. But for high transactional database have you thought about tempdb and snapshot isolation? Far too much to go over in this blog.
Lets assume the following has happened; You have a hosting requirement for CRM (I picked CRM as at the moment , I ‘do a lot’ of CRM). So your CRM Architect sizes the servers as :

Your infrastructure team build this and put in the right firewalls etc. Now comes the monitoring. The SQL server has a the following drives :

OS	120GB	C:
DATA	120GB	D:
LOGS	60GB	L:

CRM is install and it creates the database. The LOG or DATA files are not pre-sized you just let the system run. You have set backups based on your standard backup policy, e.g. LOGS nightly and DATA weekly.
On some days CRM’s ReIndex job starts and the LOG file gets very large. Your disk monitoring software detects the low disk space. Someone logs in to the database server looks at the log drive and see that it has low disk space. So… they open google and typing “Log file large SQL” and if you follow the first article without reading in correctly you may find yourself ready to enter something like :
DBCC SHRINKFILE (’CRMORG_Log’, 1000)
Lets assume that the backup has run so the log file has lots of free space.
You look at free space on disk and wow.. you have fixed it. The next day comes and the first few CRM transactions need to grow the log. In other words when a user updates a record in CRM it could trigger a transaction that has the side affect of ‘growing’ the log. This will take time and other transaction may have to queue up behind.
This performance issue is noted so you contact your DBA. The DBA may run a script like :

SELECT databasename,e.name   AS eventname,cat.name AS [CategoryName],starttime,e.category_id,
       loginname,loginsid,spid,hostname,applicationname,servername,textdata,objectname,eventclass,
       eventsubclass
FROM   ::fn_trace_gettable(@path, 0)
       INNER JOIN sys.trace_events e
         ON eventclass = trace_event_id
       INNER JOIN sys.trace_categories AS cat
         ON e.category_id = cat.category_id
WHERE e.name IN( 'Data File Auto Grow', 'Log File Auto Grow' )
ORDER BY starttime DESC

This will show that the log file has been auto growing. The DBA may then pre-size the log to fix this.
Now… we are back to the start and you run low on disk space. To be honest the DBA may not get involved and you may not pre-size the log.

What should we do?

Firstly, you need to understand the role of a DBA. The ability to install SQL and create a database does not mean you are a DBA. No offence meant its just that Microsoft have made SQL so easy to manage that people don’t hire a DBA and that's a key issue. You should at least ‘get one in’ for a day or two. I have visited many customers and sat with their DBA and pointed out many issues with the health of their SQL server.
If the databases have been sized correctly. Based on; Number of users, User Transactions, Backlog e.g. data load, data type e.g. any blobs? and Backup Strategy (You need to understand the backup strategy to size the logs).
We have this disk layout

OS	120GB	C:
DATA	120GB	D:
LOGS	60GB	L:

This is based on a CRM log requirement of 50GB, you should pre-size the log at 50GB and monitor the free space within the log. The following SQL will do show you the free space:
DBCC SQLPERF(LOGSPACE)
You will need a monitoring tool capably of view the output of SQL statements.

What else?

I intend to write more blogs on this subject as I come across issues.
I have also seen systems that monitor backups but they don't monitor the SQL backups. In simple terms you have a SQL jobs that BACKUPS a DATABASE to DISK. You have a backup tool that backs up the BACKUP from DISK to TAPE. You monitor the backup tool but what if the SQL backup jobs have been failing? who is looking at that? This would also cause large transaction logs.

Thursday, July 12, 2012

CRM 2011 / ADFS / IFD / 401 Global.ashx (now fixed, scroll to the end for the fix)

Recently I have been helping diagnose a problem with CRM and authentication.

This was not an issue before roll up 7, You do not get this issue with roll up 6 but you do get this issue with roll up 7 and 8.

CRM is hosted in our data centre and ADFS is used to authenticate. The issue reported was

“User can log in from a test PC in the data centre using the ADFS server in our data centre but user on the customer site cannot login, they keep getting a login box”.

Other symptoms are, a blank screen like :

If you are not a member of a group, you will see the user not a role message.

A work around is to enter a user name and password from our data centre’s domain. In other words log in as follows:

Login to CRM using the customers ADFS server but when we get challenged for a username and password enter it from the data centre’s AD. This is counter intuitive but does work and the user you are logged into CRM as is not the data centre account.

For example ( I use in inprivate session to make sure no auth cookies are cached )

I navigate to CRM and select the ADFS server that's in the domain that my PC belongs to, CRM is not in this domain.

Note, our ADFS server is sts1 (see the url) but I have selected dsladfs.diagonal-solutions.local (NT4 name MFTDEV) (this is a test domain and our old company name :) )

I then get asked for a username and password again.

As we can see, its asking to connect to crm.xxxx.local this is the CRM application hosted in our data centre not the selected ADFS log on server so this challenge is coming from CRM and not the ADFS server. If I put a user from the data centre it works, if I cancel I get the blank screen.

Now lets look at fiddler :

I have started from the crm.xxxx.local, got a 302 redirection this sends me to ADFS (sts1) and select diagonal (MFTDEV) as my auth domain this challenges me as I am part of this domain IE just handles this. I then get sent back the CRM. main.aspx loads correctly, but when IE requests global.ashx I get a 401. If we look at this in more detail we can see that the cookies are being sent :

Technical Investigation

We have now seen the problem, for further testing I just used fiddler, if I just reissue the request I get the same issue.

Do we get this for other files? If I copy global.ashx to test.ashx and used fiddler I don’t see this issue.

If I create a new file :

Then use fiddler to ‘get’ this

It works, HTTP 200 result and I can see the script. (Bottom right hand pain).

if I remove the cookie, i get the following :

I used Composer to edit the headers and remove all cookies, I get a 302 and a redirect the the STS server. This is what I would expect as I have no auth tokens.

If I do the same, but with Global.ashx I see (first one with cookies, second without)

As you can see, both request get a 401. One should get a 302 and the other a 200.

I am going to raise this with MS and while they look into it I am going to see if I can find a work around (watch this space).

What is causing this?

A colleague of mine (Terry Rogers) have helped decompile the code we can see that

There is a check in the MainApplication HttpApplication class (which would have been compiled from the global.asax file) that checks for certain page requests, including global.ashx.

It looks like this check is to determine does the current user have any role (either against the user, or against a team they are a member of), and if not redirect to the error page indicating there is no role.

It seems likely that this check, at the PostAuthenticateRequest phase of the application lifecycle, is getting the details of the user in a way that can cause IIS/asp.net to challenge for a windows authentication, where the claims based identity should be used.

The Fix

We need to trick the code into not checking perms to serve this file. This is not a security risk as this is just static script files loaded from the server.

Step 1

Open C:\Program Files\Microsoft Dynamics CRM\CRMWeb\_common ( or your CRMWeb\_common folder)

Copy _Global.ashx to _Global_.ashx and ScriptResx.ashx to ScriptResx_.ashx

You will now have

Step 2

Edit web.config under C:\Program Files\Microsoft Dynamics CRM\CRMWeb

Just before the </rules> add

Your <rules> section will now look like

What next?

Well…. lets hope its fixed in Roll Up 9 and if it isn't fixed lets hope that above fix still works :)

Thursday, February 23, 2012

Mock Asynchronous Web Services - Part 2

Following on from part 1, I will show how to mock out an synchronous service that has the BeginMethod and EndMethod access pattern (the IAsyncResult pattern) rather than the MethodAsync and OnMethodCompleted event variety (the Event based pattern). Also I will show how to introduce a delay to your asynchronous response, something we glossed over last post. This is fairly important as the whole point of using an async service is your app should be doing something else whilst waiting for the response. Eg. in silverlight you might have waiting spinners showing and various commands disabled until the service returns but the rest of the UI should remain responsive.

Before looking at that I just wanted to run through the some advantages of mocking out all your web service dependencies:

Unit testing – If you are unit testing bits of your app (normally the Model and maybe the View Model for silverlight) then it is far easier when you have control over the exact data returned. I’ve written and seen plenty of tests that rely on specific sets of data in a ‘real’ system and it is a pain to setup and rather brittle.

Manual testing of specific conditions – even if you are just manually testing your app there will be various conditions that are quite hard to replicate without a mock service. You can easily simulate various exceptions being thrown by the web service, odd sets of data being returned, timeouts etc… If you are having to setup data in a third party system to cover all these cases then this will take far longer than creating a few mock services will.

Off-line development – It is useful to be able to develop your app without needing various other systems to be up and running. For example the silverlight app I have been doing recently calls Dynamics CRM, Sharepoint and a custom web service, you don’t want all these running on your dev machine.

Anyway, on to creating a mock for a service that uses the standard Begin / End async pattern.

Firstly, exactly like in the last post, we pull out the relevant methods from the generated Reference.cs file into an adapter interface (you may recognise this as the signature of the Dynamics CRM service method that takes fetchXml):

public interface ICrmServiceAdapter

{

IAsyncResult BeginRetrieveMultiple(QueryBase query, AsyncCallback callback, object asyncState);

EntityCollection EndRetrieveMultiple(IAsyncResult result);

}

and then implement a ‘pass through’ version for the real service:

public class CrmServiceAdapter : ICrmServiceAdapter

{

IOrganizationService service;

public CrmServiceAdapter(IOrganizationService service)

{

this.service = service;

}

public IAsyncResult BeginRetrieveMultiple(QueryBase query, AsyncCallback callback, object asyncState)

{

return service.BeginRetrieveMultiple(query, callback, asyncState);

}

public EntityCollection EndRetrieveMultiple(IAsyncResult result)

{

return service.EndRetrieveMultiple(result);

}

Then we need to create a mock implementation of this adapter interface. The key realisation here is that you need to create your own IAsyncResult object to return from the Begin method. This interface defines a few important properties:

AsyncState – this will hold the data that lets the EndRetrieveMultiple method know what mock data it should be returning (it could store the mock data itself)

AsyncWaitHandle – this is so consumers can await the completion of the operation

IsCompleted – this should indicate whether the operation has completed.

Here is the implementation of this interface we will use:

private class AsyncResult : IAsyncResult

{

private WaitHandle waitHandle = new ManualResetEvent(false);

public bool IsCompleted

{

get { return waitHandle.WaitOne(0); }

}

private Dictionary<string, object> state;

public AsyncResult(Dictionary<string, object> state) { this.state = state; }

public object AsyncState { get { return state; } }

public WaitHandle AsyncWaitHandle { get { return waitHandle; } }

public bool CompletedSynchronously { get { return false; } }

}

We use a ManualResetEvent as the waithandle. This will allow us to signal that the operation has completed by calling the Set() method. The IsCompleted property just checks if this has been signalled yet by waiting for it with a timeout of 0.

Now our MockAdapter:

public class CrmServiceMockAdapter : ICrmServiceAdapter

{

private class AsyncResult : IAsyncResult

{

/* snip - see above*/

}

const int maxResponseDelay = 5000;

private void invokeResponse(Action action)

{

ThreadPool.QueueUserWorkItem(new WaitCallback( _ =>

{

Thread.Sleep(new Random().Next(maxResponseDelay));

action();

}));

}

public IAsyncResult BeginRetrieveMultiple(Xrm.Silverlight.Sdk.QueryBase query, AsyncCallback callback, object asyncState)

{

/* snip - set state according to the input so we know what to return from the End method */

IAsyncResult asr = new AsyncResult(state);

Action responseCompleteAction = () =>

{

((ManualResetEvent)asr.AsyncWaitHandle).Set();

callback(asr);

};

invokeResponse(responseCompleteAction);

return asr;

}

public Xrm.Silverlight.Sdk.EntityCollection EndRetrieveMultiple(IAsyncResult result)

{

/* snip - return some mock data according to the contents of result.AsyncState */

}

Note how we queue up a work item on the ThreadPool to do the following (this is the bit you need to add to the code in the last post to simulate a long running service call):

1) Delay for a random amount of time

2) Set the ManualResetEvent to indicate completion

3) Call the callback function

Note Random is not thread safe so don’t be tempted to make this a member of the class.

And that pretty much covers it. Finish by creating a factory interface and two implementations exactly like in the last post and you should be up a running with a freshly minted mock CRM service:

var crmService = crmServiceFactory.Create();

crmService.BeginRetrieveMultiple(fetchXml, asr =>

{

var result = crmService.EndRetrieveMultiple(asr);

/*snip - do whatever with your results */

}, null);

Are you sure you want to use IAsyncResult?

It’s worth noting that using IAsyncResult async when developing from scratch is only really recommended in the following situations (taken from http://msdn.microsoft.com/en-us/library/ms228966.aspx):

Only expose the IAsyncResult pattern when you specifically require support for WaitHandle or IAsyncResult objects.
Only expose the IAsyncResult pattern when you have an existing API that uses the IAsyncResult pattern.
If you have an existing API based on the IAsyncResult pattern, consider also exposing the event-based pattern in your next release.
Only expose IAsyncResult pattern if you have high performance requirements which you have verified cannot be met by the event-based pattern but can be met by the IAsyncResult pattern.

If you do have an existing service that uses this pattern and you would rather use the Event based pattern then you could easily write an adapter interface to convert from one to the other. This is pretty much what is done for you when you generate an async service reference in .Net 3.5 and above, although for some reason it only adds the event based methods to the Client class rather than the generated interface.

Friday, February 17, 2012

Mock Asynchronous Web Services (Silverlight)

Adding a Service Reference in a silverlight project will only generate asynchronous service methods. It is fairly well known how create a mock service for a normal service reference, in this post I will run through an easy way to do this for an asynchronous service. This should enable you to test your silverlight apps better (or develop them on the train without wi-fi in my case).

Example Service

Let’s use a really simple service definition which is created when you add a new WCF Service Application project:

[ServiceContract]

public interface IService

{

[OperationContract]

string GetData(int value);

}

public class Service : IService

{

public string GetData(int value)

{

return string.Format("You entered: {0}", value);

}

and add a service reference to this in a silverlight project. If you now look at the Reference.cs generated we see the following IService definition:

The Code Behind

public interface IService

{

System.IAsyncResult BeginGetData(int value, System.AsyncCallback callback, object asyncState);

string EndGetData(System.IAsyncResult result);

}

It has generated a BeginGetData and EndGetData pair of methods following the async pattern. The client proxy generated does implement this interface, however these methods are hidden and some new public methods and events are added (I’ve cut out most of the irrelevant stuff to make it clearer):

public partial class ServiceClient : ClientBase<IService>, IService

{

public ServiceClient(Binding binding, EndpointAddress remoteAddress) :

base(binding, remoteAddress)

{

}

[EditorBrowsableAttribute(EditorBrowsableState.Advanced)]

string IService.EndGetData(IAsyncResult result) { /*snip*/ }

public void GetDataAsync(int value) { /*snip*/ }

public event EventHandler<GetDataCompletedEventArgs> GetDataCompleted;

[EditorBrowsableAttribute(EditorBrowsableState.Advanced)]

IAsyncResult IService.BeginGetData(int value, AsyncCallback callback, object asyncState) { /*snip*/ }

}

We can see that the actual interface methods have been marked EditorBrowsableState.Advanced meaning you won’t see them in intellisense. The ‘visible’ members, GetDataAsync and GetDataCompleted exist only on the proxy class and it is these you normally use when calling the service in your client, something like this:

ServiceClient client = new ServiceClient(new BasicHttpBinding(), new EndpointAddress("http://service.com/service.svc"));

client.GetDataCompleted += new EventHandler<GetDataCompletedEventArgs>

(

(sender,e) => Console.WriteLine(e.Result)

);

client.GetDataAsync(10);

It is this method and event that we need to implement in our mock.

Mocktastic

The easiest way I have found to do this is to first pull out the relevant members into an adapter interface:

public interface IServiceAdapter

{

void GetDataAsync(int value);

event System.EventHandler<GetDataCompletedEventArgs> GetDataCompleted;

}

And implement a service adapter that takes an instance of the proxy and passes through the method calls and the event subscriptions by overriding the default add and remove accessors for the event:

public class ServiceAdapter : IServiceAdapter

{

ServiceClient client;

public ServiceAdapter(ServiceClient client)

{

this.client = client;

}

public void GetDataAsync(int value)

{

client.GetDataAsync(value);

}

public event EventHandler<GetDataCompletedEventArgs> GetDataCompleted

{

add

{

client.GetDataCompleted += value;

}

remove

{

client.GetDataCompleted -= value;

}

Add another adapter which implements whatever mock functionality you want:

public class MockServiceAdapter : IServiceAdapter

{

int mockData;

public MockServiceAdapter(int mockData)

{

this.mockData = mockData;

}

public void GetDataAsync(int value)

{

//if you need to simulate a long running event spin up a thread call this event after a wait

GetDataCompleted(null, new GetDataCompletedEventArgs(new object[] { mockData }, null, false, null));

}

public event EventHandler<ServiceReference.GetDataCompletedEventArgs> GetDataCompleted;

}

And finally add a service factory interface and two implementations so we have a consistent way for the client to get an instance of the mock or real service:

public interface IServiceAdapterFactory

{

IServiceAdapter Create(Binding binding, EndpointAddress endpoint);

}

public class ServiceAdapterFactory : IServiceAdapterFactory

{

public IServiceAdapter Create(Binding binding, EndpointAddress endpoint)

{

return new ServiceAdapter(new ServiceClient(binding, endpoint));

}

public class MockServiceAdapterFactory : IServiceAdapterFactory

{

public IServiceAdapter Create(Binding binding, EndpointAddress endpoint)

{

int mockData = 10;

return new MockServiceAdapter(mockData);

}

The Result

Now we can pass the appropriate instance of IServiceAdapterFactory into our silverlight model and replace all uses of the ServiceClient with a call to the factory’s Create method:

public partial class MainPage : UserControl

{

bool useMock = true;

IServiceAdapterFactory serviceFactory;

public MainPage()

{

InitializeComponent();

if (useMock)

{

serviceFactory = new MockServiceAdapterFactory();

}

else

{

serviceFactory = new ServiceAdapterFactory();

}

IServiceAdapter client = serviceFactory.Create(new BasicHttpBinding(), new EndpointAddress("http://service.com/service.svc"));