HTTPS Communication – HttpListener based Hosting and Client Certification

KEYWORDS: HTTPS, SSL, HttpListener, X509Chain, X509Certificate2, makecert, OpenSSL, client certificate

(Just found, a wonderful tool set that could help you to host easily, http://katanaproject.codeplex.com/. The site referred my post, and I didn't realize till now  :-) thanks for refering.  [2016-2-6])

HttpListener is the easiest way for you to host an HTTP/HTTPS server. This article provides you step-by-step instructions to create your own server and authenticate clients based on client certificate from ground up in C#.

Download the sample code

STEP 1

Firstly, you should create your .net application and add these four lines.

var server = new HttpListener();
server.Prefixes.Add("https://+:90/");
server.Start();
HttpListenerContext context = server.GetContext();

These fourlines will make your server started and listening on the port. Be aware of the exceptions (HttpListenerException) thrown from the invocation server.Start(), and see step 2 to solve it.

STEP 2

Step 1 shows you it’s so easy to start a server. But wait, Start() throws an exception (HttpListenerException: Access Denied, native error code 5, HRESULT 80004005), if you run your app under non-privilege account. If you want a non-privilege account to run the server, you have to add ACL (Access Control Lists) to the system. In command line:

netsh http add urlacl url=https://+:80/MyUri user=DOMAIN\user

Pay attention to the parameter ‘user’. Put whatever user you want to assign the start server right to here. If set the parameter user=users, it will grant all user account (non-privileged) to start the app and listen on the specific ip and port. The ip part ‘+’ stands for all IPs of your machine. For the server you want to handle urls from root (e.g. http://localhost/), you don’t need ‘MyUri’ part, and your command is like this:

netsh http add urlacl url=https://+:80/ user=DOMAIN\user

STEP 3

And then your app won’t throw any exception. Your app would be blocked at server.GetContext() and waiting for incoming connections. Try the url https://localhost:90/ in your browser, there is still an error page with HTTP 101 ERR_CONNECTION_RESET. This because you haven’t assign a certificate to the server and the browser can’t verify the validity of the server. Remember we are visiting an HTTPS site. The server certificate is a must.

So, let’s create the certificates. You can either create your certificates by makecert or by OpenSSL. And this How to Setup a CA gives you an easy tutorial of creating certificates hierachy by OpenSSL. First is the root CA certificate. For experimental cases, makecert is enough. But for product, you may want to use OpenSSL or apply a certificate from CA like VeriSign.

makecert -n "CN=TestCA" -r -sv TestCA.pvk TestCA.cer

And import the root certificate to the system certificate storage of Rusted Root Certification Authority. See this article.

Then create the certificate for your HTTPS web site.

makecert -iv TestCA.pvk -n "CN=TestSite" -sv TestSite.pvk -ic TestCA.cer TestSite.cer -sr LocalMachine -ss My -sky exchange -pe

If you will test your client app on a machine other than the server machine, you have to import the TestCA.cer to the client machine as well. So that the client machine trust TestCA (the root cert), it will also trust the server certificate (TestSite).

Hosting an HTTPS site, you must have a certificate with private key. But the last makecert command creates the private key in TestCA.pvk which can’t be imported to the system storage directly. We have to convert it to .pfx format:

pvk2pfx -pvk "TestSite.pvk" -spc "TestSite.cer" -pfx "TestSite.pfx"

Then you will see the certificate for your site:

STEP 4

How to use the server certificate? At this point, the when client connect to the server, the client will throw an exception (WebException The underlying connection was closed: An unexpected error occurred on a send), simply because the server doesn’t use the certificate yet. To resolve the exception,just binding the certifiate to the server’s ip and port by netsh.

netsh http add sslcert ipport=0.0.0.0:90 appid={61047666-992C-4137-9303-7C01781B054E} certhash=75d0fed71881f2141b5b6cb24801dfa554439b1c clientcertnegotiation=enable

‘0.0.0.0’ in the ipport parameter means every ip of this machine would be assigned with the certificate. The parameter appid is your application id. You can see it in the project property, the ‘Application’ page, and the dialog poped up by clicking ‘Assembly Information’ button. The parameter ‘clientcertnegotiation=enable’ will allows C/S mutually authentication based on certificates, i.e. server side could verfiy the certificate validation of the client side as well as the client side verifying the server side. If you don’t want verification for client side, just omit the parameter.

STEP 5

Visit https://localhost:90/ again, your browser will warning you that the site is not the owner of the certificate. It’s because we don’t have a domain for our experimental site and no domain name was set into the certificate. So just click continue to view the page and the browser will show you a blank page.

Let’s add responding code to the server side, so that we can see something on the page.

string message = "Hello World!";
var buffer = System.Text.Encoding.UTF8.GetBytes(message);
context.Response.OutputStream.Write(buffer, 0, buffer.Length);
context.Response.OutputStream.Close();

Now the page displays “Hello World!”.

STEP 6

We have done the work of constructing server side. The server can show its identity by providing its certificate and client can verify it. Client still shows no certificate to the server. In some cases, the server need to verify the client’s identity, and only when the client is valid (e.g. a valid member of some organization) the server would start data communication. In this case, a client app (other than web browser) is a must. So let’s create a client app.

Here is the basic client code without client certificate.

ServicePointManager.ServerCertificateValidationCallback =
       new RemoteCertificateValidationCallback(CheckValidationResult);

string url = "https://localhost:90/";
Console.WriteLine("Visiting " + url);
HttpWebRequest objRequest = System.Net.HttpWebRequest.Create(url) as HttpWebRequest;
objRequest.ProtocolVersion = HttpVersion.Version10;

var response = objRequest.GetResponse();
var responseReader = new StreamReader(response.GetResponseStream());
var responseContent = responseReader.ReadToEnd();
Console.WriteLine("Server replied: " + responseContent);

CheckValidationResult is a callback function which allows you to perform customized validation against server certificate, returns true to accept the certificate. As expected, the client gets the server reply: “Hello World!”.

STEP 7

Here we add client certification code. Basically you have two ways of creating a X509Certificate2 which could contain public/private key pair. Other ways like manipulating public/private key pair raw data directly, may be tricky and complex.

1. Load .pfx from file;
2. Load certificate with private key from the system’s certificate store.

Here is the first one, load from file:

HttpWebRequest objRequest = System.Net.HttpWebRequest.Create(url) as HttpWebRequest;
X509Certificate2 clientCertificate = new X509Certificate2("TestClient.pfx", "the key password");
objRequest.ClientCertificates.Add(clientCertificate);

You have to add certificate to the https request right after you created the request, because GetResponse() will use the certificate immediately. Here is the second way of creating X509Certificate2 - loading the certificate from the system store:

static X509Certificate2 LoadClientCertificate()
{
    // Note: Change "My" and StoreLocation.CurrentUser to where your certificate stored.
    var store = new X509Store("My", StoreLocation.CurrentUser);
    var certificates = store.Certificates.Find(X509FindType.FindBySubjectName, "TestClient", true);
    if (certificates.Count != 0)
    {
       return null;
    }

    return certificates[0];
}

Before running it, you have to import the certificate (with private key) to the store just like you did with the server certificate. Loading a certificate (without private key) can be done by a non-privileged account, while accessing private key of a certificate from the system store requires administrator privilege. So when you run above code by a non-privileged account, you will get the certificate although, but only public key is in it. While the server side needs the client to sign something to verify the client’s identity, so the client must have the private key. So when carrying out further steps of HTTPS communication

1. When the client certificate loaded from system store, the client code will get an WebException;
2. When the client cerfiticate loaded from file,  the server will get no client cert (GetClientCertificate() returns null).

Loading from store and loading from file both has pros and cons.

STEP 8

Server side still doesn’t verify the client certificate. So let’s add the code logic.

HttpListenerContext context = server.GetContext();

var clientCertificate = context.Request.GetClientCertificate();
X509Chain chain = new X509Chain();
chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
chain.Build(clientCertificate);
if (chain.ChainStatus.Length != 0)
{
    // Invalid certificate
    context.Response.OutputStream.Close();
}

X509Chain is a tool which builds the chain of trust of the certificate. If the certificate is invalid, then you can find error information in chain.ChainStatus. You can implement detailed logic upon X509Chain rather than only checking chain.ChainStatus.Length. Set RevocationMode to NoCheck because we don’t have a certificate server to tell you whether a certificate is revoked.

Render English and Chinese with mono font

(This is an unresolved issue)

How to make a Chinese character exactly 2 English characters wide?

I've tried it in WPF application with Courier New and Segoe UI Mono, but both failed. And also tried SimSun as fallback font, failed again.

Here is the summary of my test.
Successful Examples
Notepad
Notepad++
HTML with pre tag rendered by Chrome

Failed Examples
Visual Studio 2010
HTML without pre tag rendered by Chrome
HTML with and without pre tag rendered by IE

I tried Courier New and Segoe UI Mono with fallback font SimSun in my WPF app. The fallback font mechanism worked, because it can render STLiti which is 隶书, see below

One Chinese char is a little bit less than 2 English chars in width. when fallback font is SimSun, it's the same case

Figure 1

Visual Studio has the same issue

Figure 2

While Notepad++ is pretty successful

Figure 3

Different web browser has different behavior. With
<pre> tag would work well in Chrome while it can't give you Chinese char of 2 English chars width without
<pre>. And IE can't do it in both cases.

As you can see Chinese chars in Figure 3 is a little bit wider than Chinese chars in Figure 1. Is there a covered mechanism results narrow Chinese chars?

Config Test Deployment for VS2010

The MSDN document How to: Config Test Deployment is somewhat out-dated. .testrunconfig in VS2010 is actually local.testsettings. Another config file 'TraceAndTestImpact.testsettings' really fooled me. ^_^

Mean-shift Object Tracking Study Notes

Xueqing Sun

sunshaking@gmail.com

June 12^th, 2011

License CC BY

You can distribute, remix, tweak, and build upon this work, even commercially, as long as you credit Xueqing Sun for the original creation.

I was reading a book, ‘Image Processing, Analysis, and Machine Vision, Milan Sonka, Vaclav Hlavac, Roger Boyle’ for studying mean-shift object tracking algorithm. But the book was really hard to understand, especially for a beginner of machine vision. Yesterday, with the help of Guangwei, a friend of mine (he is a scientist at National Astronomical Observatories, Chinese Academy of Science), I finally understood the algorithm. And I would like to explain what I understood for sake of helping other guys who are still struggling on this algorithm.

Heads up Questions

1. What is the probability density function of an image?

2. How does the density function contribute to target model?

3. What’s kernel function? How does it work?

 

Here is the summary of the algorithm

To locate a target on an image, we must have a target model, an image which contains the target. The goal of mean-shift algorithm is to tell you the position of your target on the image. Suppose we have a target model image with 5*5 pixels. The pixel values listed below:

x	y	pixel
0	0	1
0	1	1
0	2	6
0	3	5
0	4	7
1	0	0
1	1	4
1	2	7
1	3	0
1	4	4
2	0	3
2	1	5
2	2	7
2	3	1
2	4	5
3	0	7
3	1	0
3	2	9
3	3	9
3	4	2
4	0	0
4	1	1
4	2	4
4	3	3
4	4	3

Target model image

The row x and y are the location of a pixel on the image. And the third row ‘pixel’ is the value. For convenient purpose, we limit pixel value from 0 to 9. Now we can create a frequency table like below. Normally images are colored, but we simplified the problem from colored images to gray scale images. A colored image can be considered as a composition of three gray scale images. So the rationale explained here can be easily applied to colored images.

bin	q
1	8
2	1
3	3
4	3
5	3
6	1
7	4
8	0
9	2

Frequency

The frequency means how many pixels located in a specific bin. For example, the first row means there are 8 pixels which values are less or equal to ‘1’. Furthermore we can have histogram chart for the table.

Histogram

On the book, and many other materials, the density function of target model, defined as below:

1

The density function is closely related with the frequency table. Here the black is a vector which has m components. In this case, the m is 9 because we have 9 bins. Each component is made from corresponding bin. Let look into the component definition.

2

The definition looks somewhat complex. Let’s split it into parts so we can understand it easier. First, let’s see the delta part

3

n is the pixel number. In this case we have 25 pixels. So the n is 25. is the coordinate of i^th pixel in the image. For example, the first pixel coordinate is (0,0), and the second is (0,1). u means we are currently defining u^th component corresponding to u^th bin. Function tells which bin the pixel go. For example, the first pixel (0,0) is 1, so it goes to bin 1, i.e. .

Actually, the formula 3 is a vector, let’s rewrite it as following:

4

The delta function gives you 1 if the pixel goes to bin u, otherwise it gives 0. So this vector works just like a mask. On the mask there are ‘1’s mark the pixels go to bin u. For example, if our image is 4*4, we might get a D vector like this:

1	0	0	1
0	1	1	0
0	0	0	1
1	0	0	0

The image is a matrix. D is a vector. Just put each row of the matrix to one row in sequence then we’ll get D.

IMPORTANT

What’s the relationship between D and the frequency table? The answer is for each u, . Actually we can form feature space only based on D, because D already represents the image in statistics way. So mean-shift algorithm is so called an algorithm that based on probability density function. But marginal pixels are unstable, so the kernel function which is a weight function does the work. It give marginal pixels less weight to make the objective function in optimization process more stable.

 

OK, let’s see another part of formula 2. It’s

5

Again it’s a vector. Function k is the kernel function which actually is a weight function giving each pixel position a weight. So can be defined as below:

6

It looks beautiful. I really hate which makes formula unreadable.

OK, now we get the density function of target model image. The candidate density function is pretty similar.

	7
	8

Formula 8 is easier to understand because we’ve already understood formula 2. The k function in formula 8 is just a transformed version of original k.

I think I’ve solved the hardest part of the mean-shift algorithm. And the rest of it would be much easier to understand because it is just kind of an optimism algorithm to find the y.

High Resolution Timer via Interop

Though Windows isn’t a real-time OS, we do need high resolution clocks and timers sometimes. .Net gives us Stopwatch class which is really a high resolution clock. But we can’t find a high resolution timer in .Net framework. A timer is different from a clock, which fires event periodically notifying you the period of time is up. This article is mainly to demonstrate the basic usage of interop. And a more complex and intensive material provided at the end of the article.

There are high resolution timer services on Windows. But they are native APIs. Fortunately, we can use interop to call these native APIs.

The first thing you have to do is to determine the device capabilities. Different devices have different resolutions[1]. Some may give you 1 ms of resolution, but some may be 10 ms. And if you request a resolution that the device can’t support, an error will occur. To determine the resolution of the timer hardware, use timeGetTime().

[DllImport("winmm.dll")]
private static extern int timeGetDevCaps(ref TimerCaps caps, int sizeOfTimerCaps);

TimerCaps is a struct. A C struct actually. We can use C# struct to map the data.

[StructLayout(LayoutKind.Sequential)]
public struct TimerCaps
{
    /// <summary>
    /// Minimum supported period in milliseconds.
    /// </summary>
    public uint periodMin;
    /// <summary>
    /// Maximum supported period in milliseconds.
    /// </summary>
    public uint periodMax;
}

Please keep in mind that “LayoutKind.Sequential” is essential, because it suppressed C# compiler from alter the order of the fields for optimization purpose. Here for the parameter “TimerCaps caps” we use “ref”, because we want the API to write some data into “caps”. The “ref” maps the type to the pointer of the type in C/C++. So “ref TimerCaps caps” means “TimerCaps *caps” in C/C++.

Now we can call the API.

TimerCaps caps = new TimerCaps();
int ret = timeGetDevCaps(ref caps, Marshal.SizeOf(caps));

And then we can start the timer.

[DllImport("winmm.dll")]
private static extern int timeSetEvent(uint delay, uint resolution, MmTimerCallback proc, int user, int mode);
...
actuallPeriod = Math.Max(caps.periodMin, Period);
timeBeginPeriod(actuallPeriod);
timerId = timeSetEvent(1000/*delay*/, 0/*resolution*/, TimerCallback, 0, 1);

“MmTimerCallback” is a delegate method mapping to callback function in C/C++. That’s almost the complete stuffs of the high resolution timer. Please be aware, the callback function will be called in a thread other than your main thread. You have to delegate the operation to your main thread if you want to access UI elements.

Here is a good read material talks about interop.

• Calling Win32 DLLs in C# with P/Invoke - Jason Clark

The Way of Quitting an Application

There are several ways of letting an application to quit programmatically. But you should choose the best one among them.

Environment.Exit Method

This method will force a process to terminate. So the process has no chance to do cleaning up work. No code after Environment.Exit will be executed. Any data that is unsaved will be lost. Not recommend this way.

Process.Kill Method

Killing a process just like Environment.Exit, terminates the process immediately and the process doesn't have the last chance to breath. So again, it's not recommended.

Process.CloseMainWindow Method

This method will send a request to a process' message loop, if the process does have a message loop. The process thus has a chance to do cleaning up work, and save unsaved data to disk then terminates gracefully. This method can be called either from another process or within the process that to be terminated.

Environment.Exit Method http://msdn.microsoft.com/en-us/library/system.environment.exit.aspx

Process.CloseMainWindow Method http://msdn.microsoft.com/en-us/library/system.diagnostics.process.closemainwindow.aspx

Tricks for repairing Visual Studio if it doesn't work with unit test

Sometimes, Visual Studio doesn't work with unit test. Unit test list can't be loaded when you try to load an old unit test project. Or, it can load the list, but when you try to run the unit test case, Visual Studio just tell you it is not able to run the unit test, blah blah blah...

There must be bugs in Visual Studio or in your unit project settings. Please try the following, the easiest one first:

1. Delete .suo, .vsmdi of your solution then reload your project.
2. Type devenv /setup or devenv /resetuserdata. But please be careful, it will clear your user setting data and Visual Studio will start from a factory state.
3. Check if any references of your unit test project does not work. Just click on the reference, if Visual Studio will bring you to the correct position of the Object Browser, it's the correct reference. Otherwise, not. To fix the reference, first remove the reference, then add it again. If you can't find the reference, probably you have to reinstall the relevant software. The issue I met is TeamSystem.Data.UnitTesting won't work. And the attribute TestMethod is undefined because it is defined in the assembly that hasn't been referenced correctly from the unit test project.