TCP Fast Open implementation?

[icylogic]

Proposal: TCP Fast Open

TCP Fast Open is an extension for TCP to speed up the handshaking process in successive TCP connections. This feature has been available for OSX and Linux for years and was implemented on Windows by Anniversary Update. Users could check by netsh

>netsh int tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
...
TCP Fast Open : enabled

but dotnet has not implemented this for System.Net yet.

Rational and Usage

This new API is important and powerful because when it is used with TLS False Start, we are down to 1-RTT from original 3-RTT for a successive TCP+TLS (HTTPS) connection, and furthermore, 0-RTT could be achieved by combining TFO and TLS 1.3.

And it's easy to use: programmers only need to add a few lines to set an socket option and provide a buffer filled with data to send. Here's the usage:

var s = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
var e = new SocketAsyncEventArgs();
s.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.FastOpen, true);
e.SetBuffer(buffer, offset, length); 
s.ConnectAsync(e);

Note that buffer[offset] ~ buffer[offset+length-1] might be (partially) sent together with SYN. It's up to the OS how to send.

A typical usage is the TLS handshake. In successive connections, the client could send a SYN with TLS Client Hello.

Please check the "Original Proposal" for details.

Proposed API

namespace System.Net.Sockets
{
 // Defines socket option names for the <see cref='System.Net.Sockets.Socket'/> class.
 public enum SocketOptionName
 {
 // ...

 #region SocketOptionLevel.Tcp
 // ...
 FastOpen = 15
 #endregion

 }
}

This enum value is 15 for Windows according to ws2ipdef.h.

Problems & Extra works

• Implementation
  In fact, adding just one line like the code above in SocketOptionName.cs would work for Windows.
  However, it might not work for Unix. Since dotnet simply pass the socket option to native API on Windows while the Unix Pal would check the value. For more information about the difference please check the relevant issue dotnet/corefx#14237 . To make this work for Unix we may need to modify this part

• Compatibility
  According to MSDN, this option is available since Windows 10, 1607. Maybe we should generate a warning if this option is set on the older Windows platform

• Other System.Net API
  For API using sockets in its code such as WebClient or HttpClient, it could take advantage of this feature, especially for "https://" scheme.

Original Proposal

TCP Fast Open on Windows

Windows has introduced TCP Fast Open in the Anniversary Update serveral months ago. (Announcing: New Transport Advancements in the Anniversary Update for Windows 10 and Windows Server 2016)

TFO is one full Round Trip Time (RTT) faster than the standard TCP setup which requires a three way-handshake. This leads to latency savings and is very relevant to short web transfers over the Internet where the average latency is on the order of 40 msec.

Native API for TCP Fast Open

This feature could be turned on using native Windows Socket 2 API: IPPROTO_TCP Socket Options. But it has not been implemented in dotnet (SocketOptionName.cs).

dotnet API relating to TCP Fast Open

According to the Native API Documentation, in addition to passing a new enum value to the method setSocketOption which then calls the native setsockopt, we need to set up the buffer including the data to send during the initial connection:

Note that to make use of fast opens, you should use ConnectEx to make the initial connection, and specify data in that function's lpSendBuffer parameter to be transferred during the handshake process. Some of the data in lpSendBuffer will be transferred under the Fast Open protocol.

The native method ConnectEx is called by DoOperationConnect which is called by socket.ConnectAsync on Windows. And the buffer could be set as a property of SocketAsyncEventArgs.

//TFO is defined as 15 in ws2ipdef.h
socket.SetSocketOption(SocketOptionLevel.Tcp, (SocketOptionName)15, true); 
e.SetBuffer(buffer, start, length); //e is a SocketAsyncEventArgs
socket.ConnectAsync(e);

dotnet implementation

The most straightforward way to implement this feature is to add a new value to the SocketOptionName enum. But it enables this feature only on Windows. In this issue https://github.com/dotnet/corefx/issues/14237, it seems that members are interested in introducing a new cross-platform API for TCP options.

It would be nice if the Public API contained all of the possible socket options across Windows, Linux, and Darwin.

PS. TCP Fast Open is natively supported on Linux and OSX. Here's how to use it on Linux: TCP Fast Open: expediting web services

More about TCP Fast Open:

• RFC7413
• Windows blog: Building a faster and more secure web with TCP Fast Open, TLS False Start, and TLS 1.3

[geoffkizer]

I agree, we should do this.

[karelz]

Next step: We need formal API proposal

[icylogic]

Glad you are interested in this API. I could follow the guideline to file a formal proposal so we can discuss details. Do I need to open a new issue like "Proposal: TCP Fast Open" or edit this issue?

[karelz]

Just edit this one. I often keep the original text in "Original Proposal" section or "Details" as GH doesn't version issue text.
Ping the issue whenever you update top post, summarizing the changes - it is easier to keep track of what happened.

[icylogic]

Added a proposal including a short introduction, an example and some problems while leaving the original text unchanged in the section "Original Proposal"

[karelz]

Sounds great, thank you!

Re: Compatibility: What happens when you pass the flag on lower Windows version or Linux/OSX version which does not support it? Is it no op? Or failure?

@geoffkizer @davidsh @CIPop @Priya91 any concerns/thumbs up? If I get 2 thumbs up, I can mark it as ready for API review on Tue.

[geoffkizer]

I don't think we prevent you from passing arbitrary ints to SetSocketOption. So something like this should work today:

s.SetSocketOption(SocketOptionLevel.Tcp, (SocketOptionName)15, true);

[icylogic]

@geoffkizer Yes, it does work for Windows as I stated in the original proposal. So for Windows after Win10 1607, adding one line to the enum to formalize this API is enough. I'm just not sure what will happen for Unix and Windows before 1607.

[geoffkizer]

My point is that we don't need to make any product change for users to do this today.

It might be nice to add the SocketOptionName for this, but that raises the question of whether and how we want to expose OS-specific socket options in SocketOptionName.

[icylogic]

OK, if we don't make any change, we can still use this feature on recent Windows. For programmers developing on other platforms which natively support TFO, they may need to wait for this issue https://github.com/dotnet/corefx/issues/14237#issuecomment-266035007 to be solved to use these socket options. It seems that corefx team is interested in introducing a new set of API.
Another issue is about other System.Net API, like WebClient or SslStream, which provide TLS connections and could obviously use this feature. If we don't want make any change to these API using this unexposed feature today, we could wait for TLS 1.3 to come out and make changes together.

[geoffkizer]

I think we could certainly consider other APIs, e.g. some sort of way to enable this in a platform-neutral way.

I'm reacting to the "Proposed API" above, which proposes to add a Windows-specific enum value for FastOpen. My point is that this is a convenience, not a necessity, since you can achieve the desired functionality by just passing (SocketOptionName)15.

[icylogic]

That makes sense. It's just the most direct way came to me but actually not a good way.
Maybe we could expose some other APIs that does not take a platform-specific value and translate it in the PAL(Platform Adaptation Layer) like this

[stephentoub]

I think we should do two things:

• Add the enum value and appropriately implement it on both Windows and Unix. I'm not sure why the enum value is described as being specific to Windiws... on Unix we'd translate it in the PAL into the right platform value as we do with all other options.
• Add the ability to pass option names in a platform-specific way so that we have a workaround available in the future for other such options. While on Windows it would work to just cast the number to the enum, on Unix we go through a translation layer, so we'd either misinterpret the value if it had an existing match in the enum, or we'd throw for it if the value wasn't known, neither of which is right. We've had a discussion of this in another issue, but I'm on my phone and can't easily find it at the moment. Basically you want the ability to say "pass this exact value to the OS, don't try to interpret/translate it", and leave it up to the app to do the platform detection and determine the right value to use.

[geoffkizer]

Add the enum value and appropriately implement it on both Windows and Unix. I'm not sure why the enum value is described as being specific to Windiws... on Unix we'd translate it in the PAL into the right platform value as we do with all other options.

Is this something we want to continue to do, though? I realize we did this for compat initially, but it seems like we could get into trouble here. Would we be better off adding a "bool FastOpen" property on Socket, not unlike NoDelay?

+1 to having a way to do platform-specific options names.