Transferring large data over the wire using WCF

WCF (Windows Communication Foundation) is a secure, reliable, and scalable messaging platform for developing services in .Net. It provides support for two modes of transferring data: the buffered mode and the streamed mode. While in the buffered mode (this is the default mode supported by WCF) the data is stored in the buffer in its entirety until the data transfer is complete. In the streamed mode, the data is read in chunks, and the receiver of the message can start processing the message until the transfer is complete.

While the buffered mode is the default and needs the message to be sent to the receiver in its entirety before the message can be processed at the consumer’s end, the streamed mode is useful for messages that are large in size. Processing large chunks of data requires a large buffer and a lot of available memory both at the client and also at the server side, hence you need to use streamed mode, which improves the scalability of the service as you don’t need to have large memory buffers to hold data.

To enable the any of the modes described above, you need to take advantage of the TransferMode property. This property can have any one of the following values.

1. Buffered — enables buffered mode
2. Streamed — enables streamed mode of bi-directional communication
3. StreamedRequest — enables streamed mode for the request only
4. StreamedResponse — enables streamed mode for the response only

Note that the HTTP, TCP/IP, and named pipe transports use buffered transfers by default and the streamed transfer mode needs to be turned on explicitly in the service configuration file. The streamed mode is supported only by the basicHttpBinding, netTcpBinding, and netNamedPipeBinding bindings; the TransferMode is an enum defined in the System.ServiceModel namespace.

using System;

namespace System.ServiceModel

{

    public enum TransferMode

    {

        Buffered = 0,

        Streamed = 1,

        StreamedRequest = 2,

        StreamedResponse = 3,

    }

}

The following code snippet shows how the TransferMode property can be set in the service configuration file.

<system.serviceModel>

  <services>

    <service name="IDG.LargeDataUpload">

      <endpoint address=""

                binding="basicHttpBinding"

                bindingConfiguration=" customStreamedBinding"

                contract="IDG.ILargeDataUpload"/>

    </service>

  </services>

   <bindings>

    <basicHttpBinding>

      <binding name="customStreamedBinding"

                transferMode="Streamed"/>

    </basicHttpBinding>

  </bindings>

</system.serviceModel>

The default message size is 64K (the maximum size it can support is 64MB), but you should increase the message size if you need to use streamed mode and transfer large chunks of data. Here’s how you can configure the message size in the service configuration file.

<basicHttpBinding>

<binding name="customStreamedBinding" transferMode="Streamed"

closeTimeout="00:01:00" openTimeout="00:01:00"

receiveTimeout="00:10:00" sendTimeout="00:01:00"

transferMode="Streamed" messageEncoding="Mtom"

maxBufferPoolSize="524288"

maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">

<security mode="None">

<transport clientCredentialType="None"/>

</security>

</binding>

</basicHttpBinding>

At the client side, you should set the maxBufferSize, maxBufferPoolSize, and the maxReceivedMessageSize properties appropriately. You should also set the appropriate endpoint behavior in the service configuration file as shown below.

<endpointBehaviors>

            <behavior name="IDGEndPointBehavior">

                <dataContractSerializer maxItemsInObjectGraph="2147483647" />

            </behavior>

</endpointBehaviors>

You should also set the maxRequestLength property to a higher value to ensure that the request persists for a longer period. The code snippet that follows shows how you can configure this in the service configuration file.

<system.web>   

<httpRuntime maxRequestLength="102400" />

</system.web>

Note that you should enable the MTOM message encoding by setting the “MessageEncoding” property to “Mtom” when sending large data over the wire – this message encoding would also facilitate interoperability.

Now, let’s understand the downsides of streaming too. The streamed mode isn’t that secure (it doesn’t support reliable messaging) and is not that scalable for high traffic situations. A better solution is chunking — it can support reliable messaging and can support both message and transport security. Note that you can secure the communication between your WCF service and client by using either Transport Security or Message Security. While the former is used to enforce security at the transport level, the latter is used to encrypt the message that is passed between the WCF server and the client. Hence, chunking is an ideal solution for transferring huge data over the wire using WCF in high data traffic scenarios. You can learn more on this from this MSDN article.