TCP performance differences between RH Linux and Solaris in java?

While comparing java TCP socket performance between RH Linux and Solaris, one of my test is done by using a java client sending strings and reading the replies from a java echo server. I measure the time spent to send and receive the data (i.e. the loop back round trip).

The test is run 100,000 times (more occurrence are giving similar results). From my tests Solaris is 25/30% faster on average than RH Linux, on the same computer with default system and network settings, same JVM arguments (if any) etc.

I don't understand such a big difference, is there some system/network parameters I am missing?

The code used (client and server) is shown below if anybody is interested into running it (occurrence count has to be given in command line):

import java.io.*;
import java.net.*;
import java.text.*;


public class SocketTest {

public final static String   EOF_STR = "EOF";
public final static String[] st      = {"toto"
    ,"1234567890"
    ,"12345678901234567890"
    ,"123456789012345678901234567890"
    ,"1234567890123456789012345678901234567890"
    ,"12345678901234567890123456789012345678901234567890"

    ,"123456789012345678901234567890123456789012345678901234567890"
};

public static void main(String[] args) throws UnknownHostException, IOException, InterruptedException {
    double mean = 0.0;
    int port = 30000;
    int times = Integer.parseInt(args[0]); 
    String resultFileName = "res.dat"; 
    new EchoServerSimple(port);        // instanciate and run
    Socket s = new Socket("127.0.0.1", port);        

    s.setTcpNoDelay(true);        
    PrintWriter pwOut = new PrintWriter(s.getOutputStream(), true);
    BufferedReader brIn = new BufferedReader(new InputStreamReader(s.getInputStream()));
    long[] res = new long[times];

    int j = 0;
    for(int i = 0; i < times; i++) {
        if(j >= st.length) j = 0;            
        long t0 = System.nanoTime();
        pwOut.println(st[j++]);

        brIn.readLine();
        res[i] = System.nanoTime() - t0;
        mean += ((double)res[i]) / times;
    }
    pwOut.println(EOF_STR);
    s.close();
    print(res, resultFileName);
    System.out.println("Mean = "+new DecimalFormat("#,##0.00").format(mean));
}


public static void print(long[] res, String output) {
    try {        
        PrintWriter pw;
        pw = new PrintWriter(new File(output));            
        for (long l : res) {
            pw.println(l);
        }
        pw.close();
    } catch (FileNotFoundException e) {
        e.printStackTrace();

    }
}

static class EchoServerSimple implements Runnable {

    private ServerSocket _serverSocket;

    public EchoServerSimple(int port) {
        try { _serverSocket = new ServerSocket(port); } 
        catch (IOException e) { e.printStackTrace(); }

        new Thread(this).start();
    }

    public void run() {
        try {
            Socket clientSocket = _serverSocket.accept();
            clientSocket.setTcpNoDelay(true);    
            PrintWriter pwOut = new PrintWriter(clientSocket.getOutputStream(), true);
            BufferedReader brIn = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));                
            try {

                while(true) {
                    String s = brIn.readLine();
                    pwOut.println(s);
                    if(s.equals(EOF_STR)) { clientSocket.close(); break;    }
                }
            } catch (Exception e) {
                e.printStackTrace(); 
                try { clientSocket.close(); } catch (IOException e1) { e1.printStackTrace(); }
            }
        } catch (IOException e) {e.printStackTrace(); }

    }
  }
}

I'm using the JRE 1.6.0_18 for both OS, on a single 2.3GHz dual core Nehalem.
The 2 OS ar Solaris 10 and RH Linux 5.4 with RT Kernel 2.6.24.7.

Thanks a lot.

Answer

In Solaris that's called "TCP Fusion" which means two local TCP endpoints will be "fused". Thus they will bypass the TCP data path entirely.

Try to disable it and run your test again:

# echo do_tcp_fusion/W 0 | mdb -kw
do_tcp_fusion:  0x1             =       0x0

You should probably try to create a test environment which mimics your production as close as possible. That probably means setting up a real network with network adapters.

If you want to play with connection throttling or other complex network situations, I suggest you place a FreeBSD box between your two endpoints and play with ipfw/dummynet of pf/altq.