How to create a non-caching fast nginx reverse-proxy?

Nginx should be fast - at least people say so.

I can't get nginx fast. For benchmarking, i use siege in multiple configurations for simulating high loads. So i tried to load-balance a single server (i know load balancing a single server is rather useless, but for testing the load-balancer itself this is valid). I've set up an already optimized apache reverse proxy as reference. I get around 80tps via nginx and around 350tps using apache when i try getting the same - not cached - file from the same backend server. Of course the hardware/os is the same (current dual-core cpu, 2G ram, Ubuntu Server 16.04).

I tried already changing workers, maximum connections, poll-methods, buffer sizes for proxying, waiting times and buffers for clients. I can see a low load on the system, one nginx uses around 1 percent of the CPU and connections will wait sometime around 5 to 6 seconds. As i want to measure the performance of the reverse proxy i do not want to cache anything in this test.

So the question is: How to optimize the Performance of nginx as non-caching reverse proxy?

Example siege-command: siege -c 100 -b -r 100 -v /favicon.ico

Update: A bit of config, as requested. For keeping the question serverfault-conform, please answer generally about useful about parameters.

user www-data;
worker_processes 2;
pid /run/nginx.pid;
#thread_pool default threads=1500 max_queue=65536;
worker_rlimit_nofile 40000;


events {
    worker_connections 768;
    #multi_accept on;
    #use epoll;
}

http {

    #aio threads=default;

    #proxy_buffers           32 4m;
    #proxy_busy_buffers_size     25m;
    #proxy_buffer_size 512k;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;


    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;


    access_log /var/log/nginx/access.log combined buffer=1k;
    error_log /var/log/nginx/error.log;

    gzip off;
    # gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;

    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    upstream kronos-backend {
        server kronos.example.com;
    }

    server {
        listen 80;
        listen [::]:80;


        ssl_certificate     /etc/ssl/certs/snakeoil.crt;
        ssl_certificate_key /etc/ssl/private/snakeoil.key;

        listen 443 ssl;
        listen [::]:443 ssl;

        keepalive_timeout 60;

        root /var/www/vhosts/default;


        index index.html;

        server_name ;

        server_tokens off; # Don't show that nginx is bringing out the website.

        location / {
            proxy_set_header Host            ;
            proxy_set_header X-Forwarded-For $remote_addr;

            proxy_pass http://kronos-backend;

            client_body_buffer_size 1M; # Lets keep all client-sent bodies in memory if less than 1 Megabyte. (Default for amd64: 16K)
            client_max_body_size 10M; # Maximum accepted body is 10M (Default is 1M).
        }
    }
}

This config is not very clean, i know. But it shows more or less what i already tried. It is derived from the default configuration from the nginx-package in Ubuntu.

The connection takes the following way:

       +--- apache2 ---+
       |               |
siege -+               +--> some webserver (not of interest)
       |               |
       +--- nginx -----+

So the data transmitted via nginx or apache2 does not matter as long there is no cache involved and the same url (in this case a static file) is used for testing. The caching mechanism was not activated intentionally at any of the webservers.

The config in apache uses mpm_worker.

ThreadLimit          600
StartServers         4
ServerLimit          11
MinSpareThreads      50
MaxSpareThreads      100
ThreadsPerChild      200

MaxClients           1000
MaxRequestsPerChild  20000

Example outputs of the above siege command:

$ siege -c 100 -b -r 10 /wrapper_2_bg.png
** SIEGE 3.0.5
** Preparing 100 concurrent users for battle.
The server is now under siege..      done.


Transactions:                   1000 hits
Availability:                 100.00 %
Elapsed time:                  11.01 secs
Data transferred:               0.12 MB
Response time:                  0.84 secs
Transaction rate:              90.83 trans/sec
Throughput:                     0.01 MB/sec
Concurrency:                   76.24
Successful transactions:        1000

Failed transactions:               0
Longest transaction:            6.67
Shortest transaction:           0.03


$ siege -c 100 -b -r 10 /wrapper_2_bg.png
** SIEGE 3.0.5
** Preparing 100 concurrent users for battle.
The server is now under siege..      done.


Transactions:                   1000 hits
Availability:                 100.00 %
Elapsed time:                   3.16 secs
Data transferred:               0.12 MB
Response time:                  0.22 secs
Transaction rate:             316.46 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   68.95
Successful transactions:        1000
Failed transactions:               0

Longest transaction:            3.06
Shortest transaction:           0.05

Answer

Packet Loss was the source of the problems. Between the load balancers and the backend web server was a router. The router does not seem to be fast enough as it's load goes to 100% while testing. As consequence packets get lost.

I don't know why apache seems to handle this better. Moving the load balancers to the same network as the delivering webserver, the performance is nearly the same via both ways and much higher.