How to reduce server response time for a magento e-commerce site

We have an e-commerce site developed using magento 1.9.0.1, we could not able to reduce server response time to below one sec, (as per google page speed testing tool we have now 1.4 sec) we have tried almost all optimization process as below

1 - Using magento full page cache from well reputed company

2 - Added gzip compression

3 - Using cloudflare as CDN

4 - tried magento flat catalog table and disable unwanted indexes and make it indexing manually.

Server and traffic details

Location - UK

Sessions/M: 17929

Unique users/M - 14860

Page views/M - 58323

Bandwidth usage/Month - 61.09 GB

Total processors: 8

Processor #1 x 8

Vendor

GenuineIntel

Name
Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz

Speed

2000.050 MHz

Cache
20480 KB

RAM - 8 GB

Current Memory Usage

          total       used       free     shared    buffers     cached


Mem:       8027740    7674716     353024      26052     928848    2992492
-/+ buffers/cache:    3753376    4274364
Swap:      1048572     210260     838312
Total:     9076312    7884976    1191336

Operating system Linux

Mysql Database size - 3782 MB

we have more than 40K SKUs

We have similar site running in USA with different name and its server response time is good, but it has less traffic.

Can anyone help/suggest some solution to reduce server response time?

Thanks
Prajosh