web applications - Training for load testing web apps?

We've discussed the tools used for load testing here on ServerFault, but what about training on how to use them properly? Are there companies that specialize in IT training that cover load testing? How do you properly come up with a simulated load? How long should you run the test for? What are the best metrics to be tracking on the server-side while the test is running? And so on...

Answer

1. First, start with the business representatives. They (should) know the application best. Identify the key transactions, and the end to end response times. Ideally, they'll be able to hand you a document which captures their non functional requirements. If your application is replacing a legacy application, all the better - get as many applicable usage metrics from that app as you can. This is the most critical success factor to performance testing. Understanding the size of your potential userbase, the number of users likely to be using it concurrently, the # % of each one of your key transactions executing simultaneously, growth rate per [timeframe].




2. Build an automated script which simulates the key transactions. Include think time in this script. Very few users are going to power through your application/website without having to take a few seconds to see what the app did in response to their input. Failure to adequately simulate think time can result in you subjecting your application to unrealistic load, which leads to unhappinesss all around. That being said, the business may identify that 10% of the userbase are power users, and you may want to deliver your load with 90% normal users, with 'normal' think time, and 10% power users, with faster, more aggressive think times.




3. Add your virtual users over a time period (ramp-up time) - don't go from 0-500 in 1 second, unless you will actually have that kind of load (sale starts at 9:00 AM!). It's good to understand how your application will behave under load spikes, but some apps may fail in these scenarios, which is only a problem if you're expecting that kind of load. Otherwise, you may find yourself spending a lot more money than what's required to support a load that may never come.




4. Factor in latency and network speed. For a stress test, it's great to have a gigabit Ethernet connection with less than 1 ms latency to your application, which you can use to push your application to determine when it will fail. In reality, though, your users aren't usually that close to your application - they're coming over all different types of network conditions.




5. Endurance testing - at least 24 hours is recommended, more if you can afford it. You want to capture what happens to your application when periodic batch processes run, like backups, antivirus definition updates, or even IIS app pool recycles (every 29 hours by default).




6. Understand the difference between performance testing and load testing. Load tests will generally show the perspective of the server. This isn't entirely true - many tools will show you the time a transaction takes in terms of TTLB - but most tools today don't reflect client-side rendering times, which are material in JS-heavy applications, or ones that use XSLT, for example.




7. Don't solely rely upon your automated test numbers - at least not starting on day one. Periodically manually validate the numbers you get back. Over time you can let this subside as you become more confident in your simulations.





8. Performance counters - every application will vary, but you won't go wrong starting with the four basic food groups - cpu, memory, disk i/o, network i/o. A list of my preferred counters is at ht tp://www.oneredlight.com/perf.config.txt. You can set your application up to log these counters to a 300 MB circular file with the following command line:
   logman create counter PERF -f bincirc -max 300 -si 2 --v -o "c:\perflogs\perf" -cf
   "perf.config". I've only tried these on windows 2008/IIS 7/SQL 2008, so your mileage may vary. I would also recommend reading ht tp://msdn.microsoft.com/en-us/library/ms998530.aspx, if your application is on the ms stack.

(apologies for the broken urls; new users cant post hyperlinks)