Those things may seem a bit obvious, but there are important implications when monitoring particular types of software. Here are some tips about what you should monitoring for each:
When monitoring websites or web-applications you want to be able to monitor a complete navigational path, just as a user would navigate. The data you want to look at includes each "step" in the path and the response times for each call. This could be just one page, or many pages.
- DOM Load: The time it took to complete parsing the page's documents. Note: referenced style-sheets, images and sub-frames may not have completed loading.
- Page Load: The time it took for the page's documents (all referenced objects and scripts, style-sheets, images) to be fully loaded into the browser. Note: This does not account for load time of dynamic content such as AJAX.
- Perceived User Experience (PUX): First Paint The time it takes for the page to start rendering then the time it takes for all visible content rendered on the page to "stabilize" above the fold
Monitoring Native Mobile Applications
Mobile websites need to be monitored in the context of their performance over mobile carriers and various devices. For monitoring mobile applications you want to measure the same things as web applications, however you need to be able to look at the performance
- Over a mobile carrier
- On an assortment of device configurations
- Possibly on real devices and local carriers
Web and mobile applications are usually built using component architecture with APIs gluing it all together. If you create applications with APIs you need to monitor them specifically as their performance is crucial to the application as a whole. If you produce APIs for others, they are counting on you to provide well performing APIs.
You can monitor an API in the test environment and then move to monitor in production. Monitoring in the test environment establishes a baseline for performance. Monitoring in production is crucial because the environment changes and you can measure your performance against your established baseline. API monitors should provide the following performance metrics:
- Response time for all calls made
- Data validation as requested
This information ensures that the API is not only available but that the data is correct.
Monitoring Applications in the Cloud
There are many business and technical considerations for migrating to today's cloud computing options. If you are currently managing your online assets "in the cloud", you already understand how crucial user experience is - it can make or break you.
Understanding how your PaaS, IaaS and SaaS suppliers are impacting availability and performance is key to your success with cloud computing. But for cloud vendors, especially those in industries such as retail that experience seasonal surges, traditional load testing may not uncover key scalability issues. You should monitor your cloud based applications for:
- Identify from the end-user perspective how well cloud service providers deliver performance across different geographies
- Troubleshoot performance and availability bottlenecks in the service delivery chain
- Independently validate vendors SLA claims with real data
- Determine how well and quickly vendors scale during peak traffic periods
What To Look For In A Performance Monitoring Tool
All of the monitoring activity above is driven by tests or scripts which can be created by various methods, coding, importing existing test scripts or recording new scripts without coding.
It is important to have an easy method to create tests or scripts since they may need to be changed often- whenever changes are made to the applications or environment. A web recorder that creates scripts for you without coding- is the easiest method to create web and mobile application scripts. The ability to reuse functional tests already created for APIs would be the easiest method to create API tests
The Ability to Monitor from Wherever your customers are:
Your customers may be geographically dispersed around the globe, or just a couple cities, or accessing only inside your network. In order to understand performance you have to monitor from where they are both inside and outside of your network.
Deploy monitoring tests from outside the firewall
In order to get accurate information about response times that a customer would see, it is necessary to test from locations as close to where customers are as possible. The ability to deploy monitoring tests from a wide variety of locations is important, it is also important to test from outside the same data center as your application may be deployed. While testing from the amazon cloud locations is good, if your application is also in the amazon cloud, the data may not reflect what an end user would see for response times.
Deploy monitoring tests from inside the firewall
Some critical APIs and applications operate entirely from within a private network and can not be exposed to the public internet for privacy or security reasons. It is just as crucial to keep an eye on these applications as externally facing ones. For that reason you would want to consider deploying monitoring inside your private network.
Methods to test a single action or a multistep transaction
Test against a single website URL
Any monitoring system should allow you to quickly and easily test a single URL such as a home page- this allows you to keep an eye on a lot of things that might not be critical in a transaction but shouldn’t be left unchecked.
Test against an API endpoint
A simple endpoint test can be valuable – to make sure that an API is available, without needing to test the entire functionality. There should be an easy method to monitor API endpoints.
Test complete web transactions
It is crucial to be able to test a complete navigational path of a web application, or transaction to understand if what a person wants to get done can get done. As an example: If you test just the page which displays the sale item on it, but not putting the item in the cart and purchasing it, you will not know if someone can actually buy the sale item.
Test complete set of API calls
While testing the endpoint of an API to see if it is available is valuable, you may need to test each of the calls made by the API to ensure that transactions can be completed. Just as it is important to test complete web applications. APIs are part of transactions and if there is an error in correctness or any degradation in performance, having the data on each piece of the transaction makes troubleshooting faster.
Ability to test various methods of access- devices, carriers.
Test mobile device configurations
Mobile devices change all the time, and there are many of them. Your monitoring system should have the ability to test a variety of mobile device configurations, by emulating their characteristics.
Test over mobile carriers
Monitoring over actual mobile carriers provides important performance information that can not be derived from simply looking at a mobile device configuration.
Ability to reproduce what end users do
Monitor using commercial browsers
Customers use actual browsers to interact with web based applications. As introduced previously, the browser is where the applications come together. It is crucial – if you are looking to understand the performance of web applications for a customer- that you use real browsers to drive the monitoring. The measurements that are taken for response time of the objects on the page are only accurate when the monitoring uses commercial browsers.
Monitor Infrastructure or Services
There may be a number of other web services or infrastructure pieces in your system that would be valuable to monitor as well as your applications or APIs. These could include e-mail servers, FTP services or ping tests. Depending on your needs, you may want these capabilities included with your application monitoring so that everything is on the same operations dashboard.
Alerting and Reporting
All of this measuring and data gathering is pointless if you are not notified as quickly as possible if something is wrong. Alerts should be customizable, and delivered in the manner of your choice. It is also preferable that the alerts could be configured to work with any in place notification system. Equally important is that the system provides checks so that you are not inundated with false alarms.
Reports provide the opportunity to track trends, and share information. They should be customizable and able to be automated.
Any monitoring system you use needs to play nicely with the items already in the Stack in your operations center- even better, if they can interact with other systems and feed data to them or take data in- the synchronicity can streamline operations.