WEBSITE MONITOR, WEBSITE MONITORING SERVICE, WEBSITE SPEED TEST and PERFORMANCE MONITORING

  Service FAQ (Frequently Asked Questions)


Speed Measurement Issues
Can you recommend a minimal "adequate speed" for our website?
Why are your speed data different from those of other monitoring services?
Why does a second speed test usually show faster result?
Why do several speed tests on the same site sometimes show a widely different results?
What is "Speed Variation Index"?
Why the "Speed Variation Index" may be important to you?
What are "Slowdowns"?
Do you normalize the speed test to the size of the tested page?
How can I measure the speed of my Internet connection?

Monitoring Issues
Could timeouts result from problems with your monitor, rather than problem with my server?
How do I prevent your monitoring visits from interfering with my Google Analytics reports?
How do I prevent alerts during server maintenance periods?
Should I set up to be alerted only if timeouts are detected by more than one monitor?
What should I do if my "uptime" is less than 100%?

Monitoring Design and Specification
Why monitor from the user's perspective?
What is a Scenario?
How often should I monitor?
Should I monitor from different locations?
What constitutes a server problem?
What is a timeout?
How am I notified when a problem occurs?
How do I schedule and escalate alerts?
When is it important to monitor transactions?
Which transactions should I monitor?
How many Scenarios will I need?
Monitoring Features
What are the basic features?
What are Snapshots of Error-Producing pages?
What are Full URLs of Error-Producing pages?
What are Hyper Visits?
What is a Ping Report?
What is a TraceRT Report?
What are Custom Performance reports?

Answers

Can you recommend a minimal "adequate speed" for our website?
There is no such thing as a "minimal adequate speed". What's adequate for one website may be completely unacceptable for another. You may want to look at some of our benchmarks, for example the eService Index, where you can see the wide range of speeds for different websites. Obviously, what works for American Express (4 sec) or Weather.com (7 sec) doesn't work for Google (0.3 sec). It's up to you to decide what is an adequate speed for your own website. Our job is to measure it and let you compare it to the speed of your competitors, partners, or other site that you wish to compare with.

Why are your speed data different from those of other monitoring services?
Viewing a web page in a web browser is not a "one step" process, but rather a set of back-and-forth "hand shake" interactions between browser and server, which involves several steps (called "events"). Some web pages require only 3-4 steps, while some require 10-20 steps. Each of these steps can take anywhere from 0.01 sec to more than 10-20 sec. The main reason why different monitoring services report different speed for the same URL, is that they measure different set of steps.

At Vertain we measure the entire sequence of steps required to view a web page by a "human" user via a web browser. We believe that this is the most useful result to report, because (1) it reflects most accurately the actual experience of the users, and (2) it gives our clients the most useful information on how to improve the speed of their websites. If you wish to see these steps in detail in your alerts, click Alert Management for the desired scenario, than click Customize, select "Yes" for Include visit log and click Save.

Why does a second speed test usually show faster result?
In order to duplicate the experience of "real" users, our speed test is performed by Microsoft's Internet Explorer which (like other web browsers) "remembers" the content of recently-accessed websites in order to speed up access. When you request the first speed test, the entire content of the requested page is loaded. If you request a second speed test shortly after the first, part of the content may not be loaded again (as it is already available to our browser), and the speed will appear faster. Since we use a minimal size for the "temporary Internet files" and we visit a large number of websites, your content will be remembered for only a few minutes. If you do a second speed test more than 10 minutes later, the entire page content will be loaded again.

Why do several speed tests on the same site sometimes show a widely different results?
The most common cause of wide speed variations is overloaded server or overloaded server environment. For example, if someone else is downloading a large music or image file from your server while you do one speed test, and nobody is accessing your server when you do a second speed test, the first is likely to show a slower speed.

What is "Speed Variation Index"?
Repetitive speed measurementsof a single web page, by Vertain's (or others') monitor are never exactly the same. Indeed, the observed variations are not surprising, given that downloading a web page is a complex process, involving numerous separate interactions among many network devices, some og which may be busy handling other interactions simultanously. While the "bottom line" for most webmasters concerned with the speed of their website is the average speed, large speed variations are also a concern. Large variations mean that a significant number of your online customers may be experiencing unacceptable slow speed which delay viewing the page or, worse, processing their online transactions. The "Speed Variation Index" is a statistical parameter that alerts you when the magnitude of these variations becomes an issue. This parameter is defined as the Standard Variation divided by the Average Speed. Thus, it is a measure of the scatter relative to the average speed. This parameter is a dimensionless number, which ranges from about 0.2 (very low scatter) to about 2 (large scatter). This parameter is also known as Coefficient Of Variation.


Why the "Speed Variation Index" may be important to you?
The speed chart of many websites looks like this:

Here the speed of about 75% of the visits is about 2.5 sec (the "base speed"). The speed of the rest of the visits, between 3 and 12 sec, bring the average speed down to 4 sec. This kind of a speed chart shows that the page CAN be delivered to our monitor in 2.5 sec, but some of the times it takes much longer for the page to be delivered. If 10% of your customers or users must wait 120 sec (2 minutes) to see your page (the 12 sec visits viewed through a typical residential DSL line), this might be a serious problem for your web-based business. When trying to address this problem, the maginitude of your website's Speed Variation Index may help to determine whether these slow visits are caused by your server (for example by overloading), or by slow Internet traffic (in which case you may want to bring the server closer to you customers). If they are caused by random fluctuations in packet traffic, you probably can't do anything about them.

What are Slowdowns?
The Speed Chart below shows the speed of a specific (but rather typical) website during a 24 hour period in July, 2011. The number of Slowdowns is another statistical parameter that can give you insight into website speed issues. "Slowdowns" are visits the speed of which is slower than one Standard Deviation below the average speed. The "Slowdowns" parameter is the percentage of such slow visits. Obviously, if this percentage is significant, you know that a significant portion of your users are experiencing slow response. As an example, consider the speed chart below:

Here the average speed is 6 sec, but this is not the "whole story". We see clearly a "base speed" of 4 sec, which means that the server is capable of serving the page in 4 sec. But many of the visit were slower, some as slow as 8 or even 12 sec. In fact, 14 of the 80 visits shown here are slower than 8.4 sec (the average speed of 6.0 sec plus the standard deviation of 2.4 sec), or 18% Slowdowns. In comparison, in the speed chart below there only 4% slowdowns:



Do you normalize the speed test to the size of the tested page?
The size of the page content (including images and other files) are one of the main factors effecting the speed of a website. The relationship is not exactly linear. For example, twenty files with total size of 100,000 bytes will load slower than one 100,000 bytes file. However, our results are not "normalized" by the size of the content. We measure the time required to load the content, regardless of size. So if you test two pages hosted on your server, one sized 10 bytes and the other 100,000 bytes, the first will show faster speed. You can try this yourself.

How can I measure the speed of my Internet connection?
Vertain is the leader in website and server speed measurement, but we do not provide a tool to measure the speed of your Internet connection. There are many other companies that provide this service. To locate such services, please go to your favorite Internet search engine, and search on a phrase such as "measure internet connection speed".

Could timeouts result from problems with your monitor, rather than problem with my server?
Timeouts may be caused by a problem on any of the nodes along the connection route between our monitor and your server, as well as the monitor itself or your server. Usually a TraceRoute test positively identifies the node that causes the timeout. However, this will not work if some node along the route is set up to block TraceRoute packets.

How do I prevent your monitoring visits from interfering with my Google Analytics reports?
Since our monitor visits your site via standard Web browser, your server sees our visits just like any other visits by "human" users. If the URL that we monitor is within your Google Analytics "funnel", our visits will count as "entrances", and will "distort" your Google Analytics reports. Here is a simple method to prevent our visits from influencing your Google Analytics reports:

1. Log into your Vertain account, go to the "Visiting Options" page for the desired scenario, and add a special query parameter to the "URL to monitor" field. For example, if we are monitoring the URL http://www.myDomain.com, change it to: http://www.myDomain.com?vertain

2. Make sure that your server ignores this parameter, in other words, it should handle the new URL exactly as it handles http://www.myDomain.com.

3. Tell Google Analytics to ignore visits containing this parameter, by adding a custom "exclude filter" with the regular expression ^vertain. For step-by-step instructiona on how to do that, please see How do I create a custom filter?.

How do I prevent alerts during server maintenance periods?
Set up your server so that during maintenance periods it return a page with a message such as "We are currently under maintenance". This is a good practice, better than returning an error such as "Page not found". Let us know what the message is and we'll tell our monitor to consider this a valid page and not deliver an alert (or count this as an error).

Another option (not available under the "Economy" account) is to set up an alert delivery schedule that excludes maintenance periods. Log into your Vertain account, go to My Scenarios --> Alert Management --> Schedule, select "Deliver alerts only when on duty", and enter the daily "shifts" to exclude your maintenance period. This will prevent alert delivery, but our visit log will show error for these visits. This method is useful for managers who want to know the total "down time", including maintenance time, but do not want to be alerted during maintenance.

Should I set up to be alerted only if timeouts are detected by more than one monitor?
If your site is being monitored from more than one location, you can set up to be alerted ONLY when a problem is reported by more than one monitor (to prevent "false alerts"). Before you do this, please be aware of the possibility that a specific geographical region may experience persistent timeouts or availability issues, caused by a problem in a specific node (such situations are not uncommon). So if you supress "single monitor" alerts, make sure you check your incidents online regularly, so you are informed of such a situation.

What should I do if my "uptime" is less than 100%?
Every server and/or its hosting environment must be interrupted from time to time for upgrade and maintenance. This may result in an uptime of less than 100% on our Uptime Report. In fact, a consistent uptime level of 100% over a long period may suggest to some people that the server is not upgraded or maintained regularly (which may result in security risk or less than top performance), or that the uptime data is not "honest". So an uptime level of a fraction of 1% below 100% should not be a problem. However, if your server shows an uptime level below 99% for more than 1-2 weeks in a row, you should invest some resources in diagnosing and fixing the problem.

Why monitor from the user's perspective?
Even if each component and infrastructure level that make up a complex transactional Web site operates correctly on its own, it is possible that the performance of the overall, integrated system will be less than optimal. The most effective way to measure the overall performance of your website is to monitor the ability of end users to perform the specific activities that the site supports. If the intended user of the system is able to perform such activities, the site has fulfilled its purpose. On the other hand, if a significant portion of end users (or their attempts) fail, the system's performance is inadequate. Vertain monitors your site by exactly duplicating the activities of end users through a standard Web browser that measures how well the server responds. As a result, the data we collect accurately reflect the quality of the experience of the users.

What is a Scenario?
A scenario is a set of specifications that determines how a website is monitored. At a minimum, a Scenario requires an inital URL to visit. Optional specs may include a list of items to verify on each page, a list of data items to be entered, and a link or button to be clicked on a page in order to proceed to the next page (for multi-page scenarios). Some scenarios specify the visits very strictly, so that all visits are exactly the same. Others are more "loosely" defined, resulting in different "path" through the site on each visit. The simplest scenario is one where only the Start URL is specified, and only one page is visited. To validate the content of the page, the scenario may specify the title, page signature, specific links, buttons, data entry fields, pictures, etc. If the page is a simple data entry form, the scenario may specify the data to be entered in the different fields, clicking the "Submit" button, and checking the web server response to verify that the data has been submitted correctly. A more complex scenario, consisting of 5 to 10 pages, may verify that the web site allows customers to select retail items and successfully check them out and have them shipped to the customer's address.

Unlike other website monitoring tools, which are script-driven, Vertain's scenarios are data-driven. Scenario specifications are stored in a database, and can be entered and modified through heirarchical data entry pages, including a scenario level, page level and page component level. At each level one can specify validation rules, data entry logic and "click" logic using condition and value comparisons, branching, random selections, and other logic elements designed to allow us to mimic a real user or customer. Scenarios are easy to modify when the site changes, and do not require script programmers to maintain.

How often should I monitor?
Should you monitor once a day, every 5 minutes, or anywhere in between? If your primary monitoring goal is problem notification, ask yourself how long you can afford to wait before being notified that your users or customers cannot access your site or cannot conduct business on the site. You may want to weigh the cost of more frequent monitoring with the business loss that may result from broken site functionality. If your primary monitoring goal is measuring uptime or performance, you'll want to consider the desired accuracy, or the details of the service level agreement that you are trying to enforce. For example, to see weekly uptime expressed in percent (as shown on our uptime report) with accuracy of 0.5%, you'll want to monitor at least 200 time a week, i.e., every 50 minutes.

In either case, changing the monitoring frequency is very easy, and you can adjust it as often as you need to fit your specific needs. For example, after launching a new site or deploying a new server hardware, or during a busy season, you can increase the monitoring frequency. When the site has stabilized you can reduce the frequency.

Should I monitor from different locations?
If your users or customers are served by a single server hosted in one location, monitoring from a single location may be sufficient. If you are targeting users in a specific geographical area, you may want to select a monitor located in that area. However, please note that speed and accessibility are governed as much (and even more) by backbone infrastructure than by geography, so geographic location may mean less that you may think.

If your users or customers are served by more than one server (whether your own distrubuted servers or via content distribution verdors such as Akamai), monitoring from different locations will give you much better information on the speed and accessibility by users in different locations. For example, if the bulk of your users are in the East and West Coast of the US, you may want to select our San Francisco and New York monitors.

What constitutes a server problem?
We record as an incident any visit in which the transaction specified by a scenario could not be completed successfully. We classify incidents into three main categories:

1. Availability errors occur when the server cannot be located or accessed. This type of error is primarily caused by a network connectivity problem or a technical problem with the Web server or the web hosting service.

2. Timeouts occur when the server has been successfully located and accessed, but the required content is not completely deleivered within a specified time period. Most timeouts are caused by a problem in the server or the server's infrastructure, or by a software problem.

3. Content errors occur when the server returns the wrong content (such as error messages or a wrong page), or when some of the content is missing. Content errors are caused mainly by software problems or server infrastructure problems.

What is a timeout?
The process of displaying a web page on your web browser is not a one step affair that can either succeed or fail. Rather, it involves a sequence of steps, all of which must succeed in order for the page to be displayed correctly. The sequence begins with "finding" an appropriate IP address via domain name lookup, then establishing an initial "contact" with a specific web server, optionally via "load balancer" or some other "redirection" mechanism within the hosting network infrastructure. Once the specific server has been located, your web browser begins a dialog with the server, in which the browser "asks" a sequence of "questions" and waits for the server to provide appropriate "answers". Finally, optional "client-side" processes (such as JavaScript functions) may have to be executed in the web browser. The page is considered "complete" when all the above steps have been successfully completed.

If the server cannot be located, or if the server is "down" (i.e., not operational), we report an "availability error". But what if the server is successfully located, but then doesn't respond within a reasonable time period to one of the "questions" in the dialog? In this case we report a timeout. We are all familiar with long waits, as we sit watching our broswer's icon spinning, until our patience runs out. The length of time we are willing to wait for a page to complete obviously depends on the importance of the page and on how much time we have, but everyone has a limit. When this limit is reached, we say that the page has timed out. Our benchmarks indicate that timeouts are as common as availability problems. Given that they are more frustrating to users (since they take up more of the users' time) we consider them in general more serious problem than downtime.

Timeouts are rarely caused by a slow Internet connection. Most timeouts are caused by specific server or software issues that can (and should) be diagnosed and fixed.

How am I notified when a problem occurs?
By default, we'll alert you by email on each incident. You can specify several email addresses, some of which may be redirected to text messaging and paging services. You may also select to be notified by a regular phone call, to ensure immediate notification when not monitoring incoming email. We can also deliver alerts to your customer service ticket system. (If we don't support it, let us know and we'll integrate with it).

How do I schedule and escalate alerts?
By default, the alert recipient for a new account is the account owner (yoursef), and an alert will be emailed to you on each incident. When this default alert handling no longer serves your needs, you can easily customize the alert delivery rules to fit your particular needs. We support advanced alert scheduling and escalation rules, including our exclusive "Hyper Visits" feature.

When is it important to monitor transactions?
If your website supports data entry forms or business transactions (such as purchasing, reservations or banking activities), it is not enough to monitor the server uptime or the availability of the home page. You must also ensure that customers are able to successfully submit forms or complete supported transactions. In facts, timeouts or content errors on one of the transaction pages are usually more damaging to your business than the inability to access the home page, because the customer has already invested time in entering data or selecting a product or an itinarary, and her time and efforts are now wasted. Such errors may hurt business not only because of the abandonment of the attempted transaction, but because of poor expertience that will drive customers to a competing site.

Our eTailer and eService benchmarks show that the levels of timeouts and content errors in the online retail and service industries are consistently higher than availability error levels. If you monitor only the server uptime or the availability of your home page, you wouldn't know about these important problems.

If the above statements are too "technical" for you, or if you are not sure how important it is for you to monitor transactions, please let us know and we'll be happy to explain this issue further.

Which transactions should I monitor?
It is rarely practical to monitor literally all the possible activities supported by a complex Web site. Therefore, we recommend that you initially monitor what you think are the most critical activities, based on your own understanding of the business. To find out which transactions have the highest priority, you may want to list the different functionalities offered on your site, then sort them by their importance in achieving the specific business goals of the site. Maximizing on-line revenue may not necessarily be the highest priority, even for an on-line retail site. The primary value of the site to your organization may be, for example, helping customers find store locations, browse item availability or get after-sale customer service.

The goal of the monitoring may also be technical, for example, to evaluate the performance of a legacy/Web integration link. In such cases you may want to get our help in setting up a monitoring scenario that focuses on the relevant technical activity.

How many Scenarios will I need?
This depends primarily on the range of "similar" activities that you need to monitor. For example, you may need only one scenario to monitor even a complex retail site that offers many types of items for sale, since our monitor can select items randomly, thus select a different item on each visit, and test your entire product inventory over time. On the other hand, since the cost of monitoring depends on the number of pages visited, it may be advantageous to set up several scenarios, each one customized to test issues specific to individual product categories, or related to specific technical functionality. For example, monitoring a retail site may be more cost-effective with two scenarios: (1) a one-page, simple scenario that visits fairly frequently (e.g., every 15 minutes) and verifies that the home page is available, and (2) a multi-page scenario that verifies the checkout functionality.

Our staff will be happy to help you sort out the different options and determine the best set of scenarios for your specific needs and budget.

What are the basic features?
The basic features of our Service, included for all account types, consist of visiting a URL at specified interval, performing the specified test, recording the results (e.g., speed), and emailing an alert if an error was detected. Users also have online access to performance reports, account preferences and billing information.

What are Snapshots of Error-Producing pages?
When we encounter a content error, we can optionally store a "snapshot" of the error-producing page, and include a link to this snapshot in the alert that you receive. You can then view the snapshot by clicking this link. The snapshot may contain information that will help you diagnose and fix the problem.

What are Full URLs of Error-Producing pages?
When we detect a problem with your site, we can optionally store the URL that "requested" the error-producing page, and include it in the alert that you receive. You can then re-submit the request and may be able to duplicate the problem, and help diagnose and fix the problem. Even if this is not possible, the full "request" may contain usefull information.

What are Hyper Visits?
Hyper Visits are additional visits that are automatically scheduled shortly after an error is detected, in order to determine if an error is a short-lived "glitch" or a repetitive, reproducible problem. Each Hyper Visits is scheduled about a minute after an error is detected, as long as the error persists. To illustrate, suppose you specify two (2) Hyper Visits. If an error is detected during a regularly scheduled visit, an additional visit will be scheduled about a minute after the regular visit. If this additional visit is completed successfuly (without an error), no more Hyper Visits will be scheduled. If, on the other hand, an error is detected in the first Hyper Visit, a second Hyper Visit will be scheduled, about a minute later. If you wish to be alerted only if an error persists longer than some critical period, specify the desired number of Hyper Visits, then set up an appropriate Grace Period (under Alert Management / Scheduling). For example, to be alerted only if a website error persists for more than about 5 minutes, select 4 Hyper Visits and set up a Grace Period of 5 minutes.

What is a Ping Report?
When an "availability" problem is detected, a Ping report may help determine whether the cause is in the server itself, in the domain name server, or in the Internet connection of the server. A Ping report, executed from our monitoring engine to the IP address of your server, indicates whether a connection exists between the two. If the connection exists, it is likely that the availability problem is caused by a server issue or by a DNS (Domain name Server) problem. Otherwise, it is likely that your server's Internet connection is interrupted. Please note that Ping will indicate "no connection" if ICMP packets are blocked. For more information about Ping, click here.

To activate the Ping Report feature, login to your account, go to the Visiting Options page, enter the IP address of your server in the field IP# to Ping, and click Save. When this feature is activated, we will automatically execute a Ping test to the specified IP address whenever we detect an availability error or a timeout. The outcome of this test will be emailed to you in addition to the alert.

What is a TraceRT Report?
When an "availability" or "timeout" problem is detected, a TraceRT report may help diagnose the nature of the connectivity issue. A TraceRT report is similar to Ping's, but contains more detailed information. It shows the path between our monitoring engine and your server, and the access speed to the intermediate nodes. This information may be particularly useful in understanding the cause of recurring "timeout" errors, and in diagnosing connectivity problems revealed by negative Ping results. It will also show DNS issues. Please note that TraceRT will not work when ICMP packets are blocked. For more information about TraceRT, click here.

To activate the TraceRT Report feature, login to your account, go to the Visiting Options page, select "Yes" in the field Do TraceRT on error, and click Save. When this feature is activated, we will automatically execute a TraceRT test to the start URL of the scenario whenever we detect an availability error or a timeout. The outcome of this test will be emailed to you in addition to the alert.

What are Custom Performance reports?
In addition to the standard reports (Visit Log, Incident Log and Performance History), you can create Custom Performance reports that let you see, at a glance, the "bottom line" of your site's performance, as you define it. This "bottom line" is a number from 0 to 100 called Performance Index, that summarizes the error level, page and transaction speed, and repair time according to your own priorities. To create such a report, you define your goals for the items above, and specify the importance each goal on a 1 to 5 scale. For example:

- Max total error: 0.2% (extremely important)
- Page speed: 0.5 sec (important)
- Transaction speed: 3.0 sec (less important)
- Longest repair time: 30 min (very important)

Once the goals are defined, you can view the report for the desired period (a day, week, month, or custom period). The report displays the Performance Index, showing if your goals have been reached. In case the goals have not been reached, it shows the gaps between the goals and the actual performance.



Free Speed Test
Home
Clients
Services
Benchmarks
Technology
Company
Contact Us

©2001-2015 Vertain Software. All rights reserved. Site Map Terms of Use Privacy Policy