Measurement Factory is organizing an industry-wide competition of Web Caching proxies. This competition is the fourth event in the TMF cache-off series. Below you will find preliminary details about the event. We encourage all vendors with relevant products to contact us as soon as possible and participate in the organizational meeting. No formal commitments are required at this time.

Table of Contents

1. Nature of the event
2. Scope of the tests
3. Testing methodology
4. Publication of the results
5. Organizational meeting
6. Location
7. Important dates
8. Contact information

1. Nature of the event

The cache-off is an all-under-one-roof competition of Web caching proxies. All participating products will be tested under identical rules, using the same software and workload, and at about the same time. Participants are responsible for configuration and tuning of their products. Measurement Factory provides the test harness, executes the tests, and analyses the results. The results, prices, and all configuration details will be made public shortly after the tests.

The rules of the competition and workload details are developed with the cooperation of the user community and test participants.

Participation fee is required to compete.

2. Scope of the tests

All commercial vendors, non-profit organizations, and any other entities with a Web caching product (a.k.a. forward caching proxy) may participate. One vendor may enter several products into the competition. No artificial participation limits are planned at this time.

Cache-off participants may choose to tighten or change the scope of the tests by, for example, eliminating beta-quality products. These issues will be discussed at the organizational meeting.

3. Testing methodology

Web Polygraph benchmark and PolyMix-4 workload will be used for the tests. Polygraph test suite includes simulated HTTP clients and servers. A participant will specify peak request rate they want to support (as a workload parameter). A typical Polygraph setup will include thousands of simulated clients and a hundreds of simulated servers, all with unique IP addresses. Using those simulated agents, Polygraph will collect performance measurements and verify that a product can sustain the advertised request rate.

A single test is expected to last for 15 hours or more, depending on cache fill time. Polygraph client and servers will use real HTTP/TCP/IP transactions with appropriate features (e.g., HTTP persistent connections), headers and content. Simulated Web objects will be of realistic sizes (about 11KByte mean and 4KByte median). Exact PolyMix-4 workload specifications and execution rules will be available in advance and are expected to be based on PolyMix-3 workload with new features.

Polygraph PCs will have 100baseTX interfaces. Participants will be able to use any network gear as traffic aggregation or redirection devices if necessary. Configuring the network using PolyMix-4 rules is the responsibility of a participant.

Reported measurements will include throughput (HTTP requests per second), bandwidth (bytes per second), mean response time of an HTTP transaction, hit ratios, errors, and their derivatives. The official report will also include pricing, availability, and configuration information. Official reports for previous events are available elsewhere.

4. Publication of the results

An official public report will be available at Measurement Factory Web site. At least one major trade magazine is expected to publish the results of the tests. Participants will be able to create their derivative works from the official report.

5. Organizational meeting

Measurement Factory would like to host a face-to-face meeting with potential cache-off participants. During the meeting, test scope and methodology, participation fees, preparation tips, and logistics details will be discussed, and the final shape of the event will be determined. Attendance is not mandatory but strongly encouraged. Nobody can represent participant's interests better than the participant itself.

Registration is required to attend the meeting. Please send an e-mail to <team at measurement-factory.com>. The attendance is limited to one person per organization.

The meeting will be held in Denver, Colorado unless most of the participants prefer a different location. See below for registration deadline and times.

6. Location

Measurement Factory plans to host the cache-off near Denver, Colorado. Proposals for alternative locations will be considered. The requirements are lots of power, space, and appropriate cooling for 200+ PCs.

7. Important dates

Here is the proposed schedule. Deadlines may be changed at the organizational meeting.

^*The date has been changed! The old date, May 21st, conflicted with NANOG event, and several vendors wanted to attend both switch-off and cache-off meetings in one trip. We hope that no further changes are required.

8. Contact information

Questions? E-mail <info at measurement-factory.com>.