Fourth TMF Cache-Off: Preliminary Call for Participation

The Measurement Factory

Measurement Factory is organizing an industry-wide competition of Web Caching proxies. This competition is the fourth event in the TMF cache-off series. Below you will find preliminary details about the event. We encourage all vendors with relevant products to contact us as soon as possible and participate in the organizational meeting. No formal commitments are required at this time.

Table of Contents

1. Nature of the event
2. Scope of the tests
3. Testing methodology
4. Publication of the results
5. Organizational meeting
6. Location
7. Important dates
8. Contact information

1. Nature of the event

The cache-off is an all-under-one-roof competition of Web caching proxies. All participating products will be tested under identical rules, using the same software and workload, and at about the same time. Participants are responsible for configuration and tuning of their products. Measurement Factory provides the test harness, executes the tests, and analyses the results. The results, prices, and all configuration details will be made public shortly after the tests.

The rules of the competition and workload details are developed with the cooperation of the user community and test participants.

Participation fee is required to compete.

2. Scope of the tests

All commercial vendors, non-profit organizations, and any other entities with a Web caching product (a.k.a. forward caching proxy) may participate. One vendor may enter several products into the competition. No artificial participation limits are planned at this time.

Cache-off participants may choose to tighten or change the scope of the tests by, for example, eliminating beta-quality products. These issues will be discussed at the organizational meeting.

3. Testing methodology

Web Polygraph benchmark and PolyMix-4 workload will be used for the tests. Polygraph test suite includes simulated HTTP clients and servers. A participant will specify peak request rate they want to support (as a workload parameter). A typical Polygraph setup will include thousands of simulated clients and a hundreds of simulated servers, all with unique IP addresses. Using those simulated agents, Polygraph will collect performance measurements and verify that a product can sustain the advertised request rate.

A single test is expected to last for 15 hours or more, depending on cache fill time. Polygraph client and servers will use real HTTP/TCP/IP transactions with appropriate features (e.g., HTTP persistent connections), headers and content. Simulated Web objects will be of realistic sizes (about 11KByte mean and 4KByte median). Exact PolyMix-4 workload specifications and execution rules will be available in advance and are expected to be based on PolyMix-3 workload with new features.

Polygraph PCs will have 100baseTX interfaces. Participants will be able to use any network gear as traffic aggregation or redirection devices if necessary. Configuring the network using PolyMix-4 rules is the responsibility of a participant.

Reported measurements will include throughput (HTTP requests per second), bandwidth (bytes per second), mean response time of an HTTP transaction, hit ratios, errors, and their derivatives. The official report will also include pricing, availability, and configuration information. Official reports for previous events are available elsewhere.

4. Publication of the results

An official public report will be available at Measurement Factory Web site. At least one major trade magazine is expected to publish the results of the tests. Participants will be able to create their derivative works from the official report.

5. Organizational meeting

Measurement Factory would like to host a face-to-face meeting with potential cache-off participants. During the meeting, test scope and methodology, participation fees, preparation tips, and logistics details will be discussed, and the final shape of the event will be determined. Attendance is not mandatory but strongly encouraged. Nobody can represent participant's interests better than the participant itself.

Registration is required to attend the meeting. Please send an e-mail to <team at measurement-factory.com>. The attendance is limited to one person per organization.

The meeting will be held in Denver, Colorado unless most of the participants prefer a different location. See below for registration deadline and times.

6. Location

Measurement Factory plans to host the cache-off near Denver, Colorado. Proposals for alternative locations will be considered. The requirements are lots of power, space, and appropriate cooling for 200+ PCs.

7. Important dates

Here is the proposed schedule. Deadlines may be changed at the organizational meeting.

Date/Time
(MDT)
Weeks Before
the Cache-off
Milestone(s)
May 16*
09:30
17:30
21 Organizational meeting in Denver.
June 18
15:00
16 Feature freeze and workload beta. By this date, Polyteam knows and has announced all of the features that will be present in the PolyMix-4 workload. Polyteam have begun to implement these features in Polygraph. A draft (beta) specification of the PolyMix-4 workload will be distributed.
July 9
15:00
13 Polygraph Beta Release. By this date, Polygraph is able to successfully complete a PolyMix-4 run with a proxy cache. Any known bugs should be fixed by this time. Polyteam will distribute documentation that is sufficient to run a PolyMix-4 test. This is the participant's cue to take the software and begin running tests in their own labs. During the next four weeks, participants must provide feedback on the code and documentation.
Jul 23
17:00
11 Soft Registration. Participants must provide (a) an auto-generated report from a successful PolyMix-4 run or a detailed description explaining why no report can be generated, and (b) a non-refundable deposit of US$2,000. The purpose of the report is prove to Polyteam and the participant that Polygraph can complete the PolyMix-4 run without bugs and/or errors. If a participant has an issue with the PolyMix-4 workload, the Polygraph code, or the documentation, it must be brought to Polyteam's attention by this date, or the complaints may not be considered. The contents of auto-generated reports are kept absolutely confidential by Polyteam.
Aug 13
17:00
8 Everything frozen. By this date, all problems brought to light during the ``beta test'' phase must be fixed by Polyteam. After this date, there are to be no changes to the source code and workload parameters.
Sep 10
17:00
4 Firm Registration. Participants must pay the full entry fee to Polyteam by this date. Polyteam needs four weeks lead time to rent computer and networking equipment.
Oct 8
08:00
0 First day of the Fourth Cacheoff. When participants come to the cache-off they must provide a recent auto-generated polygraph report. The report must prove they have completed a successful PolyMix-4 run. The first official cache-off test for a participant must be at this same, or lower, peak request rate to ensure that there will be at least one successful run at the cache-off.
Oct 25
11:00
  Cache-off report draft released to the participants.

*The date has been changed! The old date, May 21st, conflicted with NANOG event, and several vendors wanted to attend both switch-off and cache-off meetings in one trip. We hope that no further changes are required.

8. Contact information

Questions? E-mail <info at measurement-factory.com>.