Fourth TMF Cache-Off: Organizational Meeting (CFP)

The Measurement Factory

Note: Meeting minutes are available elsewhere.

This page discusses the agenda and logistics for the fourth cache-off organizational meeting. Attending the meeting is not required to participate in the cache-off. However, if you do not attend, you will miss an opportunity to shape the cache-off rules and workload for your taste. If you cannot attend the meeting, feel free to share your opinions on the Polygraph mailing list or contact us directly.

Table of Contents

1. Logistics
2. Attendees
3. Preliminary Agenda
    3.1 Cache-Off Deadlines and Location
    3.2 Cache-off Rules
    3.3 PolyMix-4 workload
    3.4 Budget

1. Logistics

The Measurement Factory is hosting an organizational meeting for the upcoming cache-off participants and anybody interested in the agenda. If you want to come to the meeting, please e-mail us. You must send us an e-mail to attend. At most two representatives per company, please.

The meeting is scheduled for Wednesday, May 16th in Denver, Colorado. This is an 8 hour event, starting at 9:30am. The meeting will take place in Hilton Garden Inn. The hotel is about 15 driving minutes from the Denver International Airport. There is a free shuttle service from DIA provided by the hotel. For more information on the location and shuttle service, please see hotel's home page. Several other hotels are nearby.

We hope that at least some of the participants will be able to arrive and leave on the same day. For those of you who have/want to spend a night or more in Denver area, we recommend going to Denver (~25 driving minutes from the hotel) or, better, Boulder (~55 driving minutes from the hotel).

On-site lunch will be provided. Please let us know if you have any food preferences. We will try to get an Internet connectivity in the meeting room, but do not count on it.

2. Attendees

We have invited all companies with caching products we knew about, including ARA Networks, CacheFlow, CinTel, Cisco, ClickArray, Compaq, Dell, F5, HP, IBM, iMimic, iPlanet, InfoLibria, Inktomi, Lucent, Microbits, Microsoft, NAIST, Network Appliance, Squid, Stratacache, Volera, and Swell Technology. Many have already confirmed their attendance. Attendee list will be announced soon.

We have received many requests to increase meeting attendance limits to two persons per vendor. If you absolutely must send two people, please keep the following caveats in mind.

3. Preliminary Agenda

Major agenda items are discussed below. If you want to add an item to the agenda, please let us know ASAP.

3.1 Cache-Off Deadlines and Location

Preliminary deadlines need to be discussed and adjusted as needed. Our objective is to give participants and TMF enough time to prepare for the cache-off, execute the tests, and process the results.

The cache-off is likely to be five days long. Will adding more time solve any problems?

We do not have a confirmed location yet. If you can provide a ~25,000 ft2  or larger warehouse or similar size space with lots of power and appropriate cooling, please contact us. Providing cache-off facilities saves you shipping and travel costs while giving prompt spare delivery and ``on-site'' support from the best local troubleshooters. If we receive no attractive offers, the cache-off is likely to be held in San Jose or Boulder-Denver area.

3.2 Cache-off Rules

We intend to adopt the third cache-off rules, with minor modifications, but are open to suggestions. The current version of the rules is available elsewhere.

We also need to address the following issues.

Entry availability requirements

Some vendors are concerned that beta-quality implementations are competing with production caches. This concern is not new, but we would like to re-evaluate entry availability requirements. One option would be to decrease the availability cut-off from two months since the report publication date to two month or less since the start of the competition.

An alternative is to report yet-unavailable products or ``prototypes'' differently. For example, unavailable products can be placed into a separate report. We could also try to mark prototype results while presenting them in the same report (or not).

Interestingly enough, SPEC has been dealing with similar issues/decisions and has not yet found a good solution that works (and SPEC has been around for much longer than Polygraph competitions). SPEC problem space is also somewhat smaller because every result is just a single ``number'' (easy to mark) as opposed to many measurements in Polygraph executive summaries and reports. The lack of good existing solutions might mean that the problem simply does not have any, but we should definitely try to improve the current scheme.

HTTP compliance tests

Several vendors suggested that participating products should also be tested for HTTP compliance/compatibility to weed out half-implemented products. Currently, there is no stable vendor-independent software that can perform such tests. TMF's Co-Advisor project may deliver such a tool before the cache-off, but using Co-Advisor for competition goes against other vendor preferences. We need to decide whether non-performance HTTP testing should be a part of the cache-off and, if yes, then what tool we can use to perform those tests.

Web Accelerator tests

In addition to PolyMix results targeting forward-proxying environments, we are considering offering a WebAxe tests to measure performance of Web accelerators. These tests could be optional (since not all forward proxies can be used as reverse proxies) and the results can be published in a separate report, linked from the cache-off report pages. Polygraph WebAxe workload will have to be polished for these tests. Adding another major workload will increase participation costs, but the increase will be far less than participation in a special competition dedicated to reverse proxies.

3.3 PolyMix-4 workload

The PolyMix-4 will be derived from the PolyMix-3 workload. Meeting participants are expected to be familiar with PolyMix-3 characteristics.

The addition of new features and improvement of the current ones will be based on the amount of time we will have according to the deadlines. However, our focus will be on improving existing simulation models rather than adding completely new features. As Web Polygraph matures, the number of missing essential features decreases and the quality of details becomes more important.

The following modifications are considered for PolyMix-4 and the fourth cache-off.

We need your help to grow and prioritize the list above.

Polygraph is used for tests other than caching cache-offs. This multipurpose status may affect our priorities and development schedule.

3.4 Budget

We intend to follow the second cache-off pricing model ($2-3K per client-server pair; each pair generating about 500 requests per second load), but are open to suggestions.

Should we give discounts to huge benches? How deep should those discounts be? It certainly takes a lot of TMF effort to setup and troubleshoot a large bench compared to a bench with a few client-server pairs. On the other hand, per-PC overheads decrease with the bench size.

Besides covering the expenses, our primary objectives are:

  1. Attract both ``small'' and ``large'' participants.
  2. Sustain Web Polygraph development.