Fourth Cache-off: The Rules

The Measurement Factory

These rules are not final and are subject to discussion at the organizational meeting. If you have any suggestions on improving the rules, please contact us.

Table of Contents

1. Terminology
2. Requirements for participation
    2.1 Product prerequisites
    2.2 Preparation control
    2.3 Disclosures
3. Access to the cache-off facility
4. Benches
    4.1 TMF zone
    4.2 Participant zone
    4.3 Inter-Zone communication
    4.4 Inter-Bench network
5. Testing
    5.1 Official tests
    5.2 Tests execution
    5.3 Results selection
    5.4 Test failures
    5.5 Equipment failures
6. Cache-off report
    6.1 Comments on the results
    6.2 Derivative works, publications, and such
7. Bailout and failures
8. Conflict resolution

1. Terminology

Vendor
An organization bringing a product to be tested at the cache-off. To simplify the terminology all commercial, non-profit, virtual, etc. organizations are labeled as ``vendors.''
Product under test
A product/solution/box being tested as a single proxy cache. A product includes software, hardware, network gear, etc. required for the cache operation. A product may include several units.
Entry
A (Vendor, Product) tuple.
Participant
A Vendor associated with a particular Entry.
Test Bench
A group of machines and network gear dedicated to benchmarking an entry, including the Product.
TMF
The Measurement Factory; TMF is the company responsible for Web Polygraph development and cache-off organization.

2. Requirements for participation

2.1 Product prerequisites

A Product (as defined in the ``Terminology'' section) must satisfy all of the following criteria to be considered for the cache-off.

2.2 Preparation control

A hard copy of an auto-generated Polygraph report should be presented to TMF before the start of the first official test for the corresponding entry. TMF reserves the right to make public the fact that the report was (or was not) submitted. The contents of the report will not be made public. The report must be generated with the same Polygraph version and workload as used at the cache-off and correspond to the same (to the extent possible) Product as tested at the cache-off. The report must be based on a successful performance test.

If such a report is not presented, the entry will receive the lowest handling priority during the cache-off, including troubleshooting and technical support or advice.

The first official performance test will use request rate measured in the submitted report or a lower request rate.

2.3 Disclosures

Each Participant must disclose the following:

This documentation must be made available prior to the start of the official test sequence. TMF will not start official tests unless the Participant provides all necessary details.

All prices must be US prices specified in US dollars. All prices must be based on unlimited usage licenses and must carry no discriminatory discounts or usage restrictions.

There will be no private rooms for test benches. Participants will be able to see each other's equipment.

During the cache-off, TMF will take photographs of the equipment as tested. Photographs from other sources will not be accepted. The photographs taken at the cache-off may be included in the official report to document entry configuration and appearance.

3. Access to the cache-off facility

Working hours are Monday through Friday, 9am till 7pm with an hour lunch break.

4. Benches

A Bench is logically divided into a TMF zone and a Participant zone. Zones are connected using 100BaseT Ethernet cables.

4.1 TMF zone

TMF provides a number of machines to operate as clients and servers. Client machines have similar hardware and software configuration (identical to the extent possible). Server machines have similar hardware and software configuration (identical to the extent possible).

To manage the tests, TMF provides monitoring stations, one per Bench. The configuration of monitoring stations does not affect the tests.

The number of client and server machines in a cluster is determined based on the desired workload (peak request rate). Some Benches may require more clients and servers than other Benches.

TMF is responsible for setup, operation, and maintenance of the client and server machines. The exact hardware and software specs will be included in TMF's cache-off report.

4.2 Participant zone

The Participant zone includes the cache under test, cables, and any networking gear (switch, router, hub) used to connect the client and server machines to the cache.

All equipment in the Participant zone must meet the ``Product prerequisites'' described previously.

Optional L2 aggregation devices may be used to aggregate traffic from Polygraph clients or servers. The list price of aggregation devices must be disclosed, but will not affect total Entry price. Other than these ``zero cost'' considerations, aggregation devices are treated no different from any other device in the Participant zone.

All aggregation devices must be disclosed to TMF as early as possible and before the start of the cache-off. TMF reserves the right to grant or revoke permission to use an aggregation device at ``zero cost''. Since TMF decision may be somewhat arbitrary, it is in the Participant's interests to double check with TMF early in the process, to avoid unexpected Entry price increase.

With the exception of optional L2 aggregation devices, all equipment in the Participant zone, including networking hardware, is included in the list price of the Participant's configuration. Substitutions are not allowed. For example, a participant brings a 24-port switch and uses 8 ports, the price cannot be scaled down to an 8 port switch of the same manufacturer. Removing ``blades'' or ``cards'' from network gear is allowed if the gear can be bought without those components.

The Participant is responsible for setup and maintenance of all the equipment in the Participant zone.

The Participant should specify any special accommodations (extra cooling, extra power outlets, power other than 110V AC, etc.) as early as possible.

4.3 Inter-Zone communication

Participant zone must allocate at least one 10/100BaseT Ethernet port for the monitoring station.

Participant zone setup must provide IP level connectivity between the monitoring station and all client and server machines. For example, the monitoring station must be able to ``ping'' or ``ssh'' to all clients and servers.

Participant may request that their cables are used for networking clients and servers. Otherwise, TMF will provide cables.

In the event of performance, compatibility, or other problems between TMF and Participant zones, TMF and the Participant must work together to isolate and eliminate the problem.

4.4 Inter-Bench network

In order to facilitate centralized management of all client and server machines, TMF may utilize a routed network that connects all Benches. This network is to be physically dismantled during performance tests.

All benches will be physically separated, and no connections between benches will be allowed (except for temporary network connections described above).

5. Testing

A sequence of tests is run independently and concurrently on every Bench. The number of benches is equal to the number of entries.

The rest of the section describes the test execution procedure for one Participant.

5.1 Official tests

Official tests are:

  1. MSL test
  2. No-proxy test
  3. PolyMix-4 ``performance'' test
  4. Downtime test
  5. N.B. other tests may be added; this list is not final

All other tests are unofficial.

5.2 Tests execution

All tests must start and finish within the time allocated for the cache-off. A total of 55 hours of ``busy'' time is guaranteed for each participant. ``Busy'' time includes the time when official tests are running (or can be running as far as TMF is concerned). ``Busy'' time also includes 15 minutes to prepare each test. If the participant shows up late, or is not ready for some reason, time is subtracted from their 55 hours until they are ready for testing. Practice shows that many vendors will have more than 55 hours of test time.

TMF will treat requests to repeat a test with a lower priority than requests to execute the first test. Rerunning unexpectedly failed tests will get higher priority that rerunning other tests. TMF may refuse to repeat the test when 55 hours run out and/or when further trials are unlikely to change the outcome of the tests.

Test execution rules also include the following provisions.

5.3 Results selection

Reasonable efforts will be made by TMF to share run-time performance with Participants.

A participant may repeat any test and choose the best results among all non-failed tests executed.

TMF archives all the logs from the official tests. On Participant request, TMF releases these results to the Participant after the cache-off is over (or earlier, time permitting).

5.4 Test failures

A test is considered failed if any of Polygraph processes aborted for any reason. Cache-off workloads are designed to accumulate pending transactions forever which can lead to Polygraph running out of memory if the proxy under test cannot sustain offered request rate.

A test is considered failed if the number of transaction errors exceeds 1% during any measurement phase. Transaction error is an error that causes HTTP transaction to be aborted prematurely or an error that indicates that the proxy responded incorrectly to a valid request.

A test is considered failed if during any measurement phase, measured hit ratio falls below 50% of the offered hit ratio and/or if during any measurement phase, measured mean response time exceeds no-proxy response time by at least 100%.

Other failure conditions may exist and will be assessed run-time by the testing team.

5.5 Equipment failures

In the event of equipment failure, the failed parts can be replaced, but the interrupted test (if any) is invalidated. The fact of the failure is not reported unless the failure leaves a participant without valid results.

6. Cache-off report

TMF prepares an official cache-off report. TMF will make reasonable efforts to discuss the content of the report with Vendors before making the report public. TMF makes all cache-off results publicly available on the Web. TMF must keep the cache-off archive available for at least one year.

Cache-off participants agree not to make public (or publicly discuss) the results of the cache-off prior to public release of the official report by TMF.

6.1 Comments on the results

A Participant may request inclusion of a ``Participant Comments'' section into the report. The comments must not contain any references to external documents. The comments should only discuss matters relevant to the cache-off. Vendors are strongly discouraged from include speculations and accusations in the comments.

The comments are limited to 300 words per entry and 750 words per vendor, whichever is less. All comments must be submitted in plain ASCII text format.

Once included in the report, the comments cannot be modified.

TMF reserves the right to not include the Participant Comments.

TMF may add their own comments, up to three months after the publication of the report.

The Comments become a part of the official result distribution. These Rules do not require any part of the results to be included in the derived works.

6.2 Derivative works, publications, and such

Anyone can publish or otherwise make public any derivate work based on the cache-off results (i.e., logs and the official report), provided that the ``Terms of use'' clause is satisfied.

Anyone can get a free copy of a Participant result from the Official Site, provided they agree to the ``Terms of use''.

Terms of use
  1. The Measurement Factory holds the copyright to the results and the official report.
  2. Any derived work, distribution, publication, presentation, etc. of the results must include the following explicit reference in its entirety:
    Original raw data and independent analysis of the fourth cache-off results are at http://www.measurement-factory.com/results/
  3. All modifications of the results and the official report are prohibited. Only verbatim copies may be distributed.

7. Bailout and failures

After the start of the first official test, a participant must not bailout or prevent their results from being published.

A Participant may request that a test is considered ``failed''. If all tests for the entry fail, the entry is considered ``failed''. TMF may declare an entry ``failed'' if the entry cannot produce reasonable results despite TMF and Participants efforts such as decreasing request rate by 50% of the registered peak load.

Failed entries (if any) will be counted in the report preamble and listed in the executive summary table with all measurements replaced with ``n/a''.

A special section at the end of the report will contain details necessary for a reader to understand why no results are available for each failed entry (and, possibly, links to re-test results; see below). Vendors of failed entries may submit ``vendor comments'' sections.

Failed entries will be given an option to be re-tested at TMF lab within two weeks after the end of the cache-off. The results of those tests will be linked from the official cache-off report, but will not be included into the cache-off report. The vendor of failed entries is expected to pay for any extra equipment required for the re-tests.

8. Conflict resolution

During the cache-off, situations requiring vendor input may arise. Presence of all vendors during the entire competition is strongly encouraged. Absent vendors are assumed to side with TMF in case of run-time disputes or conflicts. Participants agree to honor any addendum developed during the cache-off, regardless of participants presence at the decision making meeting.

A reasonable effort on the part of a TMF will be made to engage all present vendors and to find a compromise solution.

In the event of any form of conflict, including interpretation of these Rules, TMF will be the final arbiter and judge.