PERFORCE/CLEARCASE benchmark
January 4, 1996
SCENARIO
These comparisons were done on a single unloaded Sparc 10 with 32MB of
memory on a single 1GB disk partition. Some numbers would be different
if the client and server were on separate hosts. Both the ClearCase
VOB and PERFORCE depot were created new for the test.
The test consisted of adding a set of files into the system, checking
out and in a set of files, reading all the files using "cat" twice
(once to get the initial load and the second time to get the warmed-up
speed), branching a set of files, and labelling a set of files.
Speed is computed both for ClearCase and for PERFORCE and both in terms of
files/second and KB/sec, since some operations are sensitive to the
sizes of the files. The final section has the performance ratio of PERFORCE
to ClearCase in terms of both KB/sec and files/sec. The ratio for
KB/sec is lower than file/sec for checkout/checkin, which is probably
due to the fact that the files used in the PERFORCE sample included a lot
more small files. Since the checkout/checkin speed for small files is
dominated by per-file costs, handling many small files drives down the
KB/sec rate.
Ideally, we would have used the exact same number of files for both PERFORCE
and ClearCase, but that would have meant either unusably low times for
PERFORCE or very long waits for ClearCase. Labelling all 750 files took
somewhere around a second with PERFORCE which doesn't give very many significant
digits. On the other hand, we we had to interrupt the initial load
into ClearCase in order to get the tests done in the limited time we had.
Some activities don't match exactly between ClearCase and PERFORCE most
notably "cat(1)-ing" a bunch of files for the first time. The purpose
of this test is to see how fast the content of files can be produced.
This is similar to the performance for compiling or searching a tree of
source files. With the ClearCase MVFS you just invoke cat: on the
first call the MVFS populates its cache and produces the files; on the
second call it produces the files from the cache. With PERFORCE you must
first populate the client tree with "get." Then both first and
subsequent calls to cat are just against the local file tree. Thus
for PERFORCE the time for "Cat" and "Cat, second time" are the simply the
same figures relabelled.
The most suprising figure is that local file access is 8 times faster
than accessing cached files through the MVFS. This doesn't jibe with
Atria's claim that MVFS performance is 80% of native. We must assume
the 80% number only applies when the client and server are on separate
machines, when the file transfer is limited by network performance.
We welcome attempts to reproduce or refute these figures. We wouldn't
have believed them either if we hadn't seen them with our own eyes.
CLEARCASE TIMING
Activity # files size KB time KB/sec files/sec
----------------------------------------------------------------------
Add new files 547 2300 5580 0.41 0.10
Checkout 35 420 62 6.77 0.56
Checkin 35 420 56 7.50 0.62
Checkout+Checkin 35 420 118 3.56 0.30
Cat, first time 547 2300 281 8.19 1.95
Cat, second time 547 2300 35 65.71 15.63
Label 547 2300 103 22.33 5.31
Branch 35 420 40 10.50 0.88
PERFORCE TIMING
Activity # files size KB time KB/sec files/sec
----------------------------------------------------------------------
PERFORCE add 750 3900 42 92.86 17.86
PERFORCE submit of add 750 3900 118 33.05 6.36
Add new files 750 3900 160 24.38 4.69
PERFORCE edit 299 1300 4 325.00 74.75
PERFORCE submit of edit 299 1300 94 13.83 3.18
Checkout+Checkin 299 1300 98 13.27 3.05
PERFORCE get 750 3900 47 82.98 15.96
Cat 750 3900 6 650.00 125.00
Cat, first time 750 3900 53 73.58 14.15
Cat, second time 750 3900 6 650.00 125.00
Label 750 3900 1 3900.00 750.00
PERFORCE integ branch 750 3900 10 390.00 75.00
PERFORCE submit branch 750 3900 27 144.44 27.78
Branch 750 3900 37 105.41 20.27
PERFORCE/CLEARCASE RATIO
Activity by KB by files
---------------------------------------------
Add new files 59 x 48 x < How many times faster
Checkout+Checkin 4 x 10 x < PERFORCE is than
Cat, first time 9 x 7 x < ClearCase at various
Cat, second time 10 x 8 x < SCM activities.
Label 175 x 141 x < These numbers
Branch 10 x 23 x < are multipliers!