PNetMark FAQ

1. What is PNetMark?

The purpose of this tool is to identify areas of Portable.NET that may need further optimization. It is not intended to compare Portable.NET with other CLR implementations.

3. How do I build PNetMark?

$ gunzip pnetmark-VERSION.tar.gz | tar xvf -
$ cd pnetmark-VERSION
$ ./configure
$ make

$ ./configure --with-pnet=../pnet-0.5.0 --with-pnetlib=../pnetlib-0.5.0

4. How do I run PNetMark?

$ make check

$ ilrun pnetmark.exe

5. What do the benchmarks really indicate?

Never believe benchmark numbers for engines that you cannot inspect the source code for. Vendors have been known to modify their engines specifically to get high scores on benchmarks. If you cannot see what hacks they've added to the code to lie to the benchmark, then you shouldn't believe the numbers.

For example, some JIT's get a disproportionately large value for loop benchmarks, but that is probably due to the JIT optimising the entire benchmark away, as it has no side effects. Normal applications don't use loops without side-effects, so such an optimisation is useless in practical scenarios.

A JIT that lies to a benchmark may appear to be faster, but in fact will be slower. Lying JIT's spend extra time checking for optimisations that will not be needed in real applications. This extra checking slows the JIT down, and so real applications run slower than on an honest engine.

Never believe comparions between engines from different vendors. Different implementation techniques lead to different trade-offs. Benchmarks don't accurately measure these trade-offs.

For example, it is possible for an interpreter to out-perform a JIT if the application is I/O bound, and the interpreter has been optimised for I/O operations. A JIT that can perform fantastic loop optimisations will be useless on such an application.

It is also possible for an interpreter to out-perform a JIT in other ways. Some JIT's expend a huge amount of memory to create temporary data structures to assist them with the code conversion process. The overhead of allocating and managing all of this memory will have a noticeable impact on the performance of larger applications. Smaller benchmark applications won't show up this problem.

Finally, remember that most applications spend the bulk of their time waiting for the user to press a key or move the mouse, or waiting for a remote process to respond to a request. No amount of fancy optimisations can speed up the user or the remote machine.

8. How do I add a new benchmark?

Initialize

Initialize the benchmark. This is called once per benchmark.

Name

Name of the benchmark, for reporting purposes.

Magnification

An integer value that is used to scale the number of runs to produce a usable score value.

Run

Run the benchmark once. This will be called repeatedly until the caller is satisified that it has collected sufficient information.

CleanUp

Clean up any temporary data that was used by the benchmark. This is called once per benchmark.

Run(new FooBenchmark());