Skip to content

Benchmarks

monochrome edited this page Oct 27, 2021 · 24 revisions

Here is a preliminary report:

Preface

Each program was run on MacOS 11.1 using machine with Apple M1 (4+4 cores, 16GB RAM).
All data were compared with CRuby3.0.0.

Optcarrot benchmark results

Optcarrot is a NES emulator, and a semi-official benchmark program for Ruby, on https://github.com/mame/optcarrot.
You can see the results of optcarrot benchmark for ruruby and other Ruby implementations here.

benchmark CRuby ruruby rate
optcarrot 56.09 ± 0.13 fps 34.80 ± 0.08 fps x 1.61
optcarrot --opt 130.53 ± 0.82 fps 101.85 ± 1.08 fps x 1.28

* With --opt option, optcarrot rewrites its own code for optimization.
Frames per seconds are shown.
The higher, the better.

Execution time

benchmark CRuby ruruby rate
accessor_get.rb 0.42 ± 0.00 s 0.43 ± 0.00 s x 1.02
accessor_set.rb 0.48 ± 0.00 s 0.53 ± 0.00 s x 1.10
ivar_get.rb 0.52 ± 0.00 s 0.63 ± 0.00 s x 1.21
ivar_set.rb 0.52 ± 0.01 s 0.66 ± 0.01 s x 1.25
loop_times.rb 0.72 ± 0.01 s 0.60 ± 0.00 s x 0.84
loop_for.rb 0.78 ± 0.01 s 0.71 ± 0.00 s x 0.91
loop_whileloop.rb 0.52 ± 0.00 s 0.48 ± 0.01 s x 0.93
so_concatenate.rb 0.51 ± 0.01 s 0.48 ± 0.01 s x 0.94
string_scan_str.rb 0.63 ± 0.00 s 0.88 ± 0.01 s x 1.40
string_scan_re.rb 0.95 ± 0.00 s 0.87 ± 0.01 s x 0.92
fiber_allocate.rb 0.59 ± 0.01 s 0.41 ± 0.04 s x 0.69
fiber_switch.rb 0.64 ± 0.00 s 0.52 ± 0.01 s x 0.82
so_mandelbrot.rb 1.18 ± 0.04 s 1.36 ± 0.01 s x 1.15
app_mandelbrot.rb 0.84 ± 0.00 s 0.95 ± 0.00 s x 1.13
app_fibo.rb 0.54 ± 0.01 s 0.68 ± 0.00 s x 1.25
app_aobench.rb 5.89 ± 0.01 s 9.74 ± 0.03 s x 1.65
so_nbody.rb 0.76 ± 0.01 s 0.94 ± 0.00 s x 1.22

Mean and SD of execution time (executed 8 times) in seconds are shown.

The lower, the better.
These programs are available on https://github.com/ruby/ruby/blob/master/benchmark.

The characteristics of the benchmark programs are described below.

loop_times.rb

Simply execute a times method with block repeatedly.

loop_for.rb

Simply execute a for-loop repeatedly.

loop_whileloop.rb

Simply execute a while-loop repeatedly.

so_concatenate.rb

Join small strings repeatedly.

so_mandelbrot.rb

A lot of arythmetic calculations with local variables in for-loop, producing almost no objects.

app_mandelbrot.rb

Produce and consume a lot of objects, and execute numerous method calls.

app_fibo.rb

A huge number of recursive method calls.

app_aobench.rb

Produce and consume a lot of objects, executing a huge number of floating point calculations and method calls.

Memory consumption

To investigate memory consumption, maximum resident set size was measured using gnu-time. Memory consumption of ruruby is around 20%-25% as compared with CRuby.

benchmark CRuby ruruby rate
simple_call.rb 26.7M 3.5M x 0.13
accessor_get.rb 26.6M 3.5M x 0.13
accessor_set.rb 26.7M 3.5M x 0.13
ivar_get.rb 26.6M 3.5M x 0.13
ivar_set.rb 26.6M 3.5M x 0.13
loop_times.rb 26.7M 3.5M x 0.13
loop_for.rb 26.8M 3.5M x 0.13
loop_whileloop.rb 26.7M 3.5M x 0.13
so_concatenate.rb 75.8M 27.3M x 0.36
string_scan_str.rb 28.2M 5.7M x 0.20
string_scan_re.rb 28.5M 5.5M x 0.19
fiber_allocate.rb 209.7M 229.8M x 1.10
fiber_switch.rb 26.7M 3.5M x 0.13
so_mandelbrot.rb 27.2M 4.5M x 0.16
app_mandelbrot.rb 27.1M 4.5M x 0.16
app_fibo.rb 26.7M 3.5M x 0.13
app_aobench.rb 27.3M 5.8M x 0.21
so_nbody.rb 26.8M 3.7M x 0.14
optcarrot 100.9M 65.2M x 0.65
optcarrot --opt 117.2M 95.1M x 0.81