The following dependencies are necessary to build kudu:

• autoconf

• automake

• boost >= 1.41, on CentOS boost-devel and boost-static, including the following components:

• boost::thread

• boost::system (thread depends on it)

• curl

• gcc and g++ 4.4 or higher

• liboauth-dev

• libssl-dev (Ubuntu) or openssl-devel (CentOS)

• libsasl2-dev (Ubuntu) or cyrus-sasl-devel (CentOS)

• libtool

• patch

• python 2.6 or later

• ntp

• unzip

• pkg-config

Earlier versions of each of these may work, but have not recently been tested.

Using Debian based distributions you can use the following command to install all the required packages: sudo apt-get install autoconf automake curl gcc liboauth-dev libssl-dev\ libsasl2-dev libtool patch python ntp unzip libboost-all-dev pkg-config

Kudu depends on a number of third-party libraries and tools. In order to ensure that developers have consistent versions of these libraries, the Kudu repository includes scripts to automatically download and build these dependencies, installing them into a source-tree-local install prefix.

To download, build, and install the thirdparty dependencies, you can issue the following command:

When this command completes, the dependencies will be installed in thirdparty/installed/. If you are looking to manually run the protobuf compiler protoc, the pprof tool, etc, you can find these in thirdparty/installed/bin/

The script is also invoked by cmake, so new thirdparty dependencies added by other developers will be downloaded and built automatically in subsequent builds if necessary.

To disable the automatic invocation of build-if-necessary.sh, set the NO_REBUILD_THIRDPARTY environment variable:

This can be particularly useful when trying to run tools like git bisect between two commits which may have different dependencies.

The build artifacts, including the test binaries, will be stored in build/latest/, which itself is a symlink to a build-type specific directory such as build/debug or build/release.

To omit the Kudu unit tests during the build, add -DNO_TESTS=1 to the invocation of cmake. For example:

AddressSanitizer is a nice clang feature which can detect many types of memory errors. The Jenkins setup for kudu runs these tests automatically on a regular basis, but if you make large changes it can be a good idea to run it locally before pushing. To do so, you’ll need to build using clang:

The tests will run significantly slower than without ASAN enabled, and if any memory error occurs, the test that triggered it will fail. You can then use a command like:

to get a proper symbolized stack trace.

Similar to the above, you can use a special set of clang flags to enable the Undefined Behavior Sanitizer. This will generate errors on certain pieces of code which may not themselves crash but rely on behavior which isn’t defined by the C++ standard (and thus are likely bugs). To enable UBSAN, follow the same directions as for ASAN above, but pass the -DKUDU_USE_UBSAN=1 flag to the cmake invocation.

In order to get a stack trace from UBSan, you can use gdb on the failing test, and set a breakpoint as follows:

Then, when the breakpoint fires, gather a backtrace as usual using the bt command.

You can also run the tests with a tcmalloc feature that prints an error message and aborts if it detects memory leaks in your program.

ThreadSanitizer (TSAN) is a clang feature which can detect improperly synchronized access to data along with many other threading bugs. To enable TSAN, pass -DKUDU_USE_TSAN=1 to the cmake invocation, recompile, and run tests.

1. Enabling TSAN supressions while running tests

TSAN may truncate a few lines of the stack trace when reporting where the error is. This can be bewildering. It’s documented for TSANv1 here: http://code.google.com/p/data-race-test/wiki/ThreadSanitizerAlgorithm It is not mentioned in the documentation for TSANv2, but has been observed. In order to find out what is really happening, set a breakpoint on the TSAN report in GDB using the following incantation:

In order to generate a code coverage report, you must build with gcc (not clang) and use the following flags:

This will generate the code coverage files with extensions .gcno and .gcda. You can then use a tool like lcov or gcovr to visualize the results. For example, using gcovr:

Or using lcov (which seems to produce better HTML output):

Kudu uses cpplint.py from Google to enforce coding style guidelines. You can run the lint checks via cmake using the ilint target:

This will scan any file which is dirty in your working tree, or changed since the last gerrit-integrated upstream change in your git log. If you really want to do a full scan of the source tree, you may use the lint target instead.

Kudu’s documentation is written in asciidoc and lives in the docs subdirectory.

To build the documentation, use the docs target:

This will invoke asciidoctor to process the doc sources and produce the HTML documentation, emitted to build/docs. The target expects to find asciidoctor on the system path. To install it, make sure you have Ruby installed first, then issue the following command as root:

Or, if you’d prefer to install asciidoctor without root, do:

If asciidoctor is installed in your user directory, it probably won’t be found in your PATH. You’ll need to modify PATH when building the docs, using something like this (make sure to replace 2.1.0 with your Ruby version):

To update the documentation that is integrated into the Kudu web site, you need to first check out another copy of this repository, with the 'gh-pages' branch checked out.

For example, you can check out a shallow clone which shares its objects with your main repository using a command like:

Additionally, you’ll need to ensure that the tilt and jekyll Ruby gems are installed on your machine. Refer to the asciidoctor instructions above for instructions.

Now you can build the docs and pass the path to this checked-out repository:

You can proceed to commit the changes in the pages repository and send a code review for your changes. In the future, this step will be automated whenever changes are checked into the main Kudu repository.

The kudu build is compatible with ccache. Simply install your distro’s ccache package, prepend /usr/lib/ccache to your PATH, and watch your object files get cached. Link times won’t be affected, but you will see a noticeable improvement in compilation times. You may also want to increase the size of your cache using "ccache -M new_size".

One of the major time sinks in the Kudu build is linking. GNU ld is historically quite slow at linking large C++ applications. The alternative linker gold is much better at it. It’s part of the binutils package in modern distros (try binutils-gold in older ones). To enable it, simply repoint the /usr/bin/ld symlink from ld.bfd to ld.gold.

Note that gold doesn’t handle weak symbol overrides properly (see this bug report for details). As such, it cannot be used with shared objects (see below) because it’ll cause tcmalloc’s alternative malloc implementation to be ignored.

Kudu can be built into shared objects, which, when used with ccache, can result in a dramatic build time improvement in the steady state. Even after a make clean in the build tree, all object files can be served from ccache. By default, debug and fastdebug will use dynamic linking, while other build types will use static linking. To enable dynamic linking explicitly, run:

Subsequent builds will create shared objects instead of archives and use them when linking the kudu binaries and unit tests. The full range of options for KUDU_LINK are static, dynamic, and auto. The default is auto and only the first letter matters for the purpose of matching.