Build System

The NUbots Build System

Overview

The NUbots codebase uses a CMake build system based on the NUClear Roles framework and the Ninja build system.

Docker is used to containerise our build processes so that we may mimic the build environment of the robot and impose version control over our required 3rd party dependencies while also managing operating system level configuration to ensure that all users of our codebase can achieve repeatable results.

Requirements and Setup

See the Getting Started guide for setup instructions and how to build the code on Linux, macOS, and Windows.

NUClear Roles

The most complete documentation for NUClear Roles can be found in its github README.

Six main directories are used in a NUClear Roles system:

Modules

The modules folder contains NUClear reactors. Reactors are grouped into meta-modules. For instance, the vision meta-module groups together reactors that perform vision processing tasks (e.g. ball and goal detection), while the input meta-module groups together reactors that gather system inputs (e.g. camera and network inputs).

Roles

Role files define which modules should be grouped together in order to form a single role (a binary that performs a specific role).

Role files live in the roles folder and have a .role extension.

Every role file must call the nuclear_role() function. Each argument to the nuclear_role() function is a C++ namespaced path to a module that should be included in the role. For example, if the KinematicsConfiguration module in module/actuation/KinematicsConfiguration should be included in the role then actuation::KinematicsConfiguration should appear as an argument to nuclear_role().

Ideally, all modules should be independent of each other and, as such, it should not matter which order modules are specified to nuclear_role(). In reality, there are a couple of modules which, if they are included in a role, should be passed to nuclear_role() first. These modules are:

• extension::FileWatcher: Needed before all other modules so that the configuration and script systems can work properly
• support::SignalCatcher: Catches exceptions to print out useful stack traces. Needs to come before all other modules to allow debugging during module installation
• support::logging::ConsoleLogHandler or support::logging::FileLogHandler: Install before other modules so that log messages will get printed during module installation
• actuation::KinematicsConfiguration: Needs to come before other modules that require information about the robots kinematic configuration in order to set up properly

Below is an example of a role file. Any line preceded by a # is a comment and will be ignored.

Copy
nuclear_role(
  # FileWatcher, ConsoleLogHandler and Signal Catcher Must Go First. KinematicsConfiguration usually goes after these
  # and without it many roles do not run
  extension::FileWatcher
  support::SignalCatcher
  support::logging::ConsoleLogHandler
  # This must come first as it emits config which many roles depend on (e.g. SensorFilter, WalkEngine)
  actuation::KinematicsConfiguration
  support::configuration::GlobalConfig
  # Networking
  network::NetworkForwarder
  network::NUClearNet
  # Sensors
  platform::${SUBCONTROLLER}::HardwareIO
  input::SensorFilter
)

Extensions

Extensions add new keywords to NUClear's dictionary, allowing you to be more expressive in writing reactions.

Two common extensions are Configuration and Director. Configuration allows modules to monitor their configuration file(s) and apply updates in real-time. Director contains our behaviour algorithm and framework.

Messages

Every message in the NUbots codebase is a protobuf message and all protobuf messages live in the shared/message folder.

The general convention is to groups message files into folders based on the meta-module which introduces the message to the system. For instance, module/input/Camera introduces Image messages to the system, so Image.proto lives in shared/message/input.

NUClear Roles provides special wrappers for certain types. Vector and matrix types are mapped to Eigen types. To use these mappings simply use import Vector.proto; and import Matrix.proto;. Some examples of types that you can use are:

• fvec3: A vector with 3 components, maps to an Eigen::Vector3f
• vec3: A vector with 3 components, maps to an Eigen::Vector3d
• mat3: A 3x3 matrix, maps to an Eigen::Matrix3d

All messages are packaged in a similar fashion to the C++ namespaces used for modules. For instance, Image.proto is in the message.input package.

NUClear Roles creates a C++ wrapper class for each message to create a simpler interface for accessing and mutating message fields. The C++ namespace for a message follows its package. For instance, the message.input.Image message will have a C++ namespace of message::input::Image.

Utilities

Utilities are collections of functions that perform useful tasks. Typically, these functions are stateless and are (or feasibly could be) used by multiple different modules (or even other utilities).

As is the case with modules and messages, utilities are grouped together into meta-utilities. For instance, utilities dealing with angles, coordinate system transforms, and geometrical constructs live in the shared/utility/math meta-utility.

Tools and the b command

NUClear Roles introduces the b command that wraps up all of the functionality needed to create modules and build the NUbots code.

Build

As one might expect, ./b build will build all enabled roles and required dependencies using ninja. If you only wish to build a single role (e.g. test/sensor) then use the command

Copy
./b build test/sensor

If unit tests are enabled in the CMake configuration (-DBUILD_TESTS=ON) then ./b build test can be run to execute all of the unit tests. Test logs can be accessed by running ./b shell and navigating to /home/nubots/build/Testing/Temporary.

Configure

./b configure will ensure that a build folder is created in the current Docker image and will then run cmake on the source code ensuring that CMake creates all of the necessary build files inside of the build folder.

It is possible to specify extra arguments to cmake to control the configuration process. For example, to enable the building of unit tests one would run

Copy
./b configure -- -DBUILD_TESTS=ON

Another good argument to specify is CMAKE_BUILD_TYPE. By default, CMAKE_BUILD_TYPE is set to Release, which has better performance than Debug but does not include debug symbols. If you would like to debug your code with debugging tools, such as GDB or ASan, then you should set CMAKE_BUILD_TYPE to Debug.

Copy
./b configure -- -DCMAKE_BUILD_TYPE=Debug
./b build

Passing -i to ./b configure will open up an interactive window with ccmake so that you may modify the CMake configuration.

You can also set and unset roles by passing --set_roles or --unset_roles followed the roles you want to toggle (this supports globbing patterns). For example, you can disable all roles ending in walk, then enable keyboardwalk and all roles starting with script using the following command.

Copy
./b configure --unset_all_roles "*walk" --set_roles keyboardwalk "script*"

Roles selected with --unset flags will always be disabled before roles selected with --set flags are enabled.

You can also toggle groups of roles with --set_<group>_roles and --unset_<group>_roles. Currently there is a role group for each subdirectory in roles and a group containing all roles. For example, you can clean all docker build volumes and enable only roles in the roles/webots directory using the following command.

Copy
./b configure --clean --unset_all_roles --set_webots_roles

Edit

If you are running roles on your local machine and you need to edit the configuration or data files for the modules in the role then the preferred method is to edit the file in the source tree and then rebuild the code using ./b build to copy the edited file into the build folder.

However, it is possible to edit the configuration file in the build tree directly. This may be useful if you want to change the configuration file during runtime. To do this, run ./b edit <path/to/config_file> to open an interactive window with a text editor.

For example, to edit the DataLogging configuration file run

Copy
./b edit config/DataLogging.yaml

The edit command uses the editor that is defined in your host shell (using the EDITOR environment variable). If EDITOR is not set then it will default to nano. Currently, the only supported editors are vim and nano, with nano used as the fallback in all cases.

Foot down

This tool uses NBS files recorded from the robot(s) to gather training and testing data to train a neural network to determine if either of the robot's feet are on the ground using leg servo and sensor data.

Format

NUbots uses several different code formatters.

• clang-format is used to format C++ and protobuf files,
• black and isort are used to format python files, and
• cmake-format is used to format CMake files.
• eslint and prettier are used to format NUsight's TypeScript files

If you want to ensure that all files in the codebase are formatted according to our defined style run

Copy
./b format

If you are using Visual Studio Code with the workspace recommended extensions installed then formatting can be set up to happen when you save or when you type.

Install

Once code is built and you are ready to install it on a robot, you can run

Copy
./b install [options] <robot>

<robot> can either be a known robot designation (n1 or nugus1, for instance) or an IP address of a robot.

[options] may be one, or more, of the following

The toolchain is a collection of 3rd party libraries that have been built into the Docker images. On a new robot, all roles will fail to run unless the toolchain is installed and, more often than not, if you are receiving errors on the robot saying something to the effect of "cannot open shared object file: No such file or directory" this will usually be solved by installing the toolchain.

Module

The module command allows you to generate a new module. To generate a new line detection module one would run

Copy
./b module generate vision/LineDetector

This will generate all necessary module files in the module/vision/LineDetector folder. This includes an empty configuration file, a basic C++ source and header file implementing the LineDetector reactor, and an empty unit test file.

Add the --director flag at the end of the command to generate a module that will use Director functionality.

NBS

NBS (NUClear Binary Stream) files are, effectively, concatenations of protobuf messages with a small header packet per message. They provide a simple and effective means for recording real data from the robots in real time.

The nbs command provides a couple of subcommands to help you work with NBS data.

Run

To run roles locally on your system use the run command of b as follows

Copy
./b run <name of role>

The run command provides some options to help you debug your roles.

In order to use ASan you must recompile your role after setting USE_ASAN to ON in the cmake options (./b configure -i).

When USE_ASAN=ON an environment variable is added that will direct ASan to log all of its output to a file. This is done to ensure that even when you are running a curses-enabled role (e.g. fake/keyboardwalk) you will always be able to recover the output from ASan (and it will also be clean from other log messages). There are a number of other run-time options you could set to tune the behaviour of ASan, they can be found here. Just run your role as follows

Copy
ASAN_OPTIONS=option1=value1:option2=value ./b run fake/keyboardwalk

ASan and GDB can be run together. For more details on how this works have a look at this page.

Since ASan and valgrind perform the same task they cannot be used in conjunction and an exception will be thrown if you try to do so.

While it is possible to run GDB and valgrind simultaneously, two separate processes need to be started and then linked together. While this is technically possible, it has currently been assigned to the too hard basket and, as such, this option has been disabled and an exception will be thrown if you try to do this.

When using GDB to debug curses-enabled roles (e.g. keyboardwalk) and a breakpoint is triggered or an error occurs the role will be halted and control is returned to GDB. However, the role will likely be halted in a way that prevents curses from being disabled, leaving the screen in a reasonably unreadable state. To fix this, use the reset command and the screen should return to normal. You can then use bt or bt full to get a stack trace and continue debugging as normal.

Shell

This is an undocumented and unsupported option.

Copy
./b shell

Target

The target command of b allows you to build code for a different target. Support targets are

• generic: Useful if you would like to build and run code on your local machine.
• nuc7i7bnh: For use with the old nuc7i7bnh computer. This is no longer used on the robots.
• nuc8i7beh: For use with the nuc8i7beh computer. This is not used on the robots.
• nuc12wshi7: The current target for the robots, which use the nuc12wshi7 computer.

The target command will download (or, if needed, build) the Docker image for the specified target and then mark that target as active.

Tests

The tests command allows you to run built unit tests. It will automatically run tests in parallel, and dump a timestamped log file to the project directory. Currently, we use CTest which is included as part of CMake.

In order to run unit tests, they must first be built. This can be done with:

Copy
./b configure -- -DBUILD_TESTS=ON
./b build

The tests command provides the following subcommands:

If you would like to run an individual test (or group of tests), you can pass the run subcommand a string and it will only run tests that match the given string. Alternatively, run will run all built tests if no string is passed.

For example, if you would like to run all tests with the string 'vision' in their name, you would do:

Copy
./b tests run vision

The run subcommand also accepts the following flags:

Autocomplete

To use autocomplete with b on bash run eval "$(register-python-argcomplete b)" in the NUbots folder. Now when you press tab it will finish the command you were typing. It's recommended to add it to your .bashrc, but you need to provide the full path to the NUbots folder. e.g. eval "$(register-python-argcomplete $HOME/code/NUbots/b)".

Copy
autoload -U bashcompinit
bashcompinit
eval "$(register-python-argcomplete b)"

Docker Images

The NUbots codebase uses Docker images to containerise the build process. Two images are created, one intended for running code on your local machine, named generic, and another intended for running code on the robots named nuc12wshi7. Apart from flags provided to the build tools and the compiler both images are identical (that is, they contain the same programs and libraries).

The Docker images are cached on Docker Hub to limit the amount of building that everyone has to do. The cache will be updated whenever the main branch is updated.

The Docker images use Arch Linux as their operating system as it is lightweight, flexible, and up-to-date.

The docker folder contains the Dockerfile as well as some utility scripts to ease the burden of compiling multiple different libraries. Other files in this folder end up inside of the image and control either the behaviour of the compiler or build system, or the runtime behaviour of the libraries that are built.

If you need to add a new library or program to the Docker image you should add it to the end of the Dockerfile to minimise the amount of rebuilding that needs to happen. Look for the following marker

Copy
#######################################
### ADD NEW PROGRAMS/LIBRARIES HERE ###
#######################################

Adding a new library to the robot toolchain

If the intention is to have the new library running on the robot then you need to build the library from source. If the library you are building is well-behaved then it should be a simple matter of adding a line like the following to the Dockerfile

Copy
RUN install-from-source http://url.to.the.source.tarball.com

This process generally requires some studying of either the library's documentation or build system files to determine appropriate configure and build flags. If there are extra flags that need to be specified then modify the above command to

Copy
RUN install-from-source http://url.to.the.source.tarball.com \
    -DCONFIGURE_FLAG=XXX

If your library is not well-behaved you may need to do to a lot of googling and trawling through stackoverflow, bug reports, or github issue pages in order to find workarounds or patches to help you build your library. Another good source of information is the PKGBUILD scripts from the Arch Linux repositories. Search for the name of the library, then look for and click the Source Files link on the package page and then click on PKGBUILD. The PKGBUILD script will show you how Arch Linux builds the library including

• necessary dependencies,
• locations of source files and patches,
• commands needed to prepare the source files for compilation,
• commands needed to build the library, and
• commands needed to install the library

Borrowing commands from these scripts could help relieve your frustrations with the world.

Adding a new build utility

If the library/program is not intended to be used on the robot, but is instead intended to run inside the Docker image as an extra utility to help build the NUbots code then your first port of call should be the Arch Linux packages. Search for a package here (or google "Arch Linux" and the name of the program/library) and if you meet with success add the following command to the Dockerfile at the marker near the end of the file

Copy
#######################################
### ADD NEW PROGRAMS/LIBRARIES HERE ###
#######################################
RUN install-package <name of package>

If a package does not exist in the Arch Linux repositories then you will need to build it from source. However, this time, you need to add it where the marker says

Copy
##############################################
### ADD NEW SYSTEM PROGRAMS/LIBRARIES HERE ###
##############################################

See above for details on how to build a program/library from source.

The toolchain

The toolchain is a collection of libraries that are used by the NUbots codebase. These libraries are essential for our code (if we don't have them then we are unable to compile anything).

In order to ensure that everyone is using the same versions of every library and that everyone has compiled the libraries in the same way (enabling the correct features, etc), we use Docker to automate the build process. Docker also ensures that all of the dependencies for the built libraries are either installed or are also built and that they are also built in the correct order. To see what is currently being built and installed in the toolchain have a look at the Dockerfile.

In order to improve runtime performance on the robots, all libraries that are built as part of the toolchain (as well as the NUbots code) are compiled with optimisations targeting the CPU of the robot. A list of the currently used compiler flags for the nuc12wshi7 can be seen in generate_nuc12wshi7_toolchain.py.

If you are setting up a new toolchain targetting a robot that is using a different CPU to the nuc12wshi7, log on to the new robot, install gcc (sudo pacman -S gcc) and execute the following commands

Copy
target=$(gcc -march=native -Q --help=target | grep -- '-march=' | cut -f3 | head -n1)
diff -y --suppress-common-lines <(gcc -march=native -Q --help=target) <(gcc -march=${target} -Q --help=target)

The first command will determine the codename of the CPU family (i.e. for an Intel CPU it might be broadwell, haswell, skylake, etc).

For the rest of this example we will assume that target=skylake.

The second command compares the compiler flags that compiling with -march=native would enable compared to using -march=skylake. The output of this function could look like this

Copy
-mabm        [enabled]        |    -mabm        [disabled]
-mrtm        [enabled]        |    -mrtm        [disabled]

The left column are instructions that are enabled/disabled when compiling using -march=native and the right column are the instructions that are enabled/disabled when compiling using -march=skylake. Combining both columns tells us that to get from a generic skylake CPU to the CPU that we are using we need to enable both the abm and the rtm instructions using -mabm and -mrtm.

Copy
-mabm        [disabled]       |    -mabm        [enabled]
-mrtm        [enabled]        |    -mrtm        [disabled]

The final piece of information we need is about the CPU cache sizes

Copy
gcc -### -E - -march=native 2>&1 | sed -r '/cc1/!d;s/(")|(^.* - )//g'| grep -oP -- "--param l[12]-cache(-line|)-size=[0-9]*"

The output of this command will look like this

Copy
--param l1-cache-size=32
--param l1-cache-line-size=64
--param l2-cache-size=12288

Putting all of this together, the targets dict for our example target would be

Copy
target = {
    "flags": [
        "-march=skylake",
        "-mtune=skylake",
        "-mabm",
        "-mrtm",
        "--param l1-cache-size=32",
        "--param l1-cache-line-size=64",
        "--param l2-cache-size=12288",
        "-fPIC",
    ],
    "release_flags": ["-O3", "-DNDEBUG"],
    "asm_flags": ["-DELF", "-D__x86_64__", "-DPIC"],
    "asm_object": "elf64",
}

Dependency Declaration

CMake needs to know what dependencies we have and where to find them. In a module's folder the CMakeLists.txt file tells CMake what is needed to build a module. The nuclear_module() function tells CMake that this directory contains a module that may be added to a role. This function may take named parameters.

CMake needs to know where libraries are located. The cmake/Modules/ directory contains files called FindX.cmake, where X is the name of a library. These files contain the logic for finding library files. The function find_package(X) is used to tell CMake to look for one of these .cmake files and run it.

CMake can also look for build instruction files in /lib/cmake/X/XConfig.cmake where X is the name of the library. An example of one of these files, FindAravis.cmake which finds the Aravis library, is shown below

Copy
include(ToolchainLibraryFinder)

find_package(glib2 REQUIRED)

ToolchainLibraryFinder(
  NAME Aravis
  HEADER arv.h
  LIBRARY aravis-0.8
  PATH_SUFFIX aravis-0.8
)

target_link_libraries(Aravis::Aravis INTERFACE glib2::glib2)

The first line includes a custom function for later use. We have find_package again, as this library has its own dependency. Then there is ToolchainLibraryFinder which is the function that finds the library. Finally there is target_link_libraries which tells CMake that we have a dependency of the library. ToolchainLibraryFinder parameters: