Downloading a private release asset from GitHub given only its name
and tag requires a complicated series of interactions with the GitHub
API. This blog post explains how to write two repository rules
which make dealing with private release assets in Bazel easy.
Bazel supports scaling out builds with a remote execution system. Unfortunately,
it is very easy for ruleset authors to release rules that work when executed
locally but do not work when executed remotely. This blog post explains ruleset
authors can set up a simple remote execution system to verify that their
rulesets work correctly.
In Bazel, stamping is the process of embedding additional information into built
binaries, such as the source control revision or other workspace-related information.
Rules that support stamping typically include an integer stamp attribute, where
1 means “always stamp”, 0 means “never stamp”, and -1 means “use the Bazel
build --stamp flag. This blog post explains how to write a rule that supports
these values.
Many Bazel attributes support the use of predefined variables
and functions such as @D for output directory or
$(location //foo:bar) to get the path to a label. But what
if you want to apply some sort of tranformation to these
variables, or define your own custom make variables? This
blog post explains how.
Bazel developers are currently working on adding the ability to
retrieve secrets using a credential-helper executable, similar
to how other tools like Docker and Git handle managing secrets.
Until then, the recommended approach is to store secrets in
~/.netrc. This blog post explains how to write a custom Bazel
rule which reads secrets from ~/.netrc.
In general, one should never check in binary artifacts into Git; it is better
to retrieve them from an artifact repository or a website using http_archive().
However, sometimes convenience is more important than ideological purity.
To handle these cases, I wrote a simple workspace rule named local_archive().
When writing Bazel tests using sh_test(), I often find myself needing to compare
two collections for equivalence. For example, I might compare a directory listing
against a set of expected files or directories, or the list of files and directories
in a .tar file against a set of expected items. This blog post provides some tips
and tricks as to how to do so.
Dealing with Bazel runfiles is one of the most annoying things about using Bazel.
Fortunately, Bazel provides a library to make resolving runfiles from Bash scripts
easy.
sh_test is my most commonly used test rule by far. It is the easiest way to
write quick-and-dirty tests and works nearly everywhere. For anything beyond
the most trivial tests, I use Bazel’s Bash unit test framework. This explains
what the framework is and how to use it.
When creating a new ruleset, particularly on GitHub, start with the
official Bazel rules template. It includes a number of features
out of the box that are rather tiresome to implement yourself.
At work, we have a number of custom-written Bazel rulesets stored in organization
repositories on GitHub1. This post explains how we use these non-public
rulesets in our Bazel projects.
For most Bazel projects, I strongly recommend using a single Bazel
workspace per source code repository. However, it can be occasionally
useful to nest multiple workspaces within a single repository. For
example, when I’m writing Bazel rulesets, I will often create test
cases that contain own workspace with a slightly different configuration
in order to test various workspace-level configuration settings
for the ruleset, while maintaining a root workspace which is the
primary workspace for the ruleset.
As developers know, virtually all compilers support the notion
of compilation mode, e.g.. whether to compile the code in debug
or release mode. In Bazel, this concept is built natively into
the build system itself. When compiling software using bazel build,
one can select a compilation mode using the --compilation_mode
(often shorted to -c) flag.
When writing a custom Bazel ruleset, it is important to carefully
separate its public interface from its private implementation
and be deliberate and careful about changes to the public
interface. Here’s the pattern I use when I’m writing rulesets to
handle this.
Let’s say you are using Bazel to build a C program which links against
a system-provided version of libcurl, the multiprotocol file transfer
library. What is the best way to link your program against this library
within Bazel? This blog post provides an answer to that question.
This post describes a pattern for implementing a continuous integration
(CI) pipeline using Bazel. This pattern is my starting point whenever
I set up a new Bazel-based project in CI, after which I add any
project-specific pipeline customizations.
This pipeline is purely about the CI (build to release) stages of a
pipeline. A full continuous delivery (CD) pipeline, which includes
deployment, will be discussed in a later post.
In Bazel, a successful build should be a quiet build. While build failures
should, of course, print ample information to stderr to aide in troubleshooting,
any custom Bazel code you write should not output progress information to stdout
or stderr. Let Bazel be responsible for overall build progress reporting.
If you are passing secrets via environment variables that are retrieved
by command-line programs, there’s an even easier way to do it – use the
command rule from Atlassian’s bazel-tools repo and its
raw_environment attribute.
An executable rule which can be executed via bazel run is the natural
way to model interactions with external systems in Bazel such as uploading
build artifacts to a remote artifact repository. For example, imagine
a rules_artifactory ruleset which includes a rule artifactory_push()
executable rule which uploads a compiled .dpkg to an Artifactory
apt repository, or a rules_docker ruleset which has a rule
docker_push() which pushes a Docker image to a remote image repository.
When writing custom rules, you often need to invoke executables with
argument lists. For example, let’s say you are writing a custom rule
that executes gcc to compile a set of input source files. You
could write:
Bazel started on Linux and Mac OS, and most people use Bazel on these
platforms exclusively, but Bazel can execute on Windows as well. However,
Windows has enough idiosynchatic differences that writing a single,
operating-system agnostic rule that executes on both Windows and Linux/Mac
is quite hard. Often it is easiest to have the rule detect whether
it is running on Windows and execute different behavior.
When writing a custom rule that generates files, be sure to add
prefixes to all filenames so that multiple instances of your rule
can be instantiated within the same Bazel package.
One tends to write a lot of Bash scripts when using Bazel. In
order to make these scripts more robust, enable
Bash’s unofficial strict mode by starting all scripts
like this:
Bazel requires all files to end with LF (not CR-LF) in order to work correctly.
If you are performing cross-platform Bazel development with Windows users, you
can force this setting for all text files in your repo by creating a
.gitattributes file with the following content:
Quick Bazel tip for today: If you want to build everything except for a specific subtree, you can prefix the subtree you want to exclude with a -.
For example, to build everything except for //client_access_library/...:
bazel build -- //... -//client_access_library/...
Bazel’s philosophy strongly encourages binding to exact, specific versions of all third-party dependencies to help ensure reproducible builds. As Bazel users, we must remember to extend this philosophy to Bazel itself.
When setting up a Bazel-based build system, you should choose a specific version of Bazel and require all developers and the build system to use it. This can be done in a few ways:
Use Bazelisk and a .
Read more...
When writing custom Bazel rules, you spend a lot of time either reading or writing Bazel File objects.
File objects have two properties for accessing the underlying file path: File.path and File.short_path. When writing custom rules, I often chose one of the two properties at random, and switched to the other if it didn’t work right.
I wrote some simple custom rules to test the various combination of rule types and file types to determine when I should use path or short_path.
Read more...
Bazel is a powerful yet complicated system, and it can be intimidating to newcomers.
While the Bazel user guide and user manual preach the benefits of giving Bazel full control over your build process by rewriting all build processes using Bazel-native rulesets (as Google reportedly does internally), this is an immense amount of work. Specifically, if you are integrating third-party software into your Bazel-based build process, reverse engineering and rewriting the third-party project’s build system into Bazel can easily take days – and then you need to maintain it.
Read more...
In 2020, I led the redesign and re-implementation of the object storage system behind RelativityOne.
As part of this project we reengineered the continuous delivery pipeline of the service to embrace the philosophy of a service-wide monorepo with a Bazel-based build system. We chose Bazel because we wanted a build system that could support many different languages (the service has code written in C, C#, Python, Go, Terraform, Packer, and other languages…) while remaining fast and correct.
Read more...