Earthly ####### Earthly_ is a CI and development tool that containerizes aspects of the CI pipeline so that they run consistently across hosts and across time. .. highlight:: console .. _earthly: https://earthly.dev .. _earthly secrets: https://docs.earthly.dev/docs/guides/secrets .. _docker: https://www.docker.com/ .. _podman: https://podman.io/ Running Earthly *************** .. note:: Before you can run Earthly, you will need either Podman_ or Docker_ installed on your system. If you have trouble getting Earthly to work with Podman, refer to `the Earthly Podman Guide`__. __ https://docs.earthly.dev/docs/guides/podman While it is possible to download and install Earthly_ on your system, this task itself is automated by scripting within the ``mongo-c-driver`` repository. To run Earthly from the ``mongo-c-driver`` repository, use `tools/earthly.sh`. .. script:: tools/earthly.sh This script will download and cache an Earthly executable on-the-fly and execute it with the same command-line arguments that were passed to the script. For any ``earthly`` command, you may run `tools/earthly.sh` in its place. .. code-block:: console :caption: Example Earthly output $ ./tools/earthly.sh --version earthly-linux-amd64 version v0.8.3 70916968c9b1cbc764c4a4d4d137eb9921e97a1f linux/amd64; EndeavourOS Running Earthly via this script ensures that the same Earthly version is used across all developer and CI systems. .. envvar:: EARTHLY_VERSION The `tools/earthly.sh` script inspects the `EARTHLY_VERSION` environment variable and downloads+executes that version of Earthly. This allows one to test new Earthly versions without modifying the `tools/earthly.sh` script. This environment variable has a default value, so specifying it is not required. Updating the default value in `tools/earthly.sh` will change the default version of Earthly that is used by the script. Testing Earthly =============== To verify that Earthly is running, execute the ``+init`` Earthly target with ``--from=alpine``. This will exercise most Earthly functionality without requiring any special parameters or modifying the working directory:: $ ./tools/earthly.sh +init --from=alpine Init 🚀 ———————————————————————————————————————————————————————————————————————————————— [... snip ...] ========================== 🌍 Earthly Build ✅ SUCCESS ========================== Earthly Targets *************** This section documents some of the available Earthly targets contained in the top-level ``Earthfile`` in the repository. More are defined, and can be enumerated using ``earthly ls`` or ``earthly doc`` in the root of the repository. Build & Test Tasks ================== .. program:: +init .. earthly-target:: +init Initializes basic environment properties. Requires from `--from` argument .. option:: --from .. |--from| replace:: :option:`--from <+init --from>` :required: Specify the basis image. Should be a valid container image identifier. This target parameter is propagated to several other targets. The first step in `+init` and any targets derived from it is to execute a ``FROM `` line using this image identifier. .. program:: +build-environment .. earthly-target:: +build-environment Runs `+init` and installs all build-time dependencies for ``mongo-c-driver``. Requires |--from|. This target will automatically decide the best was to install dependencies based on the container image selected using |--from|. .. rubric:: Parameters .. option:: --c_compiler {gcc,clang} :required: Set the C compiler to be installed. .. option:: --cxx_compiler {gcc,clang,none} :required: Sets the C++ compiler to be installed. if ``none``, then no C++ compiler will be added to the container environment. .. option:: --snappy :required: Whether to install Snappy in the environment. .. option:: --tls {OpenSSL,off} :required: Toggle the TLS implementation to be used. Currently, only supports ``OpenSSL`` or ``off`` to disable TLS support. .. option:: --sasl {Cyrus,off} :required: Toggle the SASL implementation to use. .. option:: --ccache :default: ``on`` :optional: Toggle whether the build will attempt to add and use Ccache. .. option:: --lld :default: ``on`` :optional: Toggle whether the build will attempt to install and use the LLD linker. .. program:: +configure .. earthly-target:: +configure Runs `+build-environment` and then configures the project using CMake. Requires all the same arguments as `+build-environment`. .. program:: +build .. earthly-target:: +build Runs `+configure` and then builds ``mongo-c-driver``. Requires all arguments for `+configure`, transitively. .. earthly-artifact:: +build/build-tree/ A directory containing the CMake build tree created by the task. .. earthly-artifact:: +build/root/ A directory containing the installation result for ``mongo-c-driver``. This directory contains the top-level ``include/``, ``lib/``, etc. directories generated by ``cmake --install``. .. program:: +verify-headers .. earthly-target:: +verify-headers This runs `CMake's header verification`__ on the library sources, to ensure that the public API headers can be ``#include``\ 'd directly in a C++ compiler. __ https://cmake.org/cmake/help/latest/prop_tgt/VERIFY_INTERFACE_HEADER_SETS.html This target does not produce any output artifacts. This only checks that our public API headers are valid. This checks against a variety of environments to test that we are including the necessary standard library headers in our public API headers. .. program:: +deb.packages .. earthly-target:: +deb.packages Build Debian packages based on the current working copy. .. earthly-artifact:: +deb.packages/ The root artifact directory contains all the files that are generated by the dpkg build. .. rubric:: Parameters .. option:: --debian_version Control the Debian container version used for the build. .. program:: +deb.test .. earthly-target:: +deb.test Test the packages that are built by the `+deb.packages` target. Also supports the `+deb.packages --debian_version` parameter. Release Tasks ============= .. program:: +signed-release .. earthly-target:: +signed-release Creates signed release artifacts using `+release-archive` and `+sign-file`. .. seealso:: `releasing.gen-archive`, which uses this target. .. earthly-artifact:: +signed-release/dist/ A directory artifact that contains the `+release-archive/release.tar.gz`, `+release-archive/ssdlc_compliance_report.md`, and `+sign-file/signature.asc` for the release. The exported filenames are based on the `--version` argument. .. option:: --version Affects the output filename and archive prefix paths in `+signed-release/dist/` and sets the default value for `--ref`. .. option:: --ref Specify the git revision to be archived. Forwarded to `+release-archive --ref`. If unspecified, archives the Git tag corresponding to `--version`. .. rubric:: Secrets Secrets for the `+snyk-test` and `+sign-file` targets are required for this target. .. program:: +release-archive .. earthly-target:: +release-archive Generate a source release archive of the repository for the specified branch. Requires the secrets for `+snyk-test`. Requires ``etc/augmented-sbom.json`` is present (obtained from Evergreen). .. earthly-artifact:: +release-archive/release.tar.gz The resulting source distribution archive for the specified branch. The generated archive includes the source tree, but also includes other release artifacts that are generated on-the-fly when invoked. .. earthly-artifact:: +release-archive/ssdlc_compliance_report.md The SSDLC compliance report for the release. This file is based on the content of ``etc/ssdlc.md``, which has certain substrings replaced based on attributes of the release. .. option:: --ref Specifies the Git revision that is used when we use ``git archive`` to generate the repository archive snapshot. Use of ``git archive`` ensures that the correct contents are included in the archive (i.e. it won't include local changes and ignored files). This also allows a release snapshot to be taken for a non-active branch. .. option:: --prefix Specify a filepath prefix to appear in the generated filepaths. This has no effect on the files archived, which is selected by `+release-archive --ref`. .. program:: +sbom-validate .. earthly-target:: +sbom-validate Validate the `etc/cyclonedx.sbom.json`. .. program:: +sign-file .. earthly-target:: +sign-file Signs a file using Garasign. Use of this target requires authenticating against the DevProd-provided Amazon ECR instance! (Refer to: `earthly.amazon-ecr`) .. earthly-artifact:: +sign-file/signature.asc The detached PGP signature for the input file. .. rubric:: Parameters .. option:: --file :required: Specify a path to a file (on the host) to be signed. This file must be a descendant of the directory that contains the ``Earthfile`` and must not be excluded by an ``.earthlyignore`` file (it is copied into the container using the COPY__ command.) __ https://docs.earthly.dev/docs/earthfile#copy .. rubric:: Secrets .. envvar:: GRS_CONFIG_USER1_PASSWORD GRS_CONFIG_USER1_USERNAME :required: [#creds]_ .. seealso:: `earthly.secrets` .. earthly-target:: +sbom-generate Updates the `etc/cyclonedx.sbom.json` file **in-place** based on the contents of `etc/purls.txt` and the existing `etc/cyclonedx.sbom.json`. After running this target, the contents of the `etc/cyclonedx.sbom.json` file may change. .. seealso:: `sbom-lite` and `sbom-lite-updating` .. program:: +sbom-generate-new-serial-number .. earthly-target:: +sbom-generate-new-serial-number Equivalent to `+sbom-generate` but uses the ``--generate-new-serial-number`` flag to generate a new unique serial number and reset the SBOM version to 1. After running this target, the contents of the `etc/cyclonedx.sbom.json` file may change. .. seealso:: `sbom-lite` and `sbom-lite-updating` .. program:: +snyk-monitor-snapshot .. earthly-target:: +snyk-monitor-snapshot Executes `snyk monitor`__ on a crafted snapshot of the remote repository. This target specifically avoids an issue outlined in `snyk scanning` (See "Caveats"). Clones the repository at the given `--branch` for the snapshot being taken. __ https://docs.snyk.io/snyk-cli/commands/monitor .. seealso:: Release process step: `releasing.snyk` .. rubric:: Parameters .. option:: --branch :required: The name of the branch or tag to be snapshot. .. option:: --name :required: The name for the monitored snapshot ("target reference") to be stored in the Snyk server. .. note:: If a target with this name already exists in the Snyk server, then executing `+snyk-monitor-snapshot` will replace that target. .. option:: --remote The repository to be snapshot and posted to Snyk for monitoring. Defaults to the upstream repository URL. Use ``"local"`` to snapshot the repository in the working directory (not recommended except for testing). .. rubric:: Secrets .. envvar:: SNYK_ORGANIZATION The API ID of the Snyk_ organization that owns the Snyk target. For the C driver, this secret must be set to the value for the organization ID of the MongoDB **dev-prod** Snyk organization. **Do not** use the organization ID of **mongodb-default**. The `SNYK_ORGANIZATION` may be obtained from the `Snyk organization page `_. .. _snyk: https://app.snyk.io .. envvar:: SNYK_TOKEN Set this to the value of an API token for accessing Snyk in the given `SNYK_ORGANIZATION`. The `SNYK_TOKEN` may be obtained from the `Snyk account page `_. .. program:: +snyk-test .. earthly-target:: +snyk-test Execute `snyk test`__ on the local copy. This target specifically avoids an issue outlined in `Snyk Scanning > Caveats `. __ https://docs.snyk.io/snyk-cli/commands/test .. earthly-artifact:: +snyk-test/snyk.json The Snyk JSON data result of the scan. .. rubric:: Secrets .. envvar:: SNYK_TOKEN :noindex: See: `SNYK_TOKEN` .. _earthly.secrets: Setting Earthly Secrets *********************** Some of the above targets require defining `earthly secrets`_\ [#creds]_. To pass secrets to Earthly, it is easiest to use a ``.secret`` file in the root of the repository. Earthly will implicitly read this file for secrets required during execution. Your ``.secret`` file will look something like this: .. code-block:: ini :caption: Example ``.secret`` file content GRS_CONFIG_USER1_USERNAME=john.doe GRS_CONFIG_USER1_PASSWORD=hunter2 .. warning:: Earthly supports passing secrets on the command line, **but this is not recommended** as the secrets will then be stored in shell history. Shell history can be supressed by prefixing a command with an extra space, but this is more cumbersome than using environment variables or a ``.secret`` file. .. seealso:: `The Earthly documentation on passing secrets `_ .. _earthly.amazon-ecr: Authenticating with Amazon ECR ****************************** In order to run `+sign-file` or any target that depends upon it, the container engine client\ [#oci]_ will need to be authenticated with the DevProd-provided Amazon ECR instance using AWS CLI v2:: # Forward the short-term AWS credentials to the container engine client. $ aws ecr get-login-password --profile | podman login --username AWS --password-stdin 901841024863.dkr.ecr.us-east-1.amazonaws.com Configure the AWS profile using ``aws configure sso`` or modifying the ``$HOME/.aws/config`` file such that: - The SSO start URL is ``https://d-9067613a84.awsapps.com/start#/``. - The SSO and client region are ``us-east-1``. - The SSO registration scope is ``sso:account:access`` (default). - The SSO account ID is ``901841024863`` (aka ``devprod-platforms-ecr``). - The SSO role name is ``ECRScopedAccess`` (default). To refresh short-term credentials when they have expired, run ``aws sso login --profile `` followed by the same ``aws ecr get-login-password ... | podman login ...`` command described above. .. seealso:: `"DevProd Platforms Container Registry" `_ and `"Configuring IAM Identity Center authentication with the AWS CLI" `_. .. [#oci] You container engine client will probably be Docker or Podman. Wherever the :bash:`podman` command is used, :bash:`docker` should also work equivalently. .. [#creds] Credentials are expected to be available in `AWS Secrets Manager `_ under ``drivers/c-driver``.