# VEX scans
This directory contains automations to scan our repos and their binaries for
false-positives. The identified false-positives are used to generate VEX reports
(Vulnerability Exploitability eXchange), [complete spec] and [one-page summary],
according to the [OpenVEX spec].
VEX is a document that complements the SBOM (Software Bill of Materials), by
informing users of a software product about the applicability of one or more
vulnerability findings. The applicability can be used for either reporting that
a vulnerability affects a product or that it's a known false positive.
## OpenVEX
[OpenVEX] is an open source _"implementation of the Vulnerability Exploitability
Exchange (VEX for short) that is designed to be minimal, compliant,
interoperable, and embeddable"_. It provides an evolving specification, a
library and tooling, known as [`vexctl`], to manipulate VEX reports.
A VEX report contains a `statements` entry. The `statements` entry contains
details about multiple `vulnerability`(s). Each `vulnerability` contains details
about multiple affected `products`. Each product might contain details about
`subcomponents` and the `status` of the `vulnerability` with a `justification`.
From the upstream OpenVEX spec:
```text
statement = product(s) + vulnerability + status
│ │ │
└ The software product └ Typically a CVE related └ One of the impact
we are talking about to one of the product's statuses as identified
components by the VEX working group.
```
The spec is flexible enough allowing for the product to be either a container
image, a software package (binary) or a specific dependency in the format of a
[PURL] (Package URL).
The `status` label informs the impact of a given vulnerability in the product
and must be one of the following labels (from the OpenVEX spec):
| Label | Description |
| --- | --- |
| `not_affected` | No remediation is required regarding this vulnerability. A `not_affected` status required the addition of a `justification` to the statement. |
| `affected` | Actions are recommended to remediate or address this vulnerability. |
| `fixed` | These product versions contain a fix for the vulnerability. |
| `under_investigation` | It is not yet known whether these product versions are affected by the vulnerability. Updates should be provided in further VEX documents as knowledge evolves. |
A `justification` must be provided when a `not_affected` `status` label is used.
The following labels are currently defined as valid `justification` by the VEX spec:
| Label | Description |
| --- | --- |
| `component_not_present` | The product is not affected by the vulnerability because the component is not included. The status justification may be used to preemptively inform product users who are seeking to understand a vulnerability that is widespread, receiving a lot of attention, or is in similar products. |
| `vulnerable_code_not_present` | The vulnerable component is included in artifact, but the vulnerable code is not present. Typically, this case occurs when source code is configured or built in a way that excluded the vulnerable code. |
| `vulnerable_code_not_in_execute_path` | The vulnerable code (likely in `subcomponents`) can not be executed as it is used by the product.
Typically, this case occurs when the product includes the vulnerable `subcomponent` but does not call or use the vulnerable code. |
| `vulnerable_code_cannot_be_controlled_by_adversary` | The vulnerable code cannot be controlled by an attacker to exploit the vulnerability.
This justification could be difficult to prove conclusively. |
| `inline_mitigations_already_exist` | The product includes built-in protections or features that prevent exploitation of the vulnerability. These built-in protections cannot be subverted by the attacker and cannot be configured or disabled by the user. These mitigations completely prevent exploitation based on known attack vectors.
This justification could be difficult to prove conclusively. History is littered with examples of mitigation bypasses, typically involving minor modifications of existing exploit code.
### OpenVEX document example
A minimally valid OpenVEX report is exemplified below.
```json
{
"@context": "https://openvex.dev/ns/v0.2.0",
"@id": "merged-vex-9a5803a58d60baba530411ef5d90ace6d8352f7280659a28fabbd5b8c5445d11",
"author": "Rancher Security team",
"timestamp": "2024-07-23T13:38:53.548109199Z",
"version": 1,
"statements": [
{
"vulnerability": {
"name": "GO-2021-0072",
"aliases": [
"CVE-2017-11468",
"GHSA-h62f-wm92-2cmw"
]
},
"timestamp": "2024-07-23T13:29:52.30688095Z",
"products": [
{
"@id": "pkg:golang/github.com/rancher/support-bundle-kit@v0.0.38",
"subcomponents": [
{
"@id": "pkg:golang/github.com/docker/distribution@v0.0.0-20191216044856-a8371794149d"
}
]
}
],
"status": "not_affected",
"justification": "vulnerable_code_not_present"
}
]
}
```
The document contains a statement about only one `vulnerability` identified as
`GO-2021-0072`, which is also known as, has the `aliases`, `CVE-2017-11468` and
`GHSA-h62f-wm92-2cmw`. The affected product is our Go package
`pkg:golang/github.com/rancher/support-bundle-kit@v0.0.38` in PURL format. It
relates to our [`rancher/support-bundle-kit`] repo at the release [`v0.0.38`].
It then identifies that the subcomponent, also in PURL format, in this case a Go
dependency,
`pkg:golang/github.com/docker/distribution@v0.0.0-20191216044856-a8371794149d`,
doesn't affect the repo due to the `status` `not_affected` and the
`justification` `vulnerable_code_not_present`. Thus, justifying that the CVE in
question is a false positive for the reported repo.
## Status of false-positives in Rancher and ECM
Each new release of Rancher ships hundreds of images that contain their own
CVEs, be in OS (operating system) level packages inside the container images or
in the Go dependencies used in the published binaries. Some of those CVEs come
from our own projects, in which case they are under our control, and other CVEs
are from upstream dependencies or mirrored images, which are outside of our
control. The same situation applies to Harvester, Longhorn, RKE2, Elemental etc.
Internally, we all know that some of those CVEs are false-positives, be it due
to the affected code not being in use (i.e., `vulnerable_code_not_present`) or
because a certain OS level package cannot be directly called by our code or by a
possible attacker (i.e., `vulnerable_code_cannot_be_controlled_by_adversary`).
Up until now, we didn't have an easy and standard way to communicate such false
positives to our customers. This is a very time consuming work, as customers
open dozens of SURE tickets that must be manually triaged by Support, Security
and the Engineering teams.
In order to facilitate the false positive communication, the Rancher Security
team is implementing OpenVEX reports for our released artifacts in Rancher,
Harvester and Longhorn (virtually any repo and image scanned by
`image-scanning`).
More information about how to VEX CVEs in ECM is available in the [supporting
VEX docs].
## OpenVEX implementation in Rancher and ECM
The initial implementation of OpenVEX is simple enough to act as a starting
point and is composed of:
- [`vex-scan.sh`] - contains the automated logic used to scan Go code with
`govulncheck` and identify the false-positives.
- The identified false-positives are stored in [`vex_automated_cves.csv`].
- [`vex-scan.yml`] - GHA logic that calls `vex-scan.sh` as a daily cron job.
- [`vex-cve.sh`] - contains the automation used to mark CVEs identified by
image-scanning as false-positives. These are the CVEs that are manually
triaged by the engineering teams.
- The identified false-positives are stored in [`vex_manual_cves.csv`].
- [`vex-cve.yml`] - GHA logic that calls `vex-cve.sh` when manual GH issues
are created after the manual triage to identify false-positives.
- [`vex-hub-reports.sh`] - contains the automation that generates the actual VEX
reports and publishes them to Rancher's [VEX Hub repo].
- [`vex-hub-reports.yml`] - GHA logic that calls `vex-hub-reports.sh` to
continuously regenerate and upload the VEX reports to the VEX Hub.
- [`vex_cves.csv`] - contains the final list of VEXed CVEs that is generated and
deduplicated from `vex_automated_cves.csv` and `vex_manual_cves.csv`.
- [`repos.txt`] - list of repos that are automatically scanned for
false-positives.
- [`govulncheck`] - upstream Go tooling that scans Go code bases in search for
known vulnerabilities. It has logic to automatically detect when a certain
vulnerable dependency is present, but the affected code isn't used, thus marking
the vulnerability as a false positive.
- [`vexctl`] - upstream OpenVEX tooling used to generate the OpenVEX documents
based on the output of `govulncheck`.
- [report template] - main template used for the OpenVEX documents.
The ECM [ADR 4: Publishing public VEX reports of CVEs in ECM/Rancher] documents
the decision about why we are using our own VEX Hub repo to publish our VEX
reports.
### Reports
The VEX reports are generated in [`vex-hub-reports.sh`] and
[`vex-hub-reports.yml`] and then sent to the [VEX Hub repo]. The VEX Hub follows
Trivy's [VEX repo specification].
## VEX scanning
The VEX scan runs daily as a GHA workflow in [`vex-scan.yml`]. The VEX reports
are generated in [`vex-hub-reports.yml`] and pushed to the VEX Hub.
`image-scanning` uses a [Trivy feature] that can read VEX reports to remove the
reported false-positives.
## VEXing CVEs
Any vulnerability, found by image-scanning or manually, can be added to a VEX
report, be it a false positive or a valid vulnerability that we want to document
as present or already fixed in our projects.
Below we present specific guidance about which vulnerabilities can be **added as
false positives**. For vulnerabilities that we want to **document as applicable
or fixed**, please consult with the Security team for further guidance.
### Valid false positives
Only valid false positives, that were automatically confirmed with `govulncheck`
(or similar tool) or that were manually triaged, must be added to our VEX reports.
If we are unsure about a vulnerability being a false positive or not, then it
**must not** be added to the report.
### Types of vulnerabilities
#### 1. What can be VEXed
Please read carefully the examples in order, as they provide important
information about what and how to VEX vulnerabilities.
##### 1.1. CVEs in our direct dependencies
Go related CVEs in dependencies that we import directly in our projects.
Example (click to expand)
Run the following command to VEX `CVE-2023-3676` in `k8s.io/kubernetes@v1.27.4`
(note the `pkg:golang/`, because it's a Go package) as a false positive
(`not_affected`) due to `vulnerable_code_not_present`. It's being VEXed against
the Rancher webhook binary `github.com/rancher/webhook` (it's the Go package
path).
Notes:
- If the vulnerability is specific to a certain tag, then it's important to add
the exact version (tag) of the binary (`product`) being VEXed. If the
vulnerability spans multiple versions, then the tag can be ignored.
- Example: suppose that the vulnerability affects only tag `v0.4.0`, then it
should be written as `pkg:golang/github.com/rancher/webhook@v0.4.0`.
- If the tag, either in the `product` or `subcomponent`, contains the `+`
signal, it must be replaced by `%2B`.
```text
vexctl create --product="pkg:golang/github.com/rancher/webhook" \
--subcomponents="pkg:golang/k8s.io/kubernetes@v1.27.4" \
--vuln="CVE-2023-3676" \
--status="not_affected" --justification="vulnerable_code_not_present"
```
Will result in the VEX document:
```json
{
"@context": "https://openvex.dev/ns/v0.2.0",
"@id": "https://openvex.dev/docs/public/vex-dc7b7d5c983058fbf700b70d1bb49f83d0084621e757981c6e0d2beb944641c9",
"author": "Unknown Author",
"timestamp": "2024-08-05T12:36:36.115868849Z",
"version": 1,
"statements": [
{
"vulnerability": {
"name": "CVE-2023-3676"
},
"timestamp": "2024-08-05T12:36:36.115869486Z",
"products": [
{
"@id": "pkg:golang/github.com/rancher/webhook",
"subcomponents": [
{
"@id": "pkg:golang/k8s.io/kubernetes@v1.27.4"
}
]
}
],
"status": "not_affected",
"justification": "vulnerable_code_not_present"
}
]
}
```
Save the VEX document above as `vex.json` and pass it to `trivy` when scanning
the `rancher/rancher-webhook:v0.4.0` image. It will correctly suppress the
specified CVE.
```text
trivy i --vex vex.json --show-suppressed rancher/rancher-webhook:v0.4.0
(... output ommited ...)
Suppressed Vulnerabilities (Total: 1)
┌───────────────────┬───────────────┬──────────┬──────────────┬─────────────────────────────┬──────────┐
│ Library │ Vulnerability │ Severity │ Status │ Statement │ Source │
├───────────────────┼───────────────┼──────────┼──────────────┼─────────────────────────────┼──────────┤
│ k8s.io/kubernetes │ CVE-2023-3676 │ HIGH │ not_affected │ vulnerable_code_not_present │ vex.json │
└───────────────────┴───────────────┴──────────┴──────────────┴─────────────────────────────┴──────────┘
```
##### 1.2. CVEs in base OS packages
CVEs in base OS packages can also be VEXed, for example, in `curl`, `libcurl`,
`libc`, `openssl`, `wget` and similar packages, when we determine that, even if
they have the vulnerability, they cannot be exploited unless the malicious user
is able to access directly the container. If this situation happens, then it
means that the attacker already have privileged access to the container.
Some examples that can help with such evaluation are:
- The OS level package is not called by our application, nor any of its
dependencies.
- The OS level package is not used at runtime (dynamically linked) by our
application, nor by any other binaries that we (or its dependencies) call.
Example (click to expand)
Run the following command to VEX advisory `SUSE-SU-2023:4557-1` in `vim`
(`pkg:rpm/sles/vim`), at version `9.0.1632-150500.20.3.1`, as a false positive
(`not_affected`) due to `vulnerable_code_cannot_be_controlled_by_adversary`.
It's being vexed against the Shell image
`pkg:oci/shell?repository_url=index.docker.io%2Francher%2Fshell`.
Notes:
- When marking an OS base package as false positive, the affected image is the
`product`.
- VEXing OS packages requires a different [PURL] (package URL) format. For our
BCI images, it's `pkg:rpm/sles/` followed by the name of the package and its
version.
- Targeting an image also requires specific syntax -
`pkg:oci/<@sha256%3A>?repository_url=%2F%2F`.
The image digest in `<@sha256%3A>` is optional.
- In case it's specified, the vulnerability will only be VEXed against the
specific version of the image (its digest). If it's not specified, then it
will be "relaxed" and applied to all versions. A decision about being
restrictive or not depends on each CVE being VEXed.
- Example of a restrictive VEX against version `v0.1.22` is:
`pkg:oci/shell@sha256%3A098c29e11ae9bd5ef8e58401a2892aae7491f71bc2e02ce211fe67d8544b35f9?index.docker.io%2Francher%2Fshell`.
- Note that the digest is related the multi-arch image, not against a
specific arch.
- The same decision can be applied to the version of the affected package.
```text
vexctl create \
--product="pkg:oci/shell?repository_url=index.docker.io%2Francher%2Fshell" \
--subcomponents="pkg:rpm/sles/vim@9.0.1632-150500.20.3.1" \
--vuln="SUSE-SU-2023:4557-1" \
--status="not_affected" \
--justification="vulnerable_code_cannot_be_controlled_by_adversary"
```
Will result in the VEX document:
```json
{
"@context": "https://openvex.dev/ns/v0.2.0",
"@id": "https://openvex.dev/docs/public/vex-65e47a16823e9c2a39c7594186c35395216fc66b29bb3b4da15aa2c91e156e6b",
"author": "Unknown Author",
"timestamp": "2024-08-05T14:09:14.56487705Z",
"version": 1,
"statements": [
{
"vulnerability": {
"name": "SUSE-SU-2023:4557-1"
},
"timestamp": "2024-08-05T14:09:14.564880132Z",
"products": [
{
"@id": "pkg:oci/shell?repository_url=index.docker.io%2Francher%2Fshell",
"subcomponents": [
{
"@id": "pkg:rpm/sles/vim@9.0.1632-150500.20.3.1"
}
]
}
],
"status": "not_affected",
"justification": "vulnerable_code_cannot_be_controlled_by_adversary"
}
]
}
```
Save the VEX document above as `vex.json` and pass it to `trivy` when scanning
the `rancher/shell:v0.1.22` image. It will correctly suppress the specified CVE.
```text
trivy i --vex vex.json --show-suppressed rancher/shell:v0.1.22
(... output ommited ...)
Suppressed Vulnerabilities (Total: 1)
┌─────────┬─────────────────────┬──────────┬──────────────┬───────────────────────────────────────────────────┬──────────┐
│ Library │ Vulnerability │ Severity │ Status │ Statement │ Source │
├─────────┼─────────────────────┼──────────┼──────────────┼───────────────────────────────────────────────────┼──────────┤
│ vim │ SUSE-SU-2023:4557-1 │ HIGH │ not_affected │ vulnerable_code_cannot_be_controlled_by_adversary │ vex.json │
└─────────┴─────────────────────┴──────────┴──────────────┴───────────────────────────────────────────────────┴──────────┘
(... output ommited ...)
```
#### 2. What might be VEXed
##### 2.2. Any CVE in versions of our images that were already released
Similar to the previous example, CVEs that are known false positives in images
that were already released can be VEXed. This is a good way to communicate to our
customers that a certain CVE isn't affecting the image in question. This can be
done both against Go CVEs as well as CVEs in base OS packages.
When VEXing CVEs in this situation:
- If it's a CVE in one of our Go packages, the VEX must be done against the
exact version of the package in question, as demonstrated in example 1.1.
- If the CVE is on an OS level package, then check example 1.2., and consider if
it must be restricted to the exact version of the image or if it must be
relaxed to span, virtually, all recently released versions of the image.
##### 2.2 CVEs in third-party binaries
CVEs that are known to be false positives in third-party binaries, after being
carefully verified, can be VEXed similarly to example 1.1. The `product` must
target the package path of the affected binary, and the `subcomponents` must
contain the package path of the affected dependency.
Example (click to expand)
Run the following command to VEX `CVE-2024-24786` in
`google.golang.org/protobuf@v1.30.0` as a false positive (`not_affected`) due to
`vulnerable_code_not_present`. It's being VEXed against the `kustomize` binary
`sigs.k8s.io/kustomize/kustomize/v5`.
Notes about the `product` when VEXing CVEs in third-party binaries:
- `go version -m ` might have to be used to find the exact
package path of the affected binary, because the path can be different than
`github.com`. This might also require that first the affected binary be
extracted from the image in question.
- Additionally, not all third-party binaries tend to include their version tag.
In case the binary doesn't contain it, but the VEX has it, then the
vulnerability will not be matched. If this happens, then the tag must be removed
from the `product`.
```text
vexctl create --product="pkg:golang/sigs.k8s.io/kustomize/kustomize/v5" \
--subcomponents="pkg:golang/google.golang.org/protobuf@v1.30.0" \
--vuln="CVE-2024-24786" \
--status="not_affected" --justification="vulnerable_code_not_present"
```
Will result in the VEX document:
```json
{
"@context": "https://openvex.dev/ns/v0.2.0",
"@id": "https://openvex.dev/docs/public/vex-dca79e9ddf84fe184223e0665794963b3e80f8ba3d2af8de01d2ef359d18bd0b",
"author": "Unknown Author",
"timestamp": "2024-08-05T17:48:46.466013374Z",
"version": 1,
"statements": [
{
"vulnerability": {
"name": "CVE-2024-24786"
},
"timestamp": "2024-08-05T17:48:46.466014253Z",
"products": [
{
"@id": "pkg:golang/sigs.k8s.io/kustomize/kustomize/v5",
"subcomponents": [
{
"@id": "pkg:golang/google.golang.org/protobuf@v1.30.0"
}
]
}
],
"status": "not_affected",
"justification": "vulnerable_code_not_present"
}
]
}
```
Save the VEX document above as `vex.json` and pass it to `trivy` when scanning
the `rancher/shell:v0.1.22` image. It will correctly suppress the specified CVE.
```text
trivy i --vex vex.json --show-suppressed rancher/shell:v0.1.22
(... output ommited ...)
Suppressed Vulnerabilities (Total: 1)
┌────────────────────────────┬────────────────┬──────────┬──────────────┬─────────────────────────────┬──────────┐
│ Library │ Vulnerability │ Severity │ Status │ Statement │ Source │
├────────────────────────────┼────────────────┼──────────┼──────────────┼─────────────────────────────┼──────────┤
│ google.golang.org/protobuf │ CVE-2024-24786 │ MEDIUM │ not_affected │ vulnerable_code_not_present │ vex.json │
└────────────────────────────┴────────────────┴──────────┴──────────────┴─────────────────────────────┴──────────┘
```
#### 3. What cannot be VEXed
##### 3.1. Go stdlib and compiler related CVE
CVEs in Go's standard library (`stdlib`) must not be VEXed, because there is a
high chance that they really affect the binary, and determining them to be a
false positive can be error prone. When such CVEs are identified, it's because
the Go compiler version used to build the binary is old, already EOLed or not
the latest available patch version. This always leads to a couple more and up to
a dozen additional CVEs. The recommended approach is to bump the Go compiler
version used to build the binary, and then rebuild the binary, as this will
certainly fix those CVEs and bring the compiler to a more updated version.
##### 3.2 Outdated base image
Outdated base images must not be VEXed. Please read the workflow about [how to
fix CVEs] to determine the right course of action for this situation.
### Considerations
#### How to flag an image-scanning issue as false positive
Open a "VEX CVE" [issue] in this repo and follow the instructions provided.
After the issue is created, an automation will run and will create a PR adding
the VEXed CVE to [`vex_manual_cves.csv`]. In case the data provided is in a bad
format, follow the instructions and correct it. The automation will re-run
every time that the issue is updated.
#### How to add a manual VEXed vulnerability
You can do it manually with `vexctl`, as explained in this document, to make
sure that the entry is a valid OpenVEX VEXed CVE. Then submit a PR adding the
VEXed CVE with the required data only in the CSV format of
[`vex_manual_cves.csv`].
#### Known limitations
1. `govulncheck`'s OpenVEX report currently doesn't provide information about
the scanned package, only about the affected subcomponent. This information is
needed in order for the generated OpenVEX reports to be useful, otherwise,
security scanners like Trivy cannot match the scanned dependency with the
reported false-positives. There is an upstream enhancement request in
[golang/go#68152] that is being worked by the Go team. For the moment, we use
our own [forked version] to generate the needed data.
#### Tooling
Consult the [FAQ] for more information about the tooling used to VEX the
vulnerabilities (`vexctl`) and to scan for false positives (`govulncheck`).
#### PURL syntax for base OS packages
When VEXing vulnerabilities in base OS packages, depending on the base
distribution of the container, the PURL syntax will vary.
| Distro | Syntax |
| -------- | ------------------------ |
| BCI/SLES | `pkg:rpm/sles/` |
| OpenSUSE | `pkg:rpm/opensuse.leap/` |
| Alpine | `pkg:apk/alpine/` |
| Debian | `pkg:deb/debian/` |
| CentOS | `pkg:rpm/centos/` |
| RedHat | `pkg:rpm/redhat/` |
| Ubuntu | `pkg:deb/ubuntu/` |
[reports]: ../vex/README.md#reports
[complete spec]: https://www.cisa.gov/sites/default/files/2023-04/minimum-requirements-for-vex-508c.pdf
[one-page summary]: https://www.ntia.gov/files/ntia/publications/vex_one-page_summary.pdf
[OpenVEX spec]: https://github.com/openvex/spec
[OpenVEX]: https://github.com/openvex
[`vexctl`]: https://github.com/openvex/vexctl
[PURL]: https://github.com/package-url/purl-spec
[`rancher/support-bundle-kit`]: https://github.com/rancher/support-bundle-kit
[`v0.0.38`]: https://github.com/rancher/support-bundle-kit/releases/tag/v0.0.38
[supporting VEX docs]: ../docs/vex.md
[`vex-scan.sh`]: vex-scan.sh
[`vex_automated_cves.csv`]: ../reports/vex/vex_automated_cves.csv
[`vex-scan.yml`]: ../.github/workflows/vex-scan.yml
[`vex-cve.sh`]: vex-cve.sh
[`vex_manual_cves.csv`]: ../reports/vex/vex_manual_cves.csv
[`vex-cve.yml`]: ../.github/workflows/vex-cve.yml
[`vex-hub-reports.sh`]: vex-hub-reports.sh
[`vex-hub-reports.yml`]: ../.github/workflows/vex-hub-reports.yml
[VEX Hub repo]: https://github.com/rancher/vexhub
[`vex_cves.csv`]: ../reports/vex/vex_cves.csv
[`govulncheck`]: https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck
[report template]: templates/main.openvex.json
[`repos.txt`]: repos.txt
[ADR 4: Publishing public VEX reports of CVEs in ECM/Rancher]: https://docs.google.com/document/d/1rLoptno80oOUs3jOVq3de6ayC1-j1nDzzu6uQP2Oj_s
[VEX repo specification]: https://github.com/aquasecurity/vex-repo-spec
[`image-scanning`]: ../.github/workflows/scan.yml
[Trivy feature]: https://aquasecurity.github.io/trivy/v0.55/docs/supply-chain/vex/repo/
[golang/go#68152]: https://github.com/golang/go/issues/68152
[forked version]: https://github.com/macedogm/vuln/
[how to fix CVEs]: how-to-fix-cve.md
[FAQ]: faq.md
[issue]: https://github.com/rancher/image-scanning/issues/new/choose
[`vex_manual_cves.csv`]: ../reports/vex/vex_manual_cves.csv