# How to fix CVEs

The diagram below provides a suggested workflow for how to fix CVEs in:

- Base images - outdated and OS level packages;
- Application dependencies - directly or indirectly imported in our projects; or
- Third-party binaries - that are pulled in our images.

## Workflow

<!-- Note
When updating the CVE workflow, please remember to generate an image of the
diagram and to ask the Support team (Shaun Gardiner or Brian Almeida) to update
the image in the public KB in https://www.suse.com/support/kb/doc/?id=000021574

The image can be generated with:

1. Copy only the mermaid code (everything between ```mermaid and ``` tags) to a
new file called dia.md (for example).

2. Run:

podman run --userns keep-id --user ${UID} --rm -v $(pwd):/data:z ghcr.io/mermaid-js/mermaid-cli/mermaid-cli -i dia.md -o dia.png -b transparent -s 4

Found 1 mermaid charts in Markdown input
 ✅ ./dia-1.png

3. Send the generated png file to the Support team.
-->

```mermaid
---
title: How to fix a CVE
---
graph TB;
    A1((Triage the CVE)) --> EOL1{Is the issue<br>for an image<br>that is no longer<br>used in Rancher?};

    EOL1 --> |Yes| EOL2[Add the image<br>to the EOL list<br>and CVEs will be<br>ignored];

    subgraph "EOL image"
    EOL2
    end
    EOL2 ----> WAIT;

    EOL1 --> |No| A2{Which type<br>is the<br>CVE};

    A2 --> B[Outdated base image];
    A2 --> C[OS level package];
    A2 --> D[Go dependency<br>or Go stdlib<br>and compiler version];

    subgraph "Image layer CVE"
    B --> B1{Is it Rancher<br>made or<br>mirrored<br>image?};
    B1 --> |Mirrored| B2{Is there a new<br>updated tag<br>in upstream?};
    B1 --> |Rancher<br>made| B3[Bump the base image<br>to a new and supported<br>version];
    B2 --> |No| B4(((Evaluate moving<br>to a different<br>image)));
    B2 --> |Yes| B5[Mirror the new<br>tag and bump it];
    B3 --> B6[Tag a new version<br>of the image];
    C[OS level package];
    C --> B1;
    end

    subgraph "Dependency and binary CVE"
    D --> D1{Is it a<br>false-<br>positive?};
    D1 --> |No| D2{Is there a<br>suggested fixed<br>package version?};
    D2 --> |Yes| D3[Update to the<br>fixed version];
    D2 --> |No| D4[Check with<br>3rd-party or<br>upstream if<br>they can<br>fix it];
    D3 --> B6;
    D4 --> CO(((CVE is kept open<br>until it can<br>be fixed)));
    end

    subgraph "Update propagation"
    B5 --> UP[Update the new tag<br>across the places<br>where it's used];
    B6 --> UP;
    end

    subgraph VEX
    D1--> |Yes| V[VEX it as<br>false-positive];
    end

    subgraph "Fix validation"
    UP --> WAIT[Wait for the<br>next daily scan];
    WAIT --> E{Was the specific<br>CVE fixed or<br>the issue was<br>closed and a<br>new one created<br>without the CVE?};
    E --> |Yes| OK(((Done)));
    E --> |No| SEC(((Consult with<br>the Sec team)));
    end

    V --> WAIT;
```

## How to check if a CVE is fixed

Before generating a new tag of an image, it's recommended to build it locally to
confirm if the CVE is fixed or not. The local build should contain the bump of
the image, the base OS packages or the third-party binaries; or the re-build of
the binary with the updated Go dependencies. For mirrored images, scan the
upstream tag to confirm the fixes before mirroring the new tag.

Install Trivy, as explained in the [FAQ][trivy-install], then run:

```text
trivy image --scanners vuln <image-name>`
```

`<image-name>` is the name of the locally built image or the upstream image.

Verify if the CVEs that you are fixing were removed, i.e., they are no longer
present in the scan's output. If the CVEs are fixed, then proceed with the
workflow presented above. If the CVEs are still present, consult with the
Security team to verify what is happening.

## VEX

For information about VEX and how to mark false positives, please consult the
specific [documentation][vex].

## Considerations

### How should the fixes be prioritized

The fixes should be prioritized in the following order:

1. All CVEs in tier 1 images.
2. CVEs in the most recent release in its head branch, then the older supported
releases in their head branches in order (EOLed releases must be ignored). For
example, the current latest minor version is `v2.9.x`, so these should be
prioritized in `v2.9-head` and `main` (soon to be `v2.10-head`), then
`v2.8-head` and finally `v2.7-head`.
3. The remaining images, starting with critical CVEs and so on.
   1. This includes Rancher made images and mirrored images.

### How frequently should image-scanning issues be triaged

The frequency for triaging image-scanning issues depends on how each team organizes
its priorities with their Project Manager. The Security team recommends that
issues be triaged at least once per week to:

1. Avoid letting the backlog increase and the issues to accumulate.
2. The sooner the fixes happen and new tags are generated, the sooner QA can
find possible issues.
   1. Bumping at the last minute and before code freeze is about to start might
increase the chances of regressions.
3. Seeing the issue count dropping in the dashboard helps to increase the morale
and the sense of accomplishment.

### All the CVEs were fixed last night, but today there are more

Short answer: it's part of the game.

Long answer: there is no way that we can predict when new CVEs will appear and
neither stop them from appearing at all, specially when the entire open source
community relies on code and images made by third parties. So, it's expect, and
quite common, that every week new CVEs will appear.

For example, our critical CVE count in tier 1 images was below 100, then the
critical (score 10 of 10) [CVE-2024-41110] appeared on 2024-Jul-30. From one
night and to the next morning, the count raised again to above 100.

As explained, there is no way that we could have anticipated this, and is purely
part of the game, but with `govulncheck` we were able to identify a dozen of
instances where the CVE is a known false positive in images made by us. Then, by
generating the right VEX reports, we were able to silence those false positives.

### How to navigate the image-scanning issues

The [image-scanning issues] are assigned specific labels that help to search and
track them. The main labels are:

- `team/<name>` - used to track the ownership of the image, where `<name>` is a
  team name listed in the [config file]. Further rules about the automatic team
ownership are listed in [`configs/images-teams.txt`] and
[`configs/sources-teams.txt`].
  - Images without ownership are assigned the label `team/unassigned`.
- `cve/critical` - to track images that have a critical CVE.
- `cve/image/mirrored` - to track mirrored images, which normally contain the
  `mirrored-` string.
- `cve/image/rancher-built` - to track images made by us.
- `cve/release/rancher/<version>` - to track images from a specific Rancher
  release, where `<version>` can be any version that is scanned.
- `cve/release/harvester/<version>` - same as before, but for Harvester releases.
- `cve/release/longhorn/<version>` - same as before, but for Longhorn releases.

If an image has the wrong label, please open an [issue] in this repo.

Additionally, the search field in the issues tab can also be used to search for
specific images or CVE IDs, just do the search and surround the expression with
double quotes. For example, `"rancher/shell"` or `"CVE-2024-41110"`, to search
for all the issues related to the Shell image or the CVE-2024-41110,
respectively. The search filters can be combined with the labels too.

### After fixing all the CVEs in an issue, should I close the issue

No, image-scanning issues must not be closed manually. If you fixed all the CVEs
on an image, then generate a new tag for that image, make sure to update it
across all the places where it's used, and then wait for the next daily scan to
confirm if the CVEs were really fixed or not. If in question, please reach out
to the Security team.

### Further recommendations 

To try to minimize the "surprises with CVEs", the Security team recommends to:

1. Keep a good dependency hygiene and maintain dependencies and images as close
as possible to being up to date.
   - Enable the dependency automation in your team's repo.
1. When possible, minimize the need of introducing third party dependencies,
specially when the problem being solved can be done with Go's standard library
and builtin functions.
1. Prefer to use known and maintained dependencies. Avoid the use EOLed
dependencies and those that are maintained by only one individual and that have
a low frequency of updates.
1. Prefer to use slim base images, because less packages installed means less
possible CVEs to appear.
   1. Packages like `curl` and `openssl`, even `git`, are a known source of high
severity CVEs from time to time, so if your image doesn't need them, then remove
them.
1. Consult [this guide][dependency-cves-guide] for further information about
enabling the dependency automation tooling, using smaller base images etc.


<!-- Links -->
[trivy-install]: faq.md#tooling
[vex]: vex.md
[CVE-2024-41110]: https://github.com/moby/moby/security/advisories/GHSA-v23v-6jw2-98fq
[dependency-cves-guide]: https://github.com/rancher/security-team/blob/main/docs/dependencies-cves.md
[image-scanning issues]: https://github.com/rancherlabs/image-scanning/issues
[config file]: ../config.yml
[`configs/images-teams.txt`]: ../configs/images-teams.txt
[`configs/sources-teams.txt`]: ../configs/sources-teams.txt
[issue]: https://github.com/rancher/image-scanning/issues