Troubleshooting application security
DETAILS: Tier: Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated
When working with application security features, you might encounter the following issues.
Logging level
The verbosity of logs output by GitLab analyzers is determined by the SECURE_LOG_LEVEL
environment
variable. Messages of this logging level or higher are output.
From highest to lowest severity, the logging levels are:
fatal
error
warn
-
info
(default) debug
Debug-level logging
WARNING: Debug logging can be a serious security risk. The output may contain the content of environment variables and other secrets available to the job. The output is uploaded to the GitLab server and is visible in job logs.
To enable debug-level logging, add the following to your .gitlab-ci.yml
file:
variables:
SECURE_LOG_LEVEL: "debug"
This indicates to all GitLab analyzers that they are to output all messages. For more details, see logging level.
Secure job failing with exit code 1
If a Secure job is failing and it's unclear why:
- Enable debug-level logging.
- Run the job.
- Examine the job's output.
- Remove the
debug
log level to return to the defaultinfo
value.
Outdated security reports
When a security report generated for a merge request becomes outdated, the merge request shows a warning message in the security widget and prompts you to take an appropriate action.
This can happen in two scenarios:
Source branch is behind the target branch
A security report can be out of date when the most recent common ancestor commit between the target branch and the source branch is not the most recent commit on the target branch.
To fix this issue, rebase or merge to incorporate the changes from the target branch.
Target branch security report is out of date
This can happen for many reasons, including failed jobs or new advisories. When the merge request shows that a security report is out of date, you must run a new pipeline on the target branch. Select new pipeline to run a new pipeline.
… report.json: no matching files
Getting warning messages WARNING: Debug logging can be a serious security risk. The output may contain the content of environment variables and other secrets available to the job. The output is uploaded to the GitLab server and visible in job logs.
This message is often followed by the error No files to upload
,
and preceded by other errors or warnings that indicate why the JSON report wasn't generated. Check
the entire job log for such messages. If you don't find these messages, retry the failed job after
setting SECURE_LOG_LEVEL: "debug"
as a custom CI/CD variable.
This provides extra information to investigate further.
sast job: config key may not be used with 'rules': only/except
Getting error message When including a .gitlab-ci.yml
template
like SAST.gitlab-ci.yml
,
the following error may occur, depending on your GitLab CI/CD configuration:
Unable to create pipeline
jobs:sast config key may not be used with `rules`: only/except
This error appears when the included job's rules
configuration has been overridden
with the deprecated only
or except
syntax.
To fix this issue, you must either:
-
Transition your
only/except
syntax torules
. - (Temporarily) Pin your templates to the deprecated versions
For more information, see Overriding SAST jobs.
only/except
syntax to rules
Transitioning your When overriding the template to control job execution, previous instances of
only
or except
are no longer compatible
and must be transitioned to the rules
syntax.
If your override is aimed at limiting jobs to only run on main
, the previous syntax
would look similar to:
include:
- template: Jobs/SAST.gitlab-ci.yml
# Ensure that the scanning is only executed on main or merge requests
spotbugs-sast:
only:
refs:
- main
- merge_requests
To transition the above configuration to the new rules
syntax, the override
would be written as follows:
include:
- template: Jobs/SAST.gitlab-ci.yml
# Ensure that the scanning is only executed on main or merge requests
spotbugs-sast:
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_MERGE_REQUEST_ID
If your override is aimed at limiting jobs to only run on branches, not tags, it would look similar to:
include:
- template: Jobs/SAST.gitlab-ci.yml
# Ensure that the scanning is not executed on tags
spotbugs-sast:
except:
- tags
To transition to the new rules
syntax, the override would be rewritten as:
include:
- template: Jobs/SAST.gitlab-ci.yml
# Ensure that the scanning is not executed on tags
spotbugs-sast:
rules:
- if: $CI_COMMIT_TAG == null
For more information, see rules
.
Pin your templates to the deprecated versions
To ensure the latest support, we strongly recommend that you migrate to rules
.
If you're unable to immediately update your CI configuration, there are several workarounds that involve pinning to the previous template versions, for example:
include:
remote: 'https://gitlab.com/gitlab-org/gitlab/-/raw/12-10-stable-ee/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml'
Additionally, we provide a dedicated project containing the versioned legacy templates. This can be used for offline setups or for anyone wishing to use Auto DevOps.
Instructions are available in the legacy template project.
Vulnerabilities are found, but the job succeeds. How can you have a pipeline fail instead?
In these circumstances, that the job succeeds is the default behavior. The job's status indicates success or failure of the analyzer itself. Analyzer results are displayed in the job logs, merge request widget, or security dashboard.
is used for configuration only, and its script should not be executed
Error: job Changes made in GitLab 13.4
to the Security/Dependency-Scanning.gitlab-ci.yml
and Security/SAST.gitlab-ci.yml
templates mean that if you enable the sast
or dependency_scanning
jobs by setting the rules
attribute,
they fail with the error (job) is used for configuration only, and its script should not be executed
.
The sast
or dependency_scanning
stanzas can be used to make changes to all SAST or Dependency Scanning,
such as changing variables
or the stage
, but they cannot be used to define shared rules
.
There is an issue open to improve extendability. You can upvote the issue to help with prioritization, and contributions are welcomed.
Empty Vulnerability Report, Dependency List pages
If the pipeline has manual steps with a job that has the allow_failure: false
option, and this job is not finished,
GitLab can't populate listed pages with the data from security reports.
In this case, the Vulnerability Report and the Dependency List
pages are empty.
These security pages can be populated by running the jobs from the manual step of the pipeline.
There is an issue open to handle this scenario. You can upvote the issue to help with prioritization, and contributions are welcomed.