Troubleshooting

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed

Get the status of a GitLab installation

sudo gitlab-ctl status
sudo gitlab-rake gitlab:check SANITIZE=true

Information on using gitlab-ctl to perform maintenance tasks.
Information on using gitlab-rake to check the configuration.

RPM 'package is already installed' error

If you are using RPM and you are upgrading from GitLab Community Edition to GitLab Enterprise Edition you may get an error like this:

package gitlab-7.5.2_omnibus.5.2.1.ci-1.el7.x86_64 (which is newer than gitlab-7.5.2_ee.omnibus.5.2.1.ci-1.el7.x86_64) is already installed

You can override this version check with the --oldpackage option:

sudo rpm -Uvh --oldpackage gitlab-7.5.2_ee.omnibus.5.2.1.ci-1.el7.x86_64.rpm

Package obsoleted by installed package

CE and EE packages are marked as obsoleting and replacing each other so that both aren't installed and running at the same time.

If you are using local RPM files to switch from CE to EE or vice versa, use rpm for installing the package rather than yum. If you try to use yum, then you may get an error like this:

Cannot install package gitlab-ee-11.8.3-ee.0.el6.x86_64. It is obsoleted by installed package gitlab-ce-11.8.3-ce.0.el6.x86_64

To avoid this issue, either:

Use the same instructions provided in the Upgrade using a manually-downloaded package section.
Temporarily disable this checking in yum by adding --setopt=obsoletes=0 to the options given to the command.

500 error when accessing Project > Settings > Repository

This error occurs when GitLab is converted from CE > EE > CE, and then back to EE. When viewing a project's repository settings, you can view this error in the logs:

Processing by Projects::Settings::RepositoryController#show as HTML
  Parameters: {"namespace_id"=>"<namespace_id>", "project_id"=>"<project_id>"}
Completed 500 Internal Server Error in 62ms (ActiveRecord: 4.7ms | Elasticsearch: 0.0ms | Allocations: 14583)

NoMethodError (undefined method `commit_message_negative_regex' for #<PushRule:0x00007fbddf4229b8>
Did you mean?  commit_message_regex_change):

This error is caused by an EE feature being added to a CE instance on the initial move to EE. After the instance is moved back to CE and then is upgraded to EE again, the push_rules table already exists in the database. Therefore, a migration is unable to add the commit_message_regex_change column.

This results in the backport migration of EE tables not working correctly. The backport migration assumes that certain tables in the database do not exist when running CE.

To fix this issue:

Start a database console:

sudo gitlab-rails dbconsole --database main

Manually add the missing commit_message_negative_regex column:

ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR;

# Exit psql
\q

Restart GitLab:
```
sudo gitlab-ctl restart
```

500 errors with `PG::UndefinedColumn: ERROR:..` message in logs

After upgrading, if you start getting 500 errors in the logs showings messages similar to PG::UndefinedColumn: ERROR:..., these errors could be cause by either:

Database migrations not being complete. Wait until migrations are completed.
Database migrations being complete, but GitLab needing to load the new schema. To load the new schema, restart GitLab.

Error: Failed to connect to the internal GitLab API

If you receive the error Failed to connect to the internal GitLab API on a separate GitLab Pages server, see the GitLab Pages administration troubleshooting

An error occurred during the signature verification

If you receive this error when running apt-get update:

An error occurred during the signature verification

Update the GPG key of the GitLab packages server with this command:

curl --silent "https://packages.gitlab.com/gpg.key" | apt-key add -
apt-get update

`Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] [..] Command timed out after 3600s`

If database schema and data changes (database migrations) must take more than one hour to run, upgrades fail with a timed out error:

FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] (gitlab::database_migrations line 51)
had an error: Mixlib::ShellOut::CommandTimeout: bash[migrate gitlab-rails database]
(/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16)
had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:

To fix this error:

Run the remaining database migrations:
```
sudo gitlab-rake db:migrate
```
This command may take a very long time to complete. Use screen or some other mechanism to ensure the program is not interrupted if your SSH session drops.
Complete the upgrade:
```
sudo gitlab-ctl reconfigure
```

Hot reload puma and sidekiq services:

sudo gitlab-ctl hup puma
sudo gitlab-ctl restart sidekiq

Missing asset files

Following an upgrade, GitLab might not be correctly serving up assets such as images, JavaScript, and style sheets. It might be generating 500 errors, or the web UI may be failing to render properly.

In a scaled out GitLab environment, if one web server behind the load balancer is demonstrating this issue, the problem occurs intermittently.

The Rake task to recompile the assets doesn't apply to an Omnibus installation which serves pre-compiled assets from /opt/gitlab/embedded/service/gitlab-rails/public/assets.

Potential causes and fixes:

Ensure no old processes are running.
Remove duplicate sprockets files
The installation is incomplete
NGINX Gzip support is disabled

Old processes

The most likely cause is that an old Puma process is running, instructing clients to request asset files from a previous release of GitLab. As the files no longer exist, HTTP 404 errors are returned.

A reboot is the best way to ensure these old Puma processes are no longer running.

Alternatively:

Stop Puma:
```
gitlab-ctl stop puma
```
Check for any remaining Puma processes, and kill them:
```
ps -ef | egrep 'puma[: ]'
kill <processid>
```
Verify with ps that the Puma processes have stopped running.
Start Puma
```
gitlab-ctl start puma
```

Duplicate sprockets files

The compiled asset files have unique filenames in each release. The sprockets files provide a mapping from the filenames in the application code to the unique filenames.

/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest*.json

Make sure there's only one sprockets file. Rails uses the first one.

A check for duplicate sprockets files runs during Omnibus GitLab upgrades:

GitLab discovered stale file(s) from the previous install that need to be cleaned up.
The following files need to be removed:

/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest-e16fdb7dd73cfdd64ed9c2cc0e35718a.json

Options for resolving this include:

If you have the output from the package upgrade, remove the specified files. Then restart Puma:
```
gitlab-ctl restart puma
```
If you don't have the message, perform a reinstall (see incomplete installation below for more details) to generate it again.
Remove all the sprockets files and then follow the instructions for an incomplete installation.

Incomplete installation

An incomplete installation could be the cause of this issue.

Verify the package to determine if this is the problem:

For Debian distributions:

apt-get install debsums
debsums -c gitlab-ee

For Red Hat/SUSE (RPM) distributions:
```
rpm -V gitlab-ee
```

To reinstall the package to fix an incomplete installation:

Check the installed version
- For Debian distributions:
```
apt --installed list gitlab-ee
```
- For Red Hat/SUSE (RPM) distributions:
```
rpm -qa gitlab-ee
```
Reinstall the package, specifying the installed version. For example 14.4.0 Enterprise Edition:
- For Debian distributions:
```
apt-get install --reinstall gitlab-ee=14.4.0-ee.0
```
- For Red Hat/SUSE (RPM) distributions:
```
yum reinstall gitlab-ee-14.4.0
```

NGINX Gzip support

Check whether nginx['gzip_enabled'] has been disabled:

grep gzip /etc/gitlab/gitlab.rb

This might prevent some assets from being served. Read more in one of the related issues.