After restoring GitLab backup, new SSH public keys randomly supersede other users' existing keys

This occurred with a new (not upgraded) installation of GitLab 8.6.4.

I installed GitLab and my team evaluated it. Of course I and others entered our SSH public keys.

As part of our evaluation I made a GitLab backup and restored it.

After I restored the backup, new user Shung Wang entered his SSH public key.

Now, whenever I try to access the git repositories via SSH, the server thinks I am Shung Wang. For example, when I tested my SSH connection from my Ubuntu 14.04 laptop, I got this:

ssh -T git@gitserver

Welcome to GitLab, Shung Wang!


As a second test, I tried to clone a private repository to which Shung did not have access:

git clone git@gitserver:sw/devops.git
Cloning into 'devops'...
GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights

and the repository exists.


Then I made Shung a member of the devops project, and git clone succeeded. I really am accessing GitLab repositories as Shung Wang.

Obviously this is a most unsatisfactory security situation. How can I access GitLab repositories as myself other than Shung Wang?

Answer

Explanation

GitLab maintains file ~git/.ssh/authorized_keys, in which it gives each SSH public key names key-1, key-2, and so on.

Upon backup restore, the name gets reset to key-1, so subsequent keys have duplicate names. Here is my ~git/.ssh/authorized_keys file showing my key and Shung Wang's key both named key-1:

# Managed by gitlab-shell
command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAA...2SkSQ== jm...@wavecomp.com
###################################################################################################################################################################################
#####################################################################################################################################################################################
command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAA...nKQ== wang....@wavecomp.com
...

Apparently the last occurrence of key-1 is used for both me and Shung.

I reported this GitLab bug in GitLab issue 1263.

Workarounds

Until the bug is fixed, users can try these workarounds.

Workaround 1

Use sudo gitlab-rake gitlab:shell:setup to rebuild the authorized_keys file

Workaround 2

I recommend trying workaround 1 first.

The following is what I actually did myself before discovering workaround 1.

1. After you restore a backup, log in as some existing user and register a new SSH public key


2. Search file ~git/.ssh/authorized_keys for key-1. If you find two, you have the problem described above.


3. Those lines of ######... are not decoration. They are keys that were deleted by users. When a key is deleted, GitLab replaces every character in it with a #, presumably so the remaining keys do not move around in the file. Replace all characters in all SSH keys created before the backup restore with the # character in a fashion similar to what you see in the ~git/.ssh/authorized_keys shown above.


4. Tell all your users they must re-enter their SSH public keys. If they complain, remind them to be thankful that the rest of the backup worked.

Rather than all that tedious editing in step 3, I suspect that you can simply move file ~git/.ssh/authorized_keys aside and replace it with an empty file, and then tell everyone to re-enter their SSH public keys. However I did not try this myself. If this works for you, please tell us in a comment.