28 September 2011 by Published in: Code 10 comments

As part of a not-so-secret project, I’m writing a python script to make git commits on my behalf. I’d like these commits to be made “from” a different user.

GitHub in fact has a page on multiple SSH key usage, which makes the entire process look deceptively simple. Just create an alternate domain name and specify the IdentityFile for the domain name. Sounds simple, right? Ha.

So I follow the instructions, use an alternate subdomain… and the commits are still appearing from my normal git account. To eliminate the ~/.ssh/config file as a source of error, I pass -i on the command line and manually specify a key file… still no effect. What’s going on here?

sauron:semaps drew$ ssh -i ~/.ssh/glados_key -Tv git@github.com
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /Volumes/y/drew/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug1: Connecting to github.com [207.97.227.239] port 22.
debug1: Connection established.
debug1: identity file /Volumes/y/drew/.ssh/glados_key type 1
debug1: identity file /Volumes/y/drew/.ssh/glados_key-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.1p1 Debian-5github2
debug1: match: OpenSSH_5.1p1 Debian-5github2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'github.com' is known and matches the RSA host key.
debug1: Found key in /Volumes/y/drew/.ssh/known_hosts:3
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
<strong>debug1: Offering RSA public key: /Volumes/y/drew/Dropbox/keys/jeremyfletcher/jeremyfletcher.pem
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /Volumes/y/drew/.ssh/id_rsa</strong>
debug1: Remote: Forced command: gerve drewcrawford
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Pty allocation disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Remote: Forced command: gerve drewcrawford
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Pty allocation disabled.
debug1: Authentication succeeded (publickey).
Authenticated to github.com ([207.97.227.239]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Hi drewcrawford! You've successfully authenticated, but GitHub does not provide shell access.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2736, received 2920 bytes, in 0.1 seconds
Bytes per second: sent 27973.8, received 29855.0
debug1: Exit status 1
sauron:semaps drew$

Notice how no part of the key that is “offered” is in fact the key I specified on the command line. This is in contrast with the ssh manpage, which says:

-i identity_file
Selects a file from which the identity (private key) for RSA or DSA authentication is read. The default is ~/.ssh/identity for protocol version 1, and ~/.ssh/id_rsa and ~/.ssh/id_dsa for protocol version 2. Identity files may also be specified on a perhost basis in the configuration file. It is possible to have multiple -i options (and multiple identities specified in configuration files). ssh will also try to load certificate information from the filename obtained by appending -cert.pub to identity filenames.

My interpretation of “selects” is that the file specified should be the file attempted, (or if multiple files are provided, the files). Not just “append some extra files to the list of things to try”, which is in fact the implementation. This is really bad documentation for what should by all rights be the most stable piece of software anywhere.

So where is this behavior documented? It’s not. There are lots of authoritatively-sounding blog posts like “Force SSH to use the given private key“, another article with the exact same titleForce SSH Client To Use Given Private Key ( identity file )“, an article called “Manage multiple SSH private keys with IdentityFile“, and even Joel Spolsky has failed us. As best as I can tell, all the available resources give the misleading impression that -i or IdentityFile actually do something other than vaguely suggest to SSH where to find the keys as a last resort.

There is one single place in my hour-long Google search that touched on the problem:

…identities in an agent take precedence over identities used manually. If an identity in the agent can successfully authenticate, there’s no way to override the agent manually with the -i command-line option or the IdentityFile keyword. So in the earlier example, there is literally no way to use the identity id-normal. The obvious attempt:
$ ssh -i id-normal server.example.com
still authenticates with id-backup s because it is loaded first into the agent. Even nonloaded identities can’t override the agent’s selection. For example, if you load only one identity into the agent and try authenticating with the other:
$ ssh-add id-normal $ ssh -i id-backups server.example.com
your ssh connection authenticates with the loaded identity, in this case id-normal, regardless of the -i option. This undocumented behavior drove us insane until we figured out what was happening.

Unfortunately “The Secure Shell: The Definitive Guide” is itself having authentication woes (oh, the irony!) and only Google’s Cache is available. The “definitive guide” moniker notwithstanding, there is absolutely no solution offered for the issue, only a discussion of how unfortunate the problem is.

What you actually do, it turns out, is use the IdentitiesOnly “yes” directive in ~/.ssh/config. This means that none of the ssh-agent magical keys will get in the priority queue, and only things explicitly specified either in config or on the commandline are allowed.

Once you know what to Google, you find that actually a lot of places mention IdentitiesOnly, for instance SuperUser’s excellent discussion, the manpage for ssh_config (I would never have thought to look up a manpage for a config file…), and in fact there’s even a pull request on GitHub’s documentation to advise users to do this. But it does appear to be one of the Internet’s better-kept secrets.


Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.

Like this post? Contribute to the coffee fund so I can write more like it.

Comments

  1. ktu
    Tue 28th Feb 2012 at 8:28 pm

    you my friend, pointed me in the direction i needed to go. thank you
    for me this worked:

    ssh_config

    Host [personal]
    HostName github.com
    User git
    IdentityFile “[personal]\id_rsa”
    IdentitiesOnly yes

    Host [company]
    HostName github.com
    User git
    IdentityFile “[company]\id_rsa”
    IdentitiesOnly yes

    git clone [personal]:GitHubUser/repo folder

  2. Dan
    Sat 03rd Mar 2012 at 5:16 pm

    Thanks for writing this, it’s exactly what I was looking for.

    Also, just as an example for those who want to do this entirely from the command line:
    ssh -o “IdentitiesOnly yes” -i id_pub hostname

  3. Kevin Pulo
    Wed 23rd May 2012 at 11:32 pm

    This is fine, but in my case I want to run git on a remote host that I have forwarded my agent to… Now both of my github identities are in my agent, I have no private keys on the remote host (and don’t want to put them there), and absolutely no way to choose which agent id to use. For such a critically important piece of software, OpenSSH sure is infuriating sometimes.

  4. Chase Stubblefield
    Mon 18th Jun 2012 at 7:00 pm

    So glad to find this article, to confirm I wasn’t going crazy.

    I ran into the same exact issue, and found the “IdentitiesOnly yes” to be the solution.

    It appears the order in which it tries keys is:
    1. agent identities
    2. -i arguments
    3. config files
    4. default (~/.ssh{identity,id_dsa,id_rsa})

    When, ideally, I would want it to be:
    1. -i arguments
    2. config files
    3. agent identities
    4. default

  5. Jason Creel
    Sun 01st Jul 2012 at 10:47 pm

    Thanks for the helpful article. I’ve been battling this same issue for a few days now.

    Finally got things committed with the correct user by changing the email address on my global git-config file. It’s working for the moment, but will probably be borked when I get back to work. Wondering if I can override global config settings on a per repository basis….

  6. Sun 29th Jul 2012 at 7:30 pm

    Yes, the ssh_config(5) manual page is quite important. It is mentioned in the “SEE ALSO” section of the ssh(1) manual page.

    I have to deal with using multiple ssh keys for different reasons. (These are security-related; there are certain hosts I use which should have access only to keys specific to a client, not to my personal keys, thus I want to make sure I never forward authentication using my standard ssh-agent to them.) My solution is to use separate agents for this; I have a wrapper for the ssh/slogin command that, based on the host to which I’m connecting, will find the correct one of several existing ssh agents to use or start a new one if necessary.

    I suppose if there’s demand I could could clean this up and release it.

    cjs@cynic.net

  7. Vid
    Fri 30th Nov 2012 at 1:00 am

    Thanks Drew, helpful info.
    IdentitiesOnly was just what I needed. I can now authenticate just fine with ForwardAgent “yes”.

    On my work machine I have the added benefit of being able to pull from my git repo. All my ssh public keys are still forwarded but it seems as if the one specified in the IdentityFile line is prioritized. i.e. it shows up on top. Verified when I run ssh-add -l on the remote site.

    At home, however; on an older version of Mac OS and MacPorts OpenSSH while I can authenticate file I can’t pull from my repo on certain servers as the ssh keys are in their local default order. My current solution is to delete all my keys $ ssh-add -d ~/.ssh/* and then re-add them in the order I want to use them; specifying the password for each one. Not ideal… The key order also seems to reset itself on OS restart as well.

    @Curt, if you could release your solution that would be awesome. I’m looking at a similar sounding solution on StackExchange’s ServerFault: “Choose identity from ssh-agent by file name” – http://serverfault.com/questions/401737/choose-identity-from-ssh-agent-by-file-name
    There leoluk uses a python script parse out all the ssh keys and choose only the one specified in the identity file when forwarding an agent.

  8. John Weis
    Tue 15th Jan 2013 at 1:10 pm

    I tried the IdentitiesOnly trick above, but I was still seeing the problem.

    `
    $ ssh -vvv github.com pwd 2> >(grep -i offer)
    debug1: Offering RSA public key: /Users/jweis/.ssh/id_rsa
    `

    When I commented the line
    `
    # IdentityFile ~/.ssh/id_rsa
    `
    in my /etc/ssh_config, everything worked fine.

    (I’m on a mac.)

    Thanks for pointing me in the right direction!

  9. Ben
    Sun 17th Feb 2013 at 5:06 am

    That is so awesome.
    Had been starting at the -vvv output for so long..

    Weird design ssh folks..

    Thanks a mil Drew

  10. Mon 10th Jun 2013 at 12:36 am

    Thank you for this! The IdentitiesOnly line worked perfectly.

Comments are closed.

Powered by WordPress