Ubuntu Gnome Agent remembers SSH private key passwords until you log out. If someone knows an Ubuntu user password, they also have access to any SSH private keys loaded since last logon.

This also fixes error upon trying to use ssh or sshfs:

sign_and_send_pubkey: signing failed: agent refused operation

Permanently disable Gnome Keyring SSH Agent by including this line in /etc/xdg/autostart/gnome-keyring-ssh.desktop

X-GNOME-Autostart-enabled=false

Reboot and test that private key passwords aren’t being remembered.

Alternative method to disable Gnome Keyring SSH Agent: Edit /etc/xdg/autostart/gnome-keyring-ssh.desktop to include the line:

NoDisplay=false

Under Startup Applications → SSH Key Agent (uncheck). Reboot and test that private key passwords aren’t being remembered.

Related: configure SSH agent to remember SSH keys