Ubuntu Gnome Agent remembers SSH private key passwords until you log out. If someone knows an Ubuntu user password, they also have access to any SSH private keys loaded since last logon.
This also fixes error upon trying to use
sign_and_send_pubkey: signing failed: agent refused operation
Permanently disable Gnome Keyring SSH Agent by including this line in
Reboot and test that private key passwords aren’t being remembered.
Alternative method to disable Gnome Keyring SSH Agent: Edit
/etc/xdg/autostart/gnome-keyring-ssh.desktop to include the line:
Under Startup Applications → SSH Key Agent (uncheck). Reboot and test that private key passwords aren’t being remembered.
Related: configure SSH agent to remember SSH keys