• Man pages: openssl rsa, openssl rsautl and openssl enc
• Tutorial: Encryption with RSA Key Pairs
• Tutorial: How to encrypt a big file using OpenSSL and someone’s public key
• OpenSSL Command-Line HOWTO, in particular the section ‘How do I simply encrypt a file?’

If you encrypt/decrypt files or messages on more than a one-off occasion, you should really use GnuPGP as that is a much better suited tool for this kind of operations. But if you already have someone’s public SSH key, it can be convenient to use it, and it is safe.

The below notes assumes you have a (potentially big) file you want to send encrypted to a collaborator, typically on a remote server where your SSH public key is allowed (i.e. your id_rsa.pub key is added to the remote ~/.ssh/authorized_keys file). We also assume that you own a copy of the SSH public key of your collaborator (denoted by id_dst_rsa.pub) in the sequel.

Note: as a reminder, you can generate a strong RSA key pair (4096 bits) using

  ssh-keygen -t rsa -b 4096 -a 100 [-f <name>]

This will produce the key files <name> and <name>.pub, where <name> is ~/.ssh/id_rsa by default. Example:

   # Eventually ask your collaborator to generate a dedicated SSH key pair -- classical RSA keys (4096 bits)
   $> ssh-keygen -t rsa -b 4096 -a 100 -f ~/.ssh/id_$(whoami)_rsa
   Generating public/private rsa key pair.
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in /home/vagrant/.ssh/id_vagrant_rsa.
   Your public key has been saved in /home/vagrant/.ssh/id_vagrant_rsa.pub.
   The key fingerprint is:
   SHA256:tf04p/tYXfjVdbqbYgWVHHfuawKlTi8q6zAyqKWel8I vagrant@adminfront.vagrant.dev
   The key randomart image is:
   +---[RSA 4096]----+
   |             ..oo|
   |              +o.|
   |          .  o  +|
   |         . o+  ++|
   |        S .+..o =|
   |  .       o oo.++|
   |...o.o     oo+=oo|
   |.E.oo o.  . +*oo |
   |+oo   .oo. .+++  |
   +----[SHA256]-----+

/!\ IMPORTANT the below instructions are NOT compliant with the new OpenSSH format which is used for storing encrypted (or unencrypted) RSA, EcDSA and Ed25519 keys (among others) when you use the -o option of ssh-keygen. You can recognize these keys by the fact that the private SSH key ~/.ssh/id_rsa starts with -----BEGIN OPENSSH PRIVATE KEY-----

Encrypt a file using a public SSH key

(eventually) SSH RSA public key conversion to PEM PKCS8

OpenSSL encryption/decryption operations performed using the RSA algorithm relies on keys following the PEM format ¹ (ideally in the PKCS#8 format). It is possible to convert OpenSSH public keys (private ones are already compliant) to the PEM PKCS8 format (a more secure format). For that one can either use the ssh-keygen or the openssl commands, the first one being recommended.

  # Convert the public key of your collaborator to the PEM PKCS8 format (a more secure format)
  $> ssh-keygen -f id_dst_rsa.pub -e -m pkcs8 > id_dst_rsa.pkcs8.pub
  # OR use OpenSSL for that...
  $> openssl rsa -in id_dst_rsa -pubout -outform PKCS8 > id_dst_rsa.pkcs8.pub

Note that you don’t actually need to save the PKCS#8 version of his public key file – the below command will make this conversion on demand.

Generate a 256 bit (32 byte) random symmetric key

There is a limit to the maximum length of a message i.e. size of a file that can be encrypted using asymmetric RSA public key encryption keys (which is what SSH keys are). For this reason, you should better rely on a 256 bit key to use for symmetric AES encryption and then encrypt/decrypt that symmetric AES key with the asymmetric RSA keys This is how encrypted connections usually work, by the way.

Generate the unique symmetric key key.bin of 32 bytes (i.e. 256 bit) as follows:

  $> openssl rand -base64 32 -out key.bin

You should only use this key once. If you send something else to the recipient at another time, you should regenerate another key.

Encrypt the (potentially big) file with the symmetric key

  $> openssl enc -aes-256-cbc -salt -in bigdata.dat -out bigdata.dat.enc  -pass file:./key.bin

Note: for your tests, you can quickly generate random files of 1 GiB size as follows:

  # Random generation of a 1GiB file
  $> dd if=/dev/urandom of=bigfile_1GiB.dat  bs=64M count=16  iflag=fullblock
  # Random generation of a 1GiB file
  $> dd if=/dev/urandom of=bigfile_10GiB.dat bs=64M count=160 iflag=fullblock

An indicated encryption time taken for the above random file is proposed in the below table, using

     openssl enc -aes-256-cbc -salt -in bigfile_<N>GiB.dat -out bigfile_<N>GiB.dat.enc  -pass file:./key.bin

File	size	Encryption time
bigfile_1GiB.dat	1 GiB	0m5.395s
bigfile_10GiB.dat	10 GiB	2m50.214s

Encrypt the symmetric key, using your collaborator public SSH key in PKCS8 format:

  $> openssl rsautl -encrypt -pubin -inkey <(ssh-keygen -e -m PKCS8 -f id_dst_rsa.pub) -in key.bin -out key.bin.enc
  # OR, if you have a copy of the PKCS#8 version of his public key
  $> openssl rsautl -encrypt -pubin -inkey  id_dst_rsa.pkcs8.pub -in key.bin -out key.bin.enc

Delete the unencrypted symmetric key as you don’t need it any more (and you should not use it anymore)

  $> rm key.bin

Now you can transfer the *.enc files i.e. send the (potentially big) encrypted file <file>.enc and the encrypted symmetric key (i.e. key.bin.enc ) to the recipient _i.e. your collaborator. Note that you are encouraged to send the encrypted file and the encrypted key separately. Although it’s not absolutely necessary, it’s good practice to separate the two. If you’re allowed to, transfer them by SSH to an agreed remote server. It is even safe to upload the files to a public file sharing service and tell the recipient to download them from there.

Decrypt a file encrypted with a public SSH key

First decrypt the symmetric key using the SSH private counterpart:

 # Decrypt the key -- /!\ ADAPT the path to the private SSH key
 $> openssl rsautl -decrypt -inkey ~/.ssh/id_rsa -in key.bin.enc -out key.bin
 Enter pass phrase for ~/.ssh/id_rsa:

Now the (potentially big) file can be decrypted, using the symmetric key:

 $> openssl enc -d -aes-256-cbc -in bigdata.dat.enc -out bigdata.dat -pass file:./key.bin

Misc

For a ‘quick and dirty’ encryption/decryption of small files:

 # Encrypt
 $>  openssl rsautl -encrypt -inkey <(ssh-keygen -e -m PKCS8 -f ~/.ssh/id_rsa.pub) -pubin -in <cleartext_file>.dat -out <encrypted_file>.dat.enc
 # Decrypt
 $> openssl rsautl -decrypt -inkey ~/.ssh/id_rsa -in <encrypted_file>.dat.enc -out <cleartext_file>.dat