This assumes you have installed Docker mailserver. 

The following environment variables can be used to configure Docker mailserver to use a public/private key pair for SSL/TLS encryption.

SSL_TYPE=manual
SSL_CERT_PATH=/tmp/dms/custom-certs/example.com.pem
SSL_KEY_PATH=/tmp/dms/custom-certs/example.com.key

In this example, you would store example.com.pem and example.com.key somewhere on your Docker host. Let's say example.com.pem and example.com.key are located at /usr/local/docker/ssl/mailserver.

/usr/local/docker/ssl/mailserver/example.com.pem
/usr/local/docker/ssl/mailserver/example.com.key

In this scenario, when creating the Docker mailserver contain, you would mount the directory to /tmp/dms/custom-certs.

/usr/local/docker/ssl/mailserver:/tmp/dms/custom-certs:ro

Here is an example docker-compose.yml file with the SSL variables and volume mount.

version: '3'
services:
  mailserver:
    image: docker.io/mailserver/docker-mailserver:latest
    container_name: mailserver
    hostname: mail.example.com
    domainname: example.com
    ports:
      - "25:25"
      - "143:143"
      - "587:587"
      - "993:993"
    volumes:
      - /usr/local/docker/mailserver/mail-data/:/var/mail/
      - /usr/local/docker/mailserver/mail-state/:/var/mail-state/
      - /usr/local/docker/mailserver/mail-logs/:/var/log/mail/
      - /usr/local/docker/mailserver/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - /usr/local/docker/ssl/mailserver/:/tmp/dms/custom-certs/:ro
    environment:
      - ENABLE_SPAMASSASSIN=1
      - SPAMASSASSIN_SPAM_TO_INBOX=1
      - ENABLE_CLAMAV=1
      - ENABLE_FAIL2BAN=1
      - ENABLE_POSTGREY=1
      - ENABLE_SASLAUTHD=0
      - ONE_DIR=1
      - DMS_DEBUG=0
      - PERMIT_DOCKER=host
      - SSL_TYPE=manual
      - SSL_CERT_PATH=/tmp/dms/custom-certs/example.com.pem
      - SSL_KEY_PATH=/tmp/dms/custom-certs/example.com.key
    cap_add:
      - NET_ADMIN
      - SYS_PTRACE
    restart: always

The postconf command with the -n (non-default parameters) flag can be used to display the SSL settings.

~]# sudo docker exec mailserver postconf -n
smtp_tls_CApath = /etc/ssl/certs
smtp_tls_loglevel = 1
smtp_tls_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtp_tls_security_level = may
smtpd_tls_CApath = /etc/ssl/certs
smtpd_tls_chain_files = /etc/dms/tls/key /etc/dms/tls/cert
smtpd_tls_dh1024_param_file = /etc/postfix/dhparams.pem
smtpd_tls_exclude_ciphers = aNULL, SEED, CAMELLIA, RSA+AES
smtpd_tls_loglevel = 1
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_security_level = may
tls_high_cipherlist = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
tls_preempt_cipherlist = yes
tls_ssl_options = NO_COMPRESSION, NO_RENEGOTIATION

The openssl s_client command can be used to confirm that the mailserver is able to present the certificate.

~]$ openssl s_client -connect post.freekb.net:143 -starttls imap
CONNECTED(00000003)
depth=0 C = US, ST = WI, L = Appleton, O = FreeKB, OU = IT, CN = mail.example.com, emailAddress = admin@example.com