How Do Clients Securely Connect to SSL & HTTPS Servers?

This question arose from Steven Chu on my previous post about MySQL SSL Connections without Client Certificates. How is the client able to securely connect to a server using SSL if it doesn’t already know or trust the Server Certificate?

It is important to understand that there are a few different, interrelated topics here. All of these involve SSL and certificates, but in differing ways, so they are often conflated. Secure communication over SSH shares the same concepts, but has different mechanisms.

  1. Encryption of the traffic between client and server.
  2. Verification that the server is who the client believes it to be.
  3. Authentication of the client to the server.

For SSL and HTTPS communication, the first two concepts are accomplished together because there is no point in communicating securely with a remote party if you can’t verify that the remote party is who they claim to be and that there isn’t a “Man in the Middle” able to intercept the secure traffic.

You actually communicate securely with unknown servers all of the time. When you loaded this web page, your browser didn’t know anything about www.brandonchecketts.com beforehand. Same thing when you load your bank’s website. You never configured your browser specifically to trust their website. So how is it able to verify that it is actually your bank, and not an attacker who is impersonating your bank?

Certificate Authorities

Anybody can create an SSL Certificate with any name on it. In my SSL Certificate Notes post, you can find instructions for creating an SSL certificate. Note that you simply type in the name for the certificate. So you could attempt to create a certificate for any host you care to try. However, an essential part of the process is the Signing of the Certificate. You can self-sign a certificate with any name on it. But if you want your certificate to be recognized and trusted by anybody else, you need to have a recognized Certificate Authority (CA) sign it. If you were to try to create a certificate for www.google.com, no Certificate Authority is going to sign that since you can’t validate that you are authorized to create certificatesnobody else in the world is going to trust it.

When a Certificate Authority signs a certificate, it is their job to verify that the certificate owner is who they claim to be in some way. On the public internet, that is largely done through DNS or Email validation. For example, on this site, I use a certificate issued by Amazon Web Services. In order obtain that certificate, I had to verify that I own the domain. Since the domain is also hosted at AWS, it is quite easy for me to create the DNS records for verification, and AWS can validate it within seconds. I couldn’t, for example, validate a certificate that was for ‘www.google.com’. I’d be unable to validate it with any certificate authority since I can’t make the required DNS entries or receive emails to the required email addresses for google.com.

Extended Validation certificates, offered by some Certificate Authorities, and recognized by some web browsers with a different color banner, often have additional verification steps other than just DNS or Email.

Intermediate Certificates and Multiple layers of Certificate Authoritiees

When a certificate is signed by a recognized Certificate Authority, your client can trust it, because it trusts the Certificate Authority. On the public Internet, most of the time there are multiple layers of Certificate Authorities.

On OSX, you can find the list of root certificates it trusts in the “Keychain Access” system app, in the “System Roots” section. On an Ubuntu or Debian Linux system, the trusted certificates are files that exist or are symlinked in `/etc/ssl/certs`. These systems have dozens to hundreds of certificates that they trust. Look closely and you’ll see that most of them expire 10+ years into the future. These “Root” certificates are highly protected and usually don’t directly sign certificates. The Certificate Authority will often delegate access to intermediate authorities with their own keys that can further sign certificates.

In Chrome, you can click the lock icon next to the URL, and find details about the certificate, including the intermediate certificates. As of the writing of this post, you can see that the SSL Certificate issued to brandonchecketts.com is issued by “Amazon”, which is trusted by the root certificate named “Amazon Root CA 1”. I can find that root certificate in the list of certificates trusted by my OSX system.

Client Authentication

I mentioned the third step above about the client authenticating to the server. In many cases, like this website, there is no need for the client to authenticate to the server since the content is public and intended to be viewed anonymously. If authentication is required, for instance to create a new post, then I simply log in with a username and password entered of the HTTPS connection. Same as you do every day.

Many well-meaning articles about generating SSL Certificates for services other than HTTPS often mention creating an SSL client certificate. The Client Certificate is then provided to the server so that the server can validate the client is who they claim to be. The Client Certificate is simply an alternate (often thought of as “more secure”) method of authenticating than a username and password, or sometimes even in addition to a username and password. In practice, I’ve seen that usernames and passwords transmitted over an encrypted connection are very common, well understood, and just as secure as using an SSL Client Certificate.

Using LastPass to Save Passwords and Log In to Multiple AWS Accounts With Two-Factor Authentication

I have multiple businesses, so I log into AWS multiple times per day.

That is a little tricky to do using LastPass since AWS has some hidden form fields that must be filled in
when using two-factor authentication through Google Authenticator.

In order to make it work correctly, I’ve had to modify the extra details in LastPass to add some extra hidden fields. If you set these up in your LastPass credentials for AWS, you should be able to log in with just a couple clicks, like usual, instead of having to type in some of those fields every time or having them overwritten.

Also, make sure to check the “Disable Autofill” checkbox an all of your AWS LastPass entries. Otherwise, one of them will overwrite the hidden form fields on the Two-Factor authentication page

Ubuntu 20.04 Cloud-Init Example to Create a User That Can Use sudo

Use the steps below and example config to create a cloud-init file that creates a user, sets their password, and enables SSH access. The Cloud Config documentation has some examples, but they don’t actually work for being able to ssh into a server and run commands via sudo

First, create a password hash with mkpasswd command:

$ mkpasswd -m sha-512
Password:  
$6$nq4v1BtHB8bg$Oc2TouXN1KZu7F406ELRUATiwXwyhC4YhkeSRD2z/I.a8tTnOokDeXt3K4mY8tHgW6n0l/S8EU0O7wIzo.7iw1

Make note of the output string. You need to enter it exactly in the passwd line of your cloud-init config.

This is the minimal configuration to create a user using cloud-init:

users:
  - name: brandon
    groups: [ sudo ]
    shell: /bin/bash
    lock_passwd: false
    passwd: "$6$nq4v1BtHB8bg$Oc2TouXN1KZu7F406ELRUATiwXwyhC4YhkeSRD2z/I.a8tTnOokDeXt3K4mY8tHgW6n0l/S8EU0O7wIzo.7iw1"
    ssh-authorized-keys:
    - ssh-ed25519 AAAAC3NzaC1lZDI1zzzBBBGGGg3BZFFzTexMPpOdq34a6OlzycjkPhsh4Qg2tSWZyXZ my-key-name

A few things that are noteworthy:

  • The string in the passwd field is enclosed in quotes
  • lock_passwd: false is required to use sudo. Otherwise, the system user account created will have a disabled password and will be unable to use sudo. You’ll just continually be asked for a password, even if you enter it correctly.
  • I prefer the method of adding the user to the sudo group to grant access to sudo. There are other ways to make that work as well, but I feel like this is the cleanest.
  • Adding any users, will prevent the default ubuntu user from being created.
  • LastPass Challenges with Multiple Organizations

    As a parallel entrepreneur, I’m a participating member of multiple companies. That brings with it some unique challenges, as many software tools don’t gracefully handle a user belonging to multiple organizations. I’ve learned to deal with that in many situations. Typically I’ll often having to log out and back in as the desired user or have multiple browsers or browser profiles open – one for each organization.

    One area that has been particularly challenging has been group password management. There are not a lot of software options, although there are getting to be some new players. LastPass is the most mature option, and is the product that I have used for a long time. I investigated some alternatives including 1Password and DashLane. Both of those looked a little more modern and polished, but neither seemed to have mature support for multiple organizations.

    Lastpass does claim to have robust support for organizations, but there is minimal, if any, mention on their website or elsewhere that mentions belonging to multiple organizations. It has taken me a lot of experimenting, but I’ve finally come up with a solution that works well.

    You might think, as the diagram above indicates, that each organization to which you belong should invite your personal account to become a member of the organization. You would be wrong. Although this seems like the intuitive relationship, it does not work since LastPass only allows a personal account to attach to exactly one LastPass Enterprise account. Not more.

    The correct way to belong to multiple Enterprise Accounts in LastPass is to choose one of the organizations to be your “Main” account to which you log in on a daily basis. You connect your Personal account to this enterprise account so that your personal sites appear alongside your work passwords.

    Then, to add additional organizations, you don’t purchase a user license in those other organizations. Instead you create one or more shared folders, and share the folders with the email address for your “Main” organization account. There is a limitation that you can’t be an admin of the shared folders in these other organizations since you are not part of the Enterprise, but sharing and day-to-day password usage works generally as expected.

    This method seems less intuitive, but works well now that I’ve figured it out. As I’ve learned more about how LastPass works internally, I understand why this unorthodox configuration is required

    A few other quirks I’ve found, which just take some getting used-to:

    • Shared folders from my personal account DO NOT SHOW UP when logged into my enterprise account. You have to share to your main organization email address instead.
    • Folder structure in my Personal Account is not confusing in the User-Interface when browsing passwords in my enterprise account. The folder-within-folder structure doesn’t render well, and it is confusing as to which “level” I’m at.

    I hope that the folks at LastPass are able to simplify this or make it more obvious how it is to be configured.

    Do you have a better solution for password sharing with multiple organizations? Please let me and others know in the comments.

    Setting Up Virtualmin on an OpenVZ Guest

    I’m experimenting with a hosting control panel and am interested in Virtualmin. I generally avoid web-based control panels, because they generally make direct configuration via the command line and manually editing config files very difficult. However one of Virtualmin’s goals is to not interfere with such manual configurations. I’ve had plenty of clients who use Webmin, and they seem to do a good job, so Virtualmin seems like a good choice.

    These are the steps that I went through to get a new OpenVZ guest set up with the GPL version of Virtualmin.

    Download a CentOS 5 OS template and create the guest

    # wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz
    # vzctl create <VEID> --ostemplate centos-5-x86_64
    

    I replaced all of these limits in /etc/vz/<VEID>.conf. This is based off of a different running machine with some fairly generous limits. Most importantly, it includes 1GB of RAM.

    # UBC parameters (in form of barrier:limit)
    KMEMSIZE="43118100:44370492"
    LOCKEDPAGES="256:256"
    PRIVVMPAGES="262144:262144"
    SHMPAGES="21504:21504"
    NUMPROC="2000:2000"
    PHYSPAGES="0:9223372036854775807"
    VMGUARPAGES="65536:65536"
    OOMGUARPAGES="26112:9223372036854775807"
    NUMTCPSOCK="360:360"
    NUMFLOCK="380:420"
    NUMPTY="16:16"
    NUMSIGINFO="256:256"
    TCPSNDBUF="10321920:16220160"
    TCPRCVBUF="1720320:2703360"
    OTHERSOCKBUF="4504320:16777216"
    DGRAMRCVBUF="262144:262144"
    NUMOTHERSOCK="5000:5000"
    DCACHESIZE="3409920:3624960"
    NUMFILE="18624:18624"
    AVNUMPROC="180:180"
    NUMIPTENT="128:128"
    

    Then set up some host-specific parameters and start it up.

    # vzctl set <VEID> --ipadd 10.0.0.1 --hostname yourhostname.com --nameserver 4.2.2.1 --diskspace 4G --save
    # vzctl start <VEID>
    # vzctl enter <VEID>
    

    You are now logged in to the guest, where you can download and install virtualmin

    # yum update
    # cd /root
    # wget http://software.virtualmin.com/gpl/scripts/install.sh
    # sh install.sh
     Continue? (y/n) y
    

    That should install without significant errors. Finally, set a password for root, and then log in to Virtualmin to go through the post-installation configuration

    passwd root
    

    Login at https://<your-ip>:10000/ and go through the post-installation configuration

    ProFTPd allows multipled DefaultRoot lines for flexible chrooting

    The ProFTPd documentation gives good examples of how to use the DefaultRoot directive to chroot users to a specific directory.

    A customer today wanted to have different chroot directories for different groups of users. The documentation didn’t mention if it was okay to include multiple DefaultRoot lines. After some experimenting, I can verify that it is allowed and works well.

    I used something like this in /etc/proftpd/proftpd.conf

    DefaultRoot                     ~ jailed
    DefaultRoot                     ~/../.. othergroup
    

    Users in the group ‘jailed’ are chrooted to their own home directory immediately upon logging in. Users in the ‘othergroup’ are chrooted two levels up from their home directory. If you want to get really specific, each user generally has a group of their own, so you can effectively do this a the user-level as well.

    MySQLDump To a Remote Server

    I was running out of disk space on a server today. The server had a large database table that was no longer used, so I wanted to archive it and then drop the table. But the server didn’t have enough disk space to dump it out to disk before copying it off to a remote server for archiving.

    The first thought was to run mysqldump dump on the destination machine, and to access the database over the network. That however, doesn’t compress or encrypt the data. Plus I would have had to create a mysql user with permission to access the database remotely.

    The solution I came up with worked out well: mysqldump directly to the remote host with this command:

    mysqldump <DATABASE_NAME> [mysqldump options] | gzip -c | ssh user@remotehost "cat > /path/to/some-file.sql.gz"
    

    That pipes the mysqldump command through gzip, then to through and SSH connection. SSH on the remote side runs the ‘cat’ command to read the stdin, then redirects that to the actual file where I want it saved.

    Testing for Vulnerable Caching Name Servers

    Most of the technical community has probably heard of the recently found DNS weakness.  The basic premise is that if a recursive nameserver doesn’t use sufficently random source ports when making recursive queries, it can be vulnerable to an attacker who is trying to poisen the cache, or fill it with incorrect data.

    I’ve now heard reports about it from various news sources who make it sound much more drastic than it actually is.   Granted, it is a serious flaw, but fortunately most companies with any desire for security use SSL, which provides an additional layer for identity verification.  Also, for most any company with an IT staff, patching the DNS server with the required fixes should be a fairly trivial task.   The most important servers to be fixed are those run by ISPs and Datacenters, both of which should have their servers fixed by now.

    Tools for testing your DNS servers are fairly easy to come by.  dns-oarc.net has a web-based test, although I don’t know how it discovers your DNS Servers.  For windows users, you can run ‘nslookup’ like this:

    C:\Documents and Settings\Brandon>nslookup
    Default Server:  cns.manassaspr.va.dc02.comcast.net
    Address:  68.87.73.242
    
    > set type=TXT
    > porttest.dns-oarc.net
    Server:  cns.manassaspr.va.dc02.comcast.net
    Address:  68.87.73.242
    
    Non-authoritative answer:
    porttest.dns-oarc.net   canonical name = porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.
    j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net
    porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net
    text =
    
            "68.87.73.245 is GREAT: 26 queries in 2.3 seconds from 25 ports with std
     dev 16592"
    >

    To test from a linux machine, you can use dns-oarc’s test with dig like this:

    root@server:~# dig porttest.dns-oarc.net in txt +short
    porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
    "72.249.0.34 is GREAT: 26 queries in 1.2 seconds from 26 ports with std dev 20533"

    Your are looking for a response that contains GOOD or GREAT. If your results contain something else, you should notify your ISP or Data Center to have them fix their servers.

    Random Password Generator

    There are times when I’ve been focusing on programming all day, and it is easier to write a program to do something trivial, then it is to just do it the simple way. Today was such a day. Instead of typing some random character to make up new user’s password, I wrote a script to do it for me:

    #!/usr/bin/perl
    #################################################################
    ## Quick Random Password Generator
    ## Author: Brandon Checketts
    ## http://www.brandonchecketts.com/
    #################################################################
    
    my $length = $ARGV[0] || 10;
    
    my $charset = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
    
    my $pw = "";
    for (my $i=0; $i < $length; $i++) {
        my $pos = rand(length($charset));
        $pw .= substr($charset, $pos, 1);
    }
    print "\\nRandom Password: $pw\\n\\n";
    

    Of course you can modify the default length and/or characters to make something more suitable for your use.

    Sample Usage:

    [root@dev ~]# ~/bin/pwgen
    
    Random Password: mYTZrSpE8B
    
    [root@dev ~]# ~/bin/pwgen 20
    
    Random Password: EoSQpypmeK3SZCVPodaM
    

    Getting Un-locked out of Webmin

    Webmin has come a long way in the last year or two. I still (and always will) prefer the command line, but many customers are certainly much more comfortable using a web-interface to configure their server. I have to reset a password every once in a while and have to look up the steps every time. It seems I can never find them quickly, so I’ll put them here so I can find them easily next time.

    Blocked IP addresses are stored a text file named /var/webmin/blocked. IP Address should be cleared out after some period of time, but you can hasten the process by clearing it manually:

    cp /dev/null /var/webmin/blocked
    

    Webmin passwords are stored in /etc/webmin/miniserv.users. A script for changing a user password is provided with webmin. Just run it like this:

    /usr/libexec/webmin/changepass.pl /etc/webmin root