In my previous post I showed you how we integrated the Pwned Passwords check from Troy Hunt's https://haveibeenpwned.com into our login pages on the EVE Online SSO, which is used by our game launcher, when logging into our websites and when logging into 3rd party integrations.
Our feedback after the launch was very encouraging, as our users were overall very appreciative of us taking steps towards better security, and this has been noticed by other companies and organizations as well. As an example, Epic Games also started checking new passwords using the same way as of April 16th 2019, which is awesome, and I'd like to think our implementation at least served as an encouragement towards that :) Well done Epic Games!
One question I got a lot was how the password checks did from a performance perspective. Troy has gone over this on several occasions on his blog (links at the end of the article) so I'll leave the details of the way he does caching to him. What I can say is the service is absolutely rock-solid and performs great. We do some caching on our end as well, but we hardly see any failed requests and the response times are usually very good due to Cloudflare caching magic and their global CDN. Here you can see the response times for the last three months
Not only is the response time great, but just look at that histogram!
That's a median response time of 25 milliseconds (we are hitting a local Cloudflare node in London) and a 99th percentile of 710ms, which considering everything as absolutely fantastic.
As for failures....
That's a total of 2.000 failures for over 2.02 million requests over the last three months, almost all of them timeouts (we have an aggressive 5 second timeout on our end), and I know for a fact that the big spike in the middle was a proxy misconfiguration on our end, but even so, that's still less than a 0.1% request failure! So the reliability and consistency of the service is fantastic.
Overall, our Pwned Passwords integration has been quite the success.
But, we decided to do more!
Shortly after the launch me and a coworker went to NDC Oslo where I had the pleasure of attending Troy's Hack Yourself First workshop. That workshop, which I highly recommend for every developer, was great. I already knew about most of the topics in the workshop, but hadn't really spent any time into digging into them in detail. Troy gave an excellent "hand's on" approach, with engaging conversations and good examples on the topics involved and he has got the presentation of it absolutely nailed down.
We also had a few chats (with beers), I got introduced to Scott Helme (who's also an EVE player!) and after a few discussions (and perhaps a few more beers) I knew of two rather easy wins to work on when I got back.
I won't go into exact details on exactly how we did the CSP implementation but I will give a shout out to Report-URI which is an excellent service to help you set up, gather, aggregate and act on these CSP security policies (and more). That service is also run by Scott Helme.
We ran the CSP policies in Report-Only mode at first, while we tweaked the policies so they wouldn't break anything, and once we were certain we had everything working, we took them out of Report-Only mode and started enforcing them. This did not take very long. We went from the CSP being published in Report-Only mode on June 21st to being enforced on July 3rd. That's less than two weeks!
And then we went on to the next thing...
HSTS and preloading eveonline.com!
HSTS stands for HTTP Strict Transport Security and it's a simple HTTP header that tells browsers that the site should only be loaded and run over HTTPS. It's a simple thing to do once you have HTTPS up and running, and makes sure that the site can not be modified in any way between our users browsers and our servers as well as makes sure that all communication is encrypted. Once I wassure that everything was working fine over HTTPS I went ahead and submitted our https://eveonline.com domain to the HSTS Preload list, where it has since been accepted and is now preloaded on most modern browsers.
Preloading means that browsers won't even try to load the site (or any subdomains) over HTTP, even if you type http://eveonline.com in the browser, and will just immediately redirect to HTTPS. It also means that we can not easily go back to hosting anything over plain old HTTP, which is also the whole point. We want to make sure our sites are secure and that they are only ever loaded over encrypted channels.
Making these changes was all rather easy to do and implementation went well. There are excellent guides out there on how to do CSP and HSTS (I'll provide links below) so I'd higly recommend anyone running a website to do it.
I hope this encourages you to do the same :)
Troy Hunt on Pwned Passwords