Transcription
A Comparative Usability Studyof Key Management in Secure EmailScott Ruoti, University of Tennessee; Jeff Andersen, Tyler Monson, Daniel Zappala,and Kent Seamons, Brigham Young 018/presentation/ruotiThis paper is included in the Proceedings of theFourteenth Symposium on Usable Privacy and Security.August 12–14, 2018 Baltimore, MD, USAISBN 978-1-939133-10-6Open access to the Proceedings of theFourteenth Symposiumon Usable Privacy and Securityis sponsored by USENIX.
A Comparative Usability Study of Key Managementin Secure EmailScott RuotiJeff AndersenTyler MonsonUniversity of TennesseeBrigham Young UniversityBrigham Young @isrl.byu.eduDaniel ZappalaKent SeamonsBrigham Young UniversityBrigham Young RACTWe conducted a user study that compares three secure emailtools that share a common user interface and differ onlyby key management scheme: passwords, public key directory (PKD), and identity-based encryption (IBE). Our workis the first comparative (i.e., A/B) usability evaluation ofthree different key management schemes and utilizes a standard quantitative metric for cross-system comparisons. Wealso share qualitative feedback from participants that provides valuable insights into user attitudes regarding eachkey management approach and secure email generally. Thestudy serves as a model for future secure email research withA/B studies, standard metrics, and the two-person studymethodology.1.INTRODUCTIONThe cryptography needed to deploy secure email is wellstudied and has been available for years, and a numberof secure email systems have been deployed and promotedrecently, including ProtonMail, Tutanota, Mailvelope, Virtru,Voltage, Encipher.it, etc. While some of these systems havemillions of users, the vast majority of email users still do notuse secure email [21]. The lack of adoption of secure emailis often attributed to the significant gap between what thetechnology can offer and the ability of users to successfullyuse the technology to encrypt their emails.Beginning with Whitten and Tygar [36], secure email usability studies have shown that key management is a significanthurdle for users. More recent usability studies (e.g., [1, 2, 23])show signs that progress toward greater usability is beingmade, but limitations in each study make it difficult to drawconclusions regarding the impact key management has onsecure email usability, other than the need for automation.We previously conducted studies [23, 24, 26, 27] that directlycompared key management schemes from different families,but the systems implementing the various key managementschemes were wildly different, introducing a significant con-founding factor. Bai et al. [2] compared two key managementschemes, but their study explored user mental models andtrust, not usability generally.Additionally, even though public key directories have recentlyreceived significant attention [19, 29], it is unclear how theirusability compares to other key management schemes. Lerneret al. [18] studied a public key directory system but didn’tuse a standard metric, making it difficult to directly comparetheir results to past work. Atwater et al. [1] simulated apublic key directory, but permitted a user to send an email toa recipient who had not yet generated a key pair. Normally,when a user attempts to send an email to a recipient who hasnot yet generated a key pair, they must wait until the userdoes so and uploads their public key to the key directory.Because this affected numerous participants in their study,it is unclear how this issue impacted their results.Our work was motivated by the desire to build on theseearlier studies and reduce the number of confounding factorsin order to increase our confidence in the resulting usabilitymeasurements. In this paper, we describe a user study comparing three key management schemes, taken from differentfamilies, to better understand how key management impactsthe usability of secure email during initial setup and first useof the system. Using the MessageGuard research platform,we built three secure email tools which differ only in the keymanagement scheme they implement (passwords, public keydirectory [PKD], and identity-based encryption [a]), reducing potential confounding factors in the study. In our studydesign we used a standard metric, allowing comparison toresults from past studies. Finally, we replicated our earlierpaired participant study setup [23], allowing us to evaluategrass roots adoptability.In total, 47 pairs of participants completed our study. Allthree systems received favorable ratings from users, withserver-derived public keys being considered the most usable,followed by user-generated public keys, and finally sharedsecrets. Each system performed better than similar (i.e., samekey management) systems previously studied in the literature.Users also provided valuable qualitative feedback helpingidentify pros and cons of each key management scheme.The contributions of this paper include:Copyright is held by the author/owner. Permission to make digital or hardcopies of all or part of this work for personal or classroom use is grantedwithout fee.USENIX Symposium on Usable Privacy and Security (SOUPS) 2018.August 12–14, 2018, Baltimore, MD, USA.USENIX Association1. First A/B evaluation of key management usingstandard metrics. Our study was able to confirmAtwater et al.’s [1] findings that public key directoriesFourteenth Symposium on Usable Privacy and Security375
are usable. Additionally, we find evidence that thesecure email design principles we identified in previouswork [24] generalize beyond server-derived public keys.There are numerous ways to verify the authenticity of apublic key (i.e., the binding of a public key to an emailaddress), some of which include:2. The MessageGuard platform. To enable this work,we built MessageGuard, a research platform for buildingsecure email and other end-to-end encryption prototypes. MessageGuard significantly simplifies the effortrequired to work in this space and provides a meanswhereby research results may be shared and replicated.MessageGuard has a pluggable architecture, making iteasy to build prototype variants for use in A/B testing.1. Manual validation. Users can directly communicatewith each other and directly share their public key orcompare key fingerprints1 . Users are expected to knoweach other personally and thus be able to confirm theidentity of those they are communicating with.2. Web of trust. Users can have their public key signedby one or more other users, who are expected to onlysign public keys that they have verified using manualkey validation. When retrieving a public key, userscheck to see if it has been signed by a user they trustto have validated it properly. Users may choose totransitively trust public keys that are trusted by usersthey trust, forming a web of trust.3. Lessons learned and recommendations. Our studyelicits user attitudes regarding the three key management schemes we evaluate, including security and usability trade-offs identified by participants. For example, even after understanding that the user-generatedpublic key scheme protects against a stronger threatmodel than server-derived public keys, many users indicate that they do not need that level of security andprefer server-derived public keys because they can immediately send email without waiting for the recipientto generate a public/private keypair. Based on ourfindings, we give recommendations for future work.2.3. Hierarchical validation. Users can have their publickey signed by an authoritative signer (e.g., a certificate authority), which will only sign a public key afterverifying that the user who submitted it owns the associated email address. When retrieving a public key,its signature is validated to ensure that it was properly signed by an authoritative signer. This methodof key validation is most commonly associated withS/MIME [8, 13].BACKGROUNDIn this section, we first describe several key managementschemes commonly used with end-to-end email encryption.Next, we provide a chronological review of usable secureemail research.2.14. Public key directory. Users can submit their publickeys to a trusted key directory. This directory will onlyaccept and disseminate public keys for which it hasverified that the user who submitted the key owns theassociated email address. Due to its trusted nature,keys retrieved from the directory are assumed to beauthentic. The behavior of the key directory can beaudited through the use of certificate transparency [29]or a CONIKS-like ledger [19].Key ManagementWe study three families of key management schemes usedin end-to-end encryption of email: shared secrets, usergenerated public keys, and server-derived public keys. Eachhas different methods for creating, sharing, and linking cryptographic keys to email addresses. We describe each schemebriefly; a more complete treatment can be found in [9].2.1.1Shared SecretsUsers can encrypt their emails using symmetric keys derivedfrom a secret shared between pairs of users. Most commonly,these secrets are in the form of simple passwords, which aremore readily communicated and remembered by users thancryptographically secure random values. The security ofthis key management scheme is dependent on users’ abilityto satisfy the following requirements when they create andshare passwords: (1) choose a unique password for each userthey will communicate with, (2) choose passwords that willresist a brute-force attack, (3) communicate passwords overa secure channel, and (4) safely store passwords.2.1.2User-generated Public KeysBefore sending or receiving encrypted email, users must firstgenerate a cryptographic key pair. A user’s private keyshould never be shared with any other party and must besafely stored by the user. The user’s public key, with relevantmetadata, is then distributed to other users in a number ofways, such as sending the key directly to other users, postingthe key to a personal website, or uploading the key to a keydirectory.376Manual verification and the web of trust are commonly associated with PGP [12], though any of the above can be usedwith PGP.The security of these schemes depends on the ability of usersto protect their private keys, obtain necessary public keys,and faithfully validate these public keys. If users lose accessto their private keys (e.g., disk failure with no backup), theywill be unable to access their encrypted email.2.1.3Server-derived Public KeysIn this scheme, a user’s public key is generated for them bya server they trust, which may also store their private key(called key escrow). This alleviates the problems associatedwith a user losing their private key, and is often used incorporate environments. A variant of this scheme is identitybased encryption (IBE) [31]. With IBE, a user’s public key isgenerated mathematically based on their e-mail address andpublic parameters provided by an IBE key server. A user’sprivate key is also generated by the IBE key server, whichwill only release that key to the user after the user verifiesownership of the associated email address. In any situation1A public key’s fingerprint is typically derived from a cryptographic hash of the public key.Fourteenth Symposium on Usable Privacy and SecurityUSENIX Association
ComparativeA/B StudyStandard metricTwo-personGarfinkel and Miller [13] created a secure email system usingS/MIME (hierarchical key validation) and demonstrated thatautomating key management provides significant usabilitygains. However, their study also revealed that the tool “wasa little too transparent,” leading to confusion and mistakes.X X XXX X XXXThis workRuoti et al. [24]Lerner et al. [18]Bai et al. [2]Ruoti et al. [23, 26]XXXXXTable 1: Comparison of Usable Secure Email ResearchWe previously created Private WebMail (Pwm) [27], a secureemail system that tightly integrates with Gmail and usesidentity-based encryption (IBE) to provide key managementthat is entirely transparent to users. User studies of Pwmdemonstrate that it was viewed very positively by users, andsignificantly outperformed competing secure email systems.Atwater et al. [1] compared the usability and trustworthinessof automatic versus manual encryption, finding that therewere no significant differences between the two approaches.As part of this study, Atwater et al. developed two emailclients—one integrated with Gmail and one standalone—bothof which simulated the user experience of using a public keydirectory.We also developed a novel two-person methodology [23] forstudying the usability and grassroots adoptability of secureemail. In particular, this study involved recruiting pairs ofrecipients (e.g., friends, spouses), who would then be responsible for sending secure email among themselves. Comparedto single-participant studies, this methodology revealed differences between the experience of initiating others and being initiated by others into using secure email. Our studycompared systems using three different families of key management: shared password, public key directory, and IBE;unfortunately, confounding factors in this study make it difficult to draw any conclusion on how key management affectssecure email’s usability.Bai et al. explored user attitudes toward different models forobtaining a recipient’s public key in PGP [2]. In their study,they built two PGP-based secure email systems, one thatused manual key validation and one that used a public keydirectory. Users were provided with instructions on how touse each tool and given several tasks to complete. The resultsof this study showed that, overall, individuals recognized thesecurity benefits of manual key validation, but preferredthe public key directory and considered it to have sufficientsecurity. While this study gathered data on user attitudesregarding two key management schemes, it did not evaluatetheir usability.More recently, we further refined our Pwm system [24], identifying four design principles that increase the usability, correctbehavior, and understanding of secure email: (1) having informative and personalized initiation messages that guideusers through installing the secure email software and givethem confidence that the email they received is not malicious;USENIX AssociationAtwater et al. [1]Usable Secure EmailRuoti et al. [27]2.2Whitten and Tygar [36] conducted the first formal user studyof a secure email system (PGP 5 with manual key validation),uncovering serious usability issues with key management andusers’ understanding of the underlying public key cryptography. The results of their study took the security communityby surprise and helped shape modern usable security research.Garfinkel and Miller [13]Whitten and Tygar [36]when a user cannot trust a server with their private key (e.g.,an activist in an oppressive regime, or a journalist that needsto protect sources) key escrow should not be used.(2) adding an artificial delay during encryption to build trustin the system and show users who their message is beingencrypted for; (3) incorporating inline, context-aware tutorials to assist users as they are sending and receiving theirfirst encrypted emails; and (4) using a visually distinctiveinterface to clearly demarcate which content is encrypted/tobe-encrypted and helping users avoid accidentally sendingsensitive information in the clear.Lerner et al. [18] built Confidante, a secure email tool thatleverages Keybase, a public key directory, for key management. A user study of Confidante with lawyers and journalists demonstrated that these users could quickly and correctlyuse the system.The significant differences between this earlier work and ourcurrent work are summarized in Table 1.3.SYSTEMSTo limit confounding factors in our study, it was necessaryto build several secure email tools that differed only in howkey management was handled. To accomplish this we substantially modified our Private WebMail 2.0 (Pwm 2.0) system [24], leaving its UI unchanged, but otherwise completelyrewriting its codebase to add support for a pluggable keymanagement subsystem. This allowed us to rapidly developthree secure email prototypes that only differed in how theyhandled key management, while keeping the remaining system components consistent. We call this pluggable versionof Pwm 2.0 MessageGuard.We choose to extend Pwm 2.0 for several reasons. First, it isan existing system with established favorable reviews, savingus a significant amount of development time and helpingavoid the possibility of designing a new secure email tool thatwas viewed unfavorably by users. Second, it had the highestusability score [24] of any secure email systems evaluatedusing the System Usability Metric (SUS) [6]. Third, thisallowed us to test whether the secure email design principlesproposed by Ruoti et al. and implemented in Pwm 2.0 (seeSection 2.2) generalize beyond IBE-based systems.In addition to adding a pluggable key management systemto MessageGuard, we also added several other features toMessageGuard in order to allow other researchers to use itas a research platform for building end-to-end encryptionFourteenth Symposium on Usable Privacy and Security377
prototypes. First, MessageGuard supports a wide rangeof non-email sites (e.g., Facebook, Twitter, Blogger), automatically scanning these pages for user-editable contentand allowing users to encrypt this content end-to-end. Second, the page scanning functionality is pluggable, allowingresearchers to create finely-tuned, per-site end-to-end encryption plugins. Finally, MessageGuard includes pluggable userinterface, encryption, and content packaging subsystems.There are three key benefits to using MessageGuard as aresearch platform:1. Accelerates the creation of content-based encryptionprototypes. MessageGuard provides a fully functionalcontent-based encryption system, including user interfaces, messaging protocols, and key managementschemes. The modular design of MessageGuard allowsresearchers to easily modify only the portions of thesystem they wish to experiment with, while the remaining portions continue operating as intended. Thissimplifies development and allows researchers to focuson their areas of expertise—either usability or security.2. Provides a platform for sharing research results. Researchers who create prototypes using MessageGuardcan share their specialized interfaces, protocols, or keymanagement schemes as one or more patches, allowingresearchers to leverage and replicate each other’s work.Additionally, research can be merged into MessageGuard’s code base, allowing the community to benefitfrom these advances and reducing fragmentation ofefforts.(a) Composition Overlay(b) Read OverlayFigure 1: MessageGuard OverlaysThe source code for MessageGuard is available at cts/MES. The front end scans for encrypted payloads and dataentry interfaces and replaces these items with a secureoverlay. The front end is the only component thatruns outside of MessageGuard’s protected origin, andit can only communicate with overlays using the window.postMessage API. The overlay always encryptsuser data before transmitting it to the front end component and sanitizes any data it receives from the frontend. In addition, the front end also displays tutorials that instruct new users how to use MessageGuard.These are all context-sensitive, appearing as the userperforms a given task for the first time.In the remainder of this section, we give a brief overviewof MessageGuard. Additional details are available in Appendix A–C, and a complete description can be found ina technical report [25]. Next, we describe the workflowfor the three secure email variants that we created usingMessageGuard. We chose well-known instances of each keymanagement scheme and explain the rationale for that choice:passwords, public key directory (PKD), and IBE. Other alternatives and hybrids of these approaches are possible. overlays use iframes and the browser’s same-originpolicy to keep plaintext from being exposed to theemail server and its application. A read overlay displayssensitive information to the user, and a compose overlayallows users to encrypt sensitive information beforesending it to the website. Overlays have a distinctive,dark color scheme that stands out from most websites,allowing users to easily identify secure overlays frominsecure website interfaces.These systems can be downloaded and are available for testingat https://{pgp,ibe,passwords}.messageguard.io The packager encrypts/decrypts user data and encodes the encrypted data to make it suitable for transmission through web applications. The packager usesstandard cryptographic primitives and techniques toencrypt/decrypt data (e.g., AES-GCM). Ciphertext ispackaged with all information, save the key material,necessary for recipients of the message to decrypt it.3. Simplifies the comparison of competing designs. MessageGuard can be used to rapidly develop prototypesfor use in A/B testing. Two prototypes built usingMessageGuard will only differ in the areas that havebeen modified by researchers. This helps limit the confounding factors that have proven problematic in pastcomparisons of content-based encryption systems.3.1MessageGuardMessageGuard tightly integrates with existing web applications, in this case Gmail, using security overlays. Securityoverlays function by replacing portions of Gmail’s interfacewith secure interfaces that are inaccessible to Gmail. Usersthen interact with these secure overlays to create and readencrypted email (Figure 1a and Figure 1b).Figure 2 shows the MessageGuard architecture:378Fourteenth Symposium on Usable Privacy and Security The key management component enables a varietyof key management schemes to be configured, withoutchanging other aspects of MessageGuard such as theread or compose overlays.USENIX Association
dress.2 Their address is verified by having the userclick a link in an email sent to them. They are thenable to download the system.2. After installation, the user is told that the systemwill generate a key pair for them. The public key isautomatically uploaded to the key directory, as the useris already authenticated to the key directory from theprevious step.3. The user attempts to send an encrypted email but isinformed that the recipient hasn’t yet installed the system.3 They are then prompted to send their recipientan email inviting them to install the system. This emailmessage is auto-generated by MessageGuard, with thesystem able to add a custom introduction message ifdesired.A user’s sensitive data is only accessible within the MessageGuardorigin.Figure 2: MessageGuard Architecture4. Once the recipient has installed the system, whichgenerates and publishes their public key, they informthe sender that they are ready to proceed. The sendercan now send their encrypted email.3.4Identity-based Encryption (IBE)We choose to evaluate IBE because it is the key managementscheme that has been shown to be most usable in past studies,providing a good baseline for this work. The workflow forour IBE-based system is as follows:1. The user visits the MessageGuard website. They are instructed to create an account with their email address.2Their address is then verified by having the user clicka link in an email sent to them. They are then able todownload the system.Figure 3: Dialog for Entering a New Password with Whichto Encrypt Email.3.22. After installation, the user is informed that the system will retrieve their IBE key from the key server.This happens automatically because the user is alreadyauthenticated to the key server from the previous step.PasswordsWe choose to evaluate passwords as they are a scheme thatshould be familiar to users. The workflow for our passwordsystem is as follows:3. The user can send encrypted email to any address.1. The user visits the MessageGuard website. They areprompted to download the system.4. The recipient, upon receiving the encrypted email, isprompted to visit the MessageGuard website and createan account. After their address is verified and theirprivate key is downloaded from the key server, theycan read the encrypted message.2. After installation, the system is immediately ready foruse.3. When the user attempts to send an encrypted email,they are informed that they need to create a passwordfor encrypting the email (see Figure 3). After creatingthe password, the user can send their encrypted email.4. The user must communicate to the recipient the password used to encrypt the email message. This shouldhappen over an out-of-band (i.e., non-email) channel.3.3Public Key Directory (PKD)We choose to evaluate public key directories because theyhave received significant attention lately [2, 18, 19, 29]. Theworkflow for our public key directory system is as follows:1. The user visits the MessageGuard website. They areinstructed to create an account with their email ad-USENIX Association4.METHODOLOGYWe conducted a within-subjects, IRB-approved lab studywherein pairs of participants used three secure email systemsto communicate sensitive information to each other (studymaterials are found in Appendix D). Our study methodologyis patterned after our previous paired participant methodology [23], allowing us to examine usability in the context of2We chose to require a MessageGuard account in order to prevent a compromised email provider from being able to transparently upload (PKD) or download (IBE) cryptographickeys from the MessageGuard key server, which would be possible if these operations were only protected by email-basedauthentication.3The recipient must install the system and use it to uploada public key before the sender can encrypt email for therecipient.Fourteenth Symposium on Usable Privacy and Security379
two novice users, without potential bias or other behaviorsintroduced by direct involvement with a study coordinator.The study ran for two and a half weeks—beginning Monday,May 23, 2016, and ending Tuesday, June 7, 2016. In total,55 pairs of participants (110 total participants) took thestudy. Due to various reasons discussed later in this section,we excluded results from eight participant pairs. For theremainder of this paper, we refer exclusively to the remaining47 pairs (94 participants).4.1she would receive some information regarding taxes fromJohnny but was not informed that the information would beencrypted.The tasks they were asked to perform were:1. Johnny would encrypt and send his SSN and last year’stax PIN to Jane.2. Jane would decrypt this information, then reply toJohnny with a confirmation code and this year’s taxPIN. The reply was required to be encrypted.Study Setup3. After Johnny received this information, he would informJane that he had received the necessary information,and then the task would end. This confirmation stepis added to ensure that Johnny could decrypt Jane’smessage. We did not require the confirmation messageto be encrypted.Participants took 50–60 minutes to complete the study, andeach participant was compensated 15 USD in cash. Participants were required to be accompanied by a friend, whoserved as their counterpart for the study, and were instructedto use their own Gmail accounts.4When participants arrived, they were given a consent formto sign, detailing the study and their rights as participants.Participants were informed that they would be in separaterooms during the study and would need to use email to sharesome sensitive information with each other. They were toldthat they were free to communicate with each other howeverthey normally would, with the caveat that the sensitiveinformation they were provided must be transmitted overemail. Additionally, participants were informed that theycould browse the Internet, use their phones, or engage inother similar activities while waiting for email from theirfriend. This was done to provide a more natural setting forthe participants, and to avoid frustration if participants hadto wait for an extended period of time while their friendfigured out an encrypted email system. Finally, participantswere told that a study coordinator would be with them at alltimes and could answer questions related to the study butwere not allowed to provide instructions on how to use anyof the systems being tested.4.2Study TasksUsing a coin flip, one participant was randomly assigned asParticipant A and the other as Participant B (referred toas “Johnny” and “Jane”, respectively, throughout the paper).The participants were then led to separate rooms to beginthe study. The participants were then guided through thestudy by following a Qualtrics survey, which included bothinstructions and then questions regarding their experience.After answering demographic questions, participants wereasked to complete a multi-stage task three times, once foreach of the secure email systems being tested. The orderin which the participants used the systems was randomized.To complete this task, participants were asked to role-playa scenario about completing taxes. Johnny was told thathis friend, Jane, had graduated in accounting and was goingto help Johnny prepare his taxes. To do so, Johnny neededto send her his social security number and his last year’stax PIN. Johnny was told that because this information wassensitive, he should encrypt it using a secure email system hecould download at a URL we gave him. Jane was told that4Using their own accounts increases ecological validity, buthas privacy implications. To help mitigate these concerns wehave destroyed the screen recordings for this study. Thoughnot used, we did prepare study accounts for any participantswho were not comfortable using their own account.380During each stage, participants were provided with worksheets containing instructions regarding the task an
of secure email systems have been deployed and promoted recently, including ProtonMail, Tutanota, Mailvelope, Virtru, Voltage, Encipher.it, etc. While some of these systems have millions of users, the vast majority of email users still do not use secure email [21]. The lack of adoption of secure email
