WIXLIB

Copyright 2015 Splunk Inc.Best Practices for Splunk SSL(The SSLippery Slope Revisited)Duane WaddleGeorge StarcherDefense Point Security

SSL RefresherAuthentication of the server (the server is who they say they are)Optional authentication of the clientBulk encryption of data in transitSeveral moving parts, “CAs”, “keys”, “CSRs”, “certs”We often say "SSL" when we mean "TLS". True SSL is effectivelydead. (OR IS IT!?)2

Splunk Architecture and SSLSplunkweb (SSL to browsers)Splunk-to-splunk data transfer (forwarders to indexers)Splunkd REST port (Inter-Splunk)Deployment Client / Deployment ServerREST API / SDKsDistributed SearchLDAP connectionsClustering3

Splunk’s default SSL postureThe out-of-the-box configuration:All certificates are generated on a default-shipped CAconfigurationSplunkweb does not use SSLSplunkd uses SSL for the REST port - with certificate verificationdisabledNo SSL data inputs/outputs are definedSplunkd LDAP can use SSL - again with no certificate verification4

Splunk’s default SSL postureBut for Splunk 6.3.3.4 / 6.3.4Apparently turns off SSLv3 by default (undocumented)Breaks communication with some older forwarders / patch levels (like6.1.0)sslVersions tls, ssl3Don't use this as an excuse not to upgrade!http://www.splunk.com/view/SP-CAAAPKV wn-openssl-vulnerability/ (blog post w/d/l links)5

SSLType of exchangeClient functionServer functionEncryptionCertificateAuthenticationCommon NamecheckingType of dataexchangedBrowser to Splunk WebBrowserSplunk WebNOT enabled bydefaultdictated by client(browser)dictated by client(browser)search term resultsInter-SplunkcommunicationSplunk Websplunkdenabled by defaultNOT enabled by defaultNOT enabled by defaultsearch term resultsForwardingsplunkd as a forwardersplunkd as an indexerNOT enabled bydefaultNOT enabled by defaultNOT enabled by defaultdata to be indexedInter-Splunkcommunicationsplunkd as a deploymentclientsplunkd as deploymentserverenabled by defaultNOT enabled by defaultNOT enabled by defaultconfiguration dataInter-Splunkcommunicationsplunkd as a search headsplunkd as search peerEnabled by defaultNOT enabled by defaultNOT enabled by defaultsearch data6

Why this stuff mattersA DPS penetration tester found himself on a random Linux box as anunprivileged user. This box was:Running Splunk forwarder as root. with the default admin/changeme password. and default SSL configs, trusting any certificate7

Why this stuff mattersHe was able to:Use the REST API to change the deployment server IP (to his box)Restart the forwarderDownload an app to the forwarder that started a reverse root shellPivot from root on that box to downloading the site's Chef repoLift a copy of all of their recipes, including AWS API keysMoral of the story:Whoever controls your DS controls the users running your forwarders.8

(some)Best Practices ChecklistRun Splunk forwarders as an unprivileged userChange forwarder admin passwordsEnable strong SSL authentication between DS client and DS serverUse host based firewall to limit outbound connections to trusted IPsPick an appropriate cipherSuiteUse wildcard SSL certs with cautionDecide on FIPS mode early on and talk to Splunk firstNot running 6.3 yet? Still running on the default ml9

Commercial CA or Private CA?CommercialRoot certs are in everyone's browser alreadyCosts real money (potentially a LOT if you use ECC)Potential renewal nightmarePrivateYou have to run a CA (likely already are.)Free (ish)Root certs must be distributedYou can do very long expirations (in theory)10

How many certs do I need?Splunkweb - Search Head:A 3rd party CA cert and its root / intermediatesSplunkd:A root cert and its intermediates - (either 3rd party or private)One per Splunk Server non Search Head(or one per role in large envs)One throwaway certificate for all of the Splunk UFs to share11

Our Example Architecture12

Create Splunk Server Key & CSR mkdir SPLUNK HOME/etc/auth/myOrg cd SPLUNK HOME/etc/auth/myOrg openssl req -nodes -newkey rsa:2048 -keyoutsplunk-srv1.web.key -out splunk-srv1.csr openssl rsa -in splunk-srv1.web.key -des3 -outsplunk-srv1.keyGet the CA root certificate chain and put in auth/myOrg as cacert.crt.Copy the CA-returned crt file to auth/myOrg/splunk-srv1.crt13

The Search Head - Splunk Web cd SPLUNK HOME/etc/auth/myOrg cat splunk-srv1.crt cacert.crt splunk-srv1.web.pem vi SPLUNK nkWebSSL 1httpport 8443privKeyPath etc/auth/myOrg/splunk-srv1.web.keycaCertPath etc/auth/myOrg/splunk-srv1.web.pemsslVersions tls, -tls1.0cipherSuite ?14

Architecture - Status15

IndexersFirst make certificates as you would for SplunkWeb. cd SPLUNK HOME/etc/auth/myOrg openssl req -nodes -newkey rsa:2048 -keyout splunkidx01.key -out splunk-idx01.csr openssl rsa -in splunk-idx01.web.key -des3 -outsplunk-idx01.keyTo make the Indexer formatted .PEM: cat splunk-idx01.crt splunk-idx01.key cacert.crt splunk-idx01.pem16

The Indexer - Inputs.confvi SPLUNK 9998]disabled 0[SSL]password REDACTED rootCA SPLUNK HOME/etc/auth/myOrg/cacert.crtserverCert SPLUNK HOME/etc/auth/myOrg/splunk-idx01.pemsslVersions tls, -tls1.0cipherSuite ?requireClientCert true false17

The ForwarderFirst make certificates as you would for SplunkWeb.This can be done on your deployment server. cd SPLUNK HOME/etc/auth/myOrg/forwarder openssl req -nodes -newkey rsa:2048 -keyout splunkforwarder.web.key -out splunk-forwarder.csrOn forwarders only, make the key password "password" for reasons . openssl rsa -in splunk-forwarder.web.key -des3 -out splunkforwarder.keyThrow away splunk-forwarder.web.key cat splunk-forwarder.crt splunk-forwarder.key cacert.crt splunkforwarder.pemCopy the splunk-forwarder.pem and cacert.crt to your Forwarder(s): Yes you could use an APP for this.18

Forwarder to Indexer - Outputs.confvi SPLUNK roup myIndexers[tcpout:myIndexers]server splunk-idx01.myorg.com:9998sslCertPath SPLUNK d password # For ReasonssslRootCAPath SPLUNK HOME/etc/auth/myOrg/cacert.crtsslVerifyServerCert truesslCommonNameToCheck splunk-idx01.myorg.com19

Gotcha - Forwarder to IndexerIf you mistype the sslRootCAPath argument in outputs.conf, the forwarder willdefault to not-SSL when trying to talk to indexer. The error on the indexer will looklike the following:6-23-2014 20:46:48.918 0000 ERROR TcpInputProc - Errorencountered for connection from src 10.0.1.57:41778. error:140760FC:SSL routines:SSL23 GET CLIENT HELLO:unknownprotocol20

Architecture - Status21

The Deployment ServerFirst make certificates as you would for SplunkWeb. cd SPLUNK HOME/etc/auth/myOrg openssl req -nodes -newkey rsa:2048 -keyout splunkd.web.key -out splunk-d.csr openssl rsa -in splunk-d.web.key -des3 -out splunk-d.keyTo make the Deployment Server formatted pem: cat splunk-d.crt splunk-d.key cacert.crt splunk-d.pem22

The Deployment Server - server.confvi SPLUNK File cacert.crtcaPath SPLUNK HOME/etc/auth/myOrgsslKeysfile splunk-d.pemsslKeysfilePassword REDACTED sslVersions tls, -tls1.0cipherSuite ?requireClientCert false23

Splunk Forwarder - DS Clientsvi SPLUNK File cacert.crtcaPath SPLUNK HOME/etc/auth/myOrgsslKeysfile splunk-forwarder.pemsslKeysfilePassword password# ReasonssslVersions tls, -tls1.0sslVerifyServerCert truesslCommonNameToCheck splunk-d.myorg.com24

Architecture - Status25

Splunk - Server to Servervi SPLUNK File cacert.crtcaPath SPLUNK HOME/etc/auth/myOrgsslKeysfile splunk-srvXX.pemsslKeysfilePassword REDACTED sslVersions tls, -tls1.0cipherSuite ?requireClientCert falsesslVerifyServerCert truesslCommonNameList splunk-srv01.myorg.com, splunk-d.myorg.com, splunkidx01.myorg.com, splunk-idx02.myorg.com, .26

Architecture - Status27

Splunk LDAPSEach LDAP strategy has an SSL toggle on/offIn GUI, it’s a checkboxIn authentication.conf, each LDAP stanza needs SSLEnabled 1Minimum Certificate settings in SPLUNK HOME/etc/openldap/ldap.confTLS REQCERT demandTLS CACERT /opt/splunk/etc/auth/LDAProotcert.crtTLS CIPHER SUITE (equivalent to cipherSuite)28

Indexer ClusteringIndexer clustering uses both REST API and a dedicated cluster data transfer portCerts & config for REST API are all covered aboveSSL signature and common name checking occur BEFORE pass4SymmKeycheckingProtip: If building a cluster from scratch, use the same splunk.secret onall cluster nodesConverting a cluster from default certs to production certs can be brittleEnable sslVerifyServerCert and sslCommonNameList LASTsslCommonNameList needs to list all possible REST communications partnersAll indexers, cluster master, license server, and search heads .29

Indexer Clustering - SSL Data TransferMinimal documentation - only one reference to it in the test/Admin/ServerconfIn server.conf comment out replication-port stanza and add:[replication port-ssl://8002]password REDACTED rootCA SPLUNK HOME/etc/auth/myOrg/cacert.crtserverCert SPLUNK HOME/etc/auth/myOrg/splunk-idx01.pemTry this out in a test cluster first!This is NOT a common setting in the wild30

SHC and KVStoreSHC - same REST port rules apply as with indexer clusteringKVStore has its own SSL config stanza in server.conf:[KVstore]caCertPath sslKeysPath sslKeysPassword Docs mention these ONLY work in FIPS mode - needs more testing31

Thank You!Other resourcesSplunk IRC ( EFNet #splunk )Splunk Answers ( http://answers.splunk.com )Splunk community wiki ( http://wiki.splunk.com )Splunk User Group Slack ( http://splunk402.com/chat/ dle.com/Past (and future!) virtual.conf presentations:http://wiki.splunk.com/Virtual .conf32

Bonus MaterialDeleted Scenes

Be your own Certificate Authority We will use ECC crypto for higher performanceStart out by making a CA Root key and certificate.Very helpful Splunk Blogs post by Jose te-elliptical-curve-certkeys-for-splunk/ You will be prompted for passphrases for multiple keys–Keep them secret–Keep them safe–Use a different passphrase for every key34

Create the CA Root Key & Cert - ECC cd SPLUNK HOME/etc/auth/myOrg splunk cmd openssl ecparam -name "prime256v1" -genkey splunk cmd openssl ec -des3 -out CAroot.keyEnter PEM pass phrase: abc123 Verifying - Enter PEM pass phrase: abc123 splunk cmd openssl req -key CAroot.key -sha1 -subj"/CN Splunk Root CA/O myOrg" -new -x509 -days 3650-set serial 1 -out cacert.crtEnter pass phrase for CAroot.key: abc123 35

Create Splunk Server Key & CSR - ECC splunk cmd openssl ecparam -name "prime256v1" -genkey-out splunk-d.web.key splunk cmd openssl ec -des3 -in splunk-d.web.key-out splunk-d.keyEnter PEM pass phrase: def234 Verifying - Enter PEM pass phrase: def234 splunk cmd openssl req -key splunk-d.key -subj"/CN splunk-d.myorg.com/O myOrg" -new -outsplunk-d.csr36

Sign the Splunk Cert using Root Cert - ECC splunk cmd openssl x509 -req -days 1095 -in splunk-d.csr-CA cacert.crt -CAkey CAroot.key -set serial 02 -outsplunk-d.crtSignature oksubject /CN splunk-d.myorg.com/O myOrgGetting CA Private KeyEnter pass phrase for CAroot.key: abc123 Now we have a keyfile (both encrypted and not) and a cert issued by our CA37

Some other ways to be your own CAActive Directory Certificate ServicesFedora Certificate Serverhttp://pki.fedoraproject.org/wiki/PKI Main PageAlso a part of the FreeIPA suite(Commercially as Red Hat Directory Server)38

Bonus MaterialSplunk heck39

Test connectivity with openssl s clientOpenSSL has a built-in SSL client that you can use to do basic connectivity testing.Works ‘just like TELNET’ but over SSLNo certificate verification by default, but you can get it to dump the presentedcerts so you can check them by hand.It will also dump TLS protocol version and negotiated cipher specification openssl s client -connect 10.10.10.10:8089 -showcertsThe returned certs can be checked in plaintext by copypasting into a file andrunning openssl x509 -text -noout -in xxxx.crt40

Forwarder to LB Indexers - Outputs.conf -1vi SPLUNK HOME/etc/system/local/outputs.conf(or use an app)[tcpout]defaultGroup myIndexers[tcpout:myIndexers]maxQueueSize 128MBuseACK trueautoLB trueserver splunk-idx01.myorg.com:9998, splunk-idx02.myorg.com:9998sslCertPath SPLUNK d REDACTED sslRootCAPath SPLUNK HOME/etc/auth/myOrg/cacert.crt41