Transcription
A peek under the Blue CoatProxySG internalsRaphaël Rigo / AGI / TX5ITBlack Hat Europe - 2015-11-12
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-122
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-123
A peek under the Blue CoatWhat? Why?Blue Coat ProxySG?enterprise (Web) proxyone of the most deployed in big companieslots of complex features:URL categorization (WebSense and others)video streaming / instant messaging specifichandlingMAPI and SMB proxy / cache / prefetcheretc.runs proprietary SGOSWhy research ProxySG?widely used in Airbus Groupinteresting target for malicious actors: log bypass, Internet exposed, MITM, etc.no known previous research: unknown security levelsecurity bulletins: mostly OpenSSL and Web administration interface bugsBlack Hat Europe - 2015-11-124
A peek under the Blue CoatResearchStudy objectives:assess the global security levelwrite recommendations for secure deploymentbe prepared for forensics in case of a compromised ProxySGWhy publish?first public info but surely not first researchfoster research better securityToday’s presentation:raw technical results, as a starting point for researchgoes from low level (FS) to high level, following our approachapplies to all ProxySG models and 6.x versions up to Q1 2015Black Hat Europe - 2015-11-125
A peek under the Blue CoatGetting startedRunning ProxySG:hardware: commodity x86 CPUs, HDD, etc.VMware appliancesCommon versions:5.5: older version, EOL Aug 20146.2: previous long term release, EOL Oct 20156.5: latest long term release, recommended by BCTo get a first look, we need to access the filesystem:6.? ( 6.4): small FAT32 partition containing proprietary BCFS imageolder versions: fully proprietary disk partitionning/data (no FAT32)Black Hat Europe - 2015-11-126
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-127
A peek under the Blue CoatOn disk data: introHardwareBasic architecture: 3 disks (or more)small CompactFlash or SSD for OS (FAT32)2 or more drives for data (proprietary FS)FilesystemsRemarksunknowns:static, read-only FS for OS (BCFS):CEFS structureslog storage formatOS fileslow level (static) configuration: kerneloptions, resource limitson-disk partition structures are verycomplexcache engine FS based on hashtables (CEFS) (Patent US7539818)today: only static FS (BCFS) for OSfilesregistry in CEFS for settingsBlack Hat Europe - 2015-11-128
A peek under the Blue CoatSystem disk organization (BIOS mode)Files on FAT32 partitionbootloader: starter.si6 xt/sgos/fbr.conbasic SGOS (UP kernel, drivers, noapplication)looks up available systemsdisplays GRUB-like boot menuBoth starter.si and system1 use BCFSReal OS: system1210 MiBfull blown OS:SMP kernelWeb UIactual applicationsetc.Black Hat Europe - 2015-11-129
A peek under the Blue CoatBoot sequence (BIOS)1BIOS2MBR3boot sector of active partition4boot.exe, found by hardcoded sector number5kernel.exe, first file entry in starter.si FS6kernel starts sequencer.exe, second entry in starter.si7sequencer.exe parses the main.cfg script and starts the necessary drivers8main.cfg finally launches starter.exe which displays the boot menu9starter.exe loads the selected systemBlack Hat Europe - 2015-11-1210
A peek under the Blue CoatBCFS (read-only FS) formatString TableCP xxxx xxxx HP.crc32 czk.crc32 data.HMAC czk (6.5).HMAC data (6.5)CP xxxx xxxx CZK.data size.nr cpce0x4000CP xxxx xxxx CE.elmnts {.nr .sz}.offset str tableCP xxxx xxxx CE.elmnts {.nr .sz}.offset cpve tableCP xxxx xxxx CE.elmnts {.nr .sz}.offset cpie tableCP xxxx xxxx CEempty.size.size. 0xc00 0xd0 0x40 0x40 0x40.offset.offsetHow to extract?1read CPCE entries,note offsets forstrings table andfiles table2parse files table(CPIE) linearly3get file name fromstrings tableStringsFiles TableCP xxxx xxxx IE.abs off.rel off———–.offset.sizeCP xxxx xxxx IE.abs off.rel off———–.offset.size.How to modify?string tableCPVE table1cannot increase filesize2fix CRC and HMACFiles contentCPIE tableBlack Hat Europe - 2015-11-1211
A peek under the Blue CoatSystem image configuration variables (CPVE)offset and size specified by 3rd CP CE entrymodifying the variable implies fixing CRC/HMAC and rebootvariable names can be found in sequencer.exeStructureKnown variables (section, number : description)struct cpve entry {uint32 t magic1; /* CP */uint64 t unk;uint32 t magic2; /* VE */uint16 t number;uint16 t section;uint32 t unk2;uint64 t value; }Section 4, kernel:4,0: flags:Black Hat Europe - 2015-11-120x8: GDB monitor enabled0x200: int3 at OS startup0x400: kernel debug logs enabled4,1: arch flags1: activate Write Protect in cr04,3: console speed (in bauds)12
A peek under the Blue CoatCache Engine FS (CEFS): writable storagehash-table object storage with disk backendmostly used for cache data:web contentCIFS filesMAPI mailsetc.regular files are also supported, with prefix /legacy/cache engine/Some files (paths straight from the code, no typo)./persistent/replicated/authorized nt/replicated/licensing certificateBlack Hat Europe - 2015-11-1213
A peek under the Blue CoatRegistry: settings storagetree structure used for all settingsentries are referenced by strings like “config:Authenticator:local users”on-disk storage: xml file on writable CEFSURLs (admin rights needed)Interesting CLI extensions (cf slide streg-traceBlack Hat Europe - 2015-11-1214
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-1215
A peek under the Blue CoatOS Filesystem organization/*.cfgvar/[.]/lib/lib(gcc s stdc ) sgos.sohome/jenkins/workspace/SGOS6 sg 6 5 xx7/scorpius/sg 6 5 xx7/bootchain/x86/release/bin/x86 64/sgos native/release/gcc v4.4.2/data filesstripped/libs and programsmp cr/kernel.exestorage/drivers.exeBlack Hat Europe - 2015-11-1216
A peek under the Blue CoatELF files: kernel, libs, programsEverything interesting is located in ./stripped/:.exe, .exe.so and .so extensions (version 5 was using PE files)32 or 64 bits ELF files, depending on model (RAM size?)everything in C , compiled with g with custom sgos targetlots of unit testsmore than 2600 source files referencedeverything is stripped, but lots of external symbolsheavy template use: AMI::Config Data::Config Data(AMI::Storage Class, AMI::String Ref const&, AMI::Shared Ptr AMI::Installed Systems const const&, AMI::Shared Ptr AMI::Config General const const&, AMI::Shared Ptr AMI::Shell const const&, AMI::Shared Ptr AMI::SSL const const&,AMI::Shared Ptr AMI::SMTP Data const const&, AMI::Shared Ptr AMI::BC Threat Protection const const&, AMI::Shared Ptr AMI::Banner Settings const const&,AMI::Shared Ptr AMI::Policy Settings const const&, AMI::Shared Ptr AMI::Statistics Export Settings const const&)“custom” ABI in 32 bits (probably gcc called with -mregparm):EAX, EDX, ECX, stackin 64 bits, standard SysV ABI:RDI, RSI, RDX, RCX, R8, R9, stackBlack Hat Europe - 2015-11-1217
A peek under the Blue CoatKnown code?Interesting open source libraries (version numbers from 6.5 release, Aug 2014):BGET: memory allocator (first dev in 1972!)NET-SNMP 5.4.2.1 (2008-10-31)newlib: libcexpat 1.95.2: XML parser (2001!)libxml2 2.7.7-82143f4 (2010-11-04)OpenSSH 6.3 (2013-09-13)OpenSSL 1.0.1e (2013-02-11)zlib 1.2.3 (2005-07-18)Blue Coat states that they backport fixes regularly (without necessarily changing the version string).Black Hat Europe - 2015-11-1218
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-1219
A peek under the Blue CoatKernelThe kernel in practiceSome syscallskernel access partially abstracted inlibknl api.sosmall ( 800 KiB), basic primitives:NopSuicideEnable event loggingRegister worker addressSymbol addressProcessor voltageSemaphore signal allGrow stackinterrupt/exception handlingsemaphores/locksmessage passingdriversds:1014h points to a “TEB”-likestructureBlack Hat Europe - 2015-11-1220
A peek under the Blue CoatKernel: syscall32 bits64 bitscall dword ptr ds:1018hcall [ds:0FFFFFF8000000020h]parameters in structure pointed by eaxparameters in structure pointed by rdikernel reqstrucfield 0ddreturn codeddreturn code2ddarg0ddarg1ddarg2ddarg3ddsys numddkernel reqendsBlack Hat Europe - 2015-11-12knl req64field 0retcodearg0arg1arg2arg3sysnumfield 38knl req64?21strucdqdqdqdqdqdqdqdqends?
A peek under the Blue CoatMemory organizationBack to the 90sprotected modeeverything in ring 0 (mentioned in US7539818 patent ;)ELF mapping: at boot, once and for allUnpacking executables.Unpacking sequencer.exe elapsed time: 0s, 0ms, 326usUnpacking ata.exe elapsed time: 0s, 0ms, 413us[.]Relocating executables.Relocating sequencer.exe elapsed time: 0s, 2ms, 356usRelocating ata.exe elapsed time: 0s, 0ms, 559us10 executables relocated; total unpack and reloc time 0s, 20ms, 550usBlack Hat Europe - 2015-11-1222
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-1223
A peek under the Blue CoatMaking things easier: our toolsIDA plugins:automatically comment function with source filename (from debug logs)automatically rename functions from debug log stringsautomated syscall recognition (with syscall name, parameters)CLI structures dumper to list all CLI commandsBCFS: FUSE tool to mount system images:file access: read/write (without size change)CPVE access: read/writeautomatic CRC/HMAC calculationTools are internal PoCs and are not going to be released.Black Hat Europe - 2015-11-1224
A peek under the Blue CoatGetting more info: useful tricksEnable debug info, by modifying BCFS (physical access or RCE needed):kernel “printk ”: CPVE 4,0 0x400debug mode: set customer release to 0 in main cr.cfg230 CLI extensions in debug mode:list with “.”, access with “.extension”examples: cfg, policy, cag, mgmt, etc.Example commands.mgmt show-adv-urls.svc ashowstate. ext logaddmask all then . ext logshow.policy dbgtraceonBlack Hat Europe - 2015-11-1225
A peek under the Blue CoatCLI extension exampleBlack Hat Europe - 2015-11-1226
A peek under the Blue CoatGDBKernel includes GDB stub! But finding how to activate it took me weeks :(CPVE 4,0 0x8multiplexed on COM1 with consolesend 0x18, 0x14 on COM port to activate(non-standard) text paging is handled server-side, patch client or use monitorutil height 1000000GDB monitor extensions (kernel side)Some knl extensionsprocesses: display all activeprocesses.Current debug extensions:name knl, Function 0x1261500name util, Function 0x1028786E0name scorpius, Function 0x1028487E0pd: display the contents of a processdescriptor.images: display details of loaded ELFfiles.Black Hat Europe - 2015-11-1227
A peek under the Blue CoatPractical understanding: HTTP parsingGoal: find function for HTTP response parsingactivate HTTP debug mode at https://x.x.x.x:8082/HTTP/debugmake request through proxyget logread interesting function namelook for function in libhttp.exe.soExample log (simplified, most recent 5B72F20: Parse request called. beg 57DE3000 end 57DE30DC length 22095B72F20: Parse request95B72F20: Should tunnel on error95B72F20: Read request95B72F20 POLICY: Evaluating PE POLICY CHECKPOINT NEW CONNECTION95B72F20: Transaction startup95B72F20: Init stateBlack Hat Europe - 2015-11-1228
A peek under the Blue CoatGoing deeper: HellLocating the code is the easy part.Problems:HUGE functions (16 KiB!, see CFG)C everywhereIDA struggles with calling conventionthreads, everywhere!Dynamic debugging howto:find image base using monitor knlimage libhttp.exe.so in GDBrelocate binary in IDAset breakpoint in Proxy SG CLI: conft; debug; breakpoint-set 0 B X ADDR break and connect!Black Hat Europe - 2015-11-1229
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-1230
A peek under the Blue CoatApplication securityauthentication:local passwords are hashed with FreeBSD MD5 crypt ( 1 ), Blowfish supporteddozens of schemes supported: LDAP, AD, etc.default protocols: only HTTPS and SSHread-only or admin accountsOS trust:PKCS7 signed updates (SHA-512/RSA-2048)local images: 6.5: CRC only 6.5: HMAC SHA-1crypto:opensslcritical random data is generated securelyBlack Hat Europe - 2015-11-1231
A peek under the Blue CoatAdministration interface (Flash)Black Hat Europe - 2015-11-1232
A peek under the Blue CoatAdministration interface (Java)Black Hat Europe - 2015-11-1233
A peek under the Blue CoatAdministration interfaceactually POST s CLI commands, in an enable shellrestricted commands for read-only usersJava interface specifics:also uses a kind for RPC mechanism (/Secure/Local/console/pod)also implements its own HTTPS “client”Response dataRequest (simplified)POST /Secure/Local/console/install upload action/cli post setup.txtHost: 10.0.10.3:8082Authorization: Basic YWRtaW46dGVzdA [.]Cookie: bcsi.logout 0; BCSI MC –-7d518638300904Content-Disposition: form-data; name "file"show 7d518638300904–Black Hat Europe - 2015-11-1234ProxySG VA 1818181818#(config)show versionVersion: SGOS 6.4.1.2 MACH5 EditionRelease id: 90192UI Version: 6.4.1.2 Build: 90192Serial number: XXXXXXXXXXNIC 0 MAC: 000FF9B6006FThere were 0 errors and 0 warnings
A peek under the Blue CoatSystem-level securityBADno stack canariesno ASLReverything in ring0kernel callgate at a fixed addressGOODNX enabled on most platforms since 6.2.300/600 support added in 6.5.7.1 and 6.2.16.3BGET heap: asserts check for meta-data coherence (unlink attacks impossible)read-only FS for binaries makes it (way) harder to backdoor OSphysical access (or code exec) is needed to change system image as updatesare signedBlack Hat Europe - 2015-11-1235
A peek under the Blue CoatExploitabilityFacilitating exploitsprevious slide :)vtables everywhereonly C code more memory corruption bugs (vs script/safer languages)Hurdlesno second chance (ring0)no ASLR but mapping different for each versioncustom payload neededguard pagesBlack Hat Europe - 2015-11-1236
A peek under the Blue CoatOutline1Introduction2Storage: filesystems and registry3Binaries4Kernel and OS mechanisms5Understanding internals6Security mechanisms7ConclusionBlack Hat Europe - 2015-11-1237
A peek under the Blue CoatConclusionFindingsunusual, entirely proprietary OS designno user/kernel isolation or exploit hardening (historical for performance?)no vulnerabilities found (I didn’t look for them!). . . . . but Blue Coat release notes document plenty of fixes for “software restarts”Recommendationsuse a dedicated (V)LAN for administrationmonitor the event loginvestigate rebootsphysically protect appliancesuse secure passwords (of course!)Black Hat Europe - 2015-11-1238
A peek under the Blue CoatEvolutions in ProxySGSecurity enhancements in recent versionsNX support for 300/600 added in 6.5.7.1 and 6.2.16.3bootchain and system image validation (hashes published by Blue Coat)Secure boot in pre-release, available in a future releasedebug (GDB, CLI extensions) support removedWe are currently discussing further security enhancements, such as user/supervisorseparation, with Blue Coat. Release timing and platform support are still underdiscussion.Black Hat Europe - 2015-11-1239
A peek under the Blue CoatEndQuestions?Thanks!Stéphane D. for his work on BCFS and the tikz figures :)Stéphane L. and AGI for giving me the opportunity to work on Blue CoatBlack Hat Europe - 2015-11-1240
A peek under the Blue CoatOutline8Backup slidesBlack Hat Europe - 2015-11-1241
A peek under the Blue CoatSystem disk organization (UEFI mode)Files on FAT32 partitionNew: UEFIBOOTx64.EFI replaces /BOOT/BOOTx64.EFI/EFI/BOOT/osloader.siosloader.si contains a copy ofBOOTx64.EFINew: diagLinux diagnostic system:check hardware health.si files use BCFSBlack Hat Europe - 2015-11-12interesting cli binary, with symbols :)42
A peek under the Blue CoatBoot sequence (UEFI)1UEFI2BOOTx64.EFI3desired system is selected4prekernel.exe is started, first file entry in system.si FS5prekernel.exe setups GDT, IDT, etc. and starts kernel.exe (2nd entry)6kernel starts sequencer.exe, (3rd entry)7sequencer.exe parses the main cr.cfg script8main cr.cfg includes main common.cfg which starts everythingWay simpler than BIOS boot.Black Hat Europe - 2015-11-1243
A peek under the Blue Coat Outline 1 Introduction 2 Storage: filesystems and registry 3 Binaries 4 Kernel and OS mechanisms 5 Understanding internals 6 Securit
