WIXLIB

Shooting the trouble down to theWireshark Lua PluginJune 2019Shakthi KannanVersion 0.9shakthi@aerospike.com

MotivationHello Production Support Engineer,We are seeing client timeouts in our cluster,Can you analyze the logs on the server,And let us know how to proceed further?”input from client unsupported proto version”Is the message that we see for our operation,What in the client can cause this assertion?We need your help to provide us with a solution.We are making batch read queries in production,And receiving the values in a timely fashion,Aerospike uses RIPEMD160 hash function,But, can we also read the digests in conjunction?2

Wireshark Lua DissectorsDecode packet data. Chained DissectorsAccess to one dissector’s data. Post-dissectorsCalled after every other dissectorhas been called. ListenersCalled for every packet thatmatches a filter or tap.Source: https://wiki.wireshark.org/Lua/Dissectors3

Tap Listener-- simple http.lua-- implements a very simple tap in Lua-- this is going to be our counterhttp packets 0-- this is going to be our taptap http nil-- first we declare the tap called “http tap” with the filter it is going to usetap http Listener.new(nil, “http”)-- this function will get called at the end(3) of the capture to print the summaryfunction tap http.draw()debug(“http packets:” . http packets)end-- this function is going to be called once each time the filter of the tap matchesfunction tap http.packet()http packets http packets 1end-- this function will be called at the end of the capture runfunction tap http.reset()http packets 0end4

Wireshark User InterfaceMain MenuMain lDissectionPacketBytes5

UsageHelp - About WiresharkFree (Libre)Open Source wireshark -X lua script:aerospike.lua capture.pcapng tshark-X lua script:aerospike.lua capture.pcapngYou can also place plugins in /.wireshark/plugins folder.6

Hello World Lua! lua -vLua 5.3.4Copyright (C) 1994-2017 Lua.org, PUC-Rio luaLua 5.3.4Copyright (C) 1994-2017 Lua.org, PUC-Rio print(“Hello, World!”)Hello World cat hello world.lua#!/usr/bin/luaprint(“Hello, World!”) lua hello world.luaHello, World!7

Lua: Assignment and Operations luaLua 5.3.4Copyright (C) 1994-2017 Lua.org, PUC-Rio i, j 1, 2CategoryOperatorAssociativityUnarynot # -Right to leftConcatenation.Right to leftMultiplicative*/%Left to rightAdditive -Left to rightRelational Left to right k % jEquality Left to right1Logical ANDandLeft to rightLogical ORorLeft to right -3-3 k i j k3 j k8.0 not ifalse8

Lua: Strings name “Lua”Use type(name)EscapeSequencestring\aBell print(name . “ Language”)\bBackspaceLua Language\fForm feed print(“99” 1)\nNew line100.0\rCarriage return\tTab\vVertical tab\\Backslash [[ This is also a string ]]\”Double quotesThis is also a string\[Left squarebracket\]Right squarebracket print(“Value of k”, k)Value of k 3 print(“Lua’s \”syntax\” is simple!”)Lua’s “syntax” is simple!9

Lua: Tables work days {“Mon”, “Tue”, “Wed”, “Thu”, “Fri”} work daystable: 0x9f10b0 work days[0]1Monnil2Tue work days[1]3WedMon4Thu5Fri6Sat7Sun work days[6] “Sat” table.insert(work days, “Sun”) work days[7]Sun10table: 0x9f10b0

Lua: Functions-- fact.luafunction fact (n)local f 1module clockmodule httpmodule yamlmodule stringfor i 2, n dof f * iendreturn fendprint(fact(5)) lua fact.lua12011

Lua: Conditions and Loopsif number 10 thenprint(“Less than 10”)elseprint(“Greater than 10”)repeatprint(number)number number 1until number 10endwhile number 10 doprint(number)number number 1end12for number 0, 9, 1 doprint(number)end

Literate Programming"I believe that the time is ripe for significantly betterdocumentation of programs, and that we can best achieve thisby considering programs to be works of literature. Hence, mytitle: “Literate Programming.”Let us change our traditional attitude to the construction ofprograms: Instead of imagining that our main task is toinstruct a computer what to do, let us concentrate rather onexplaining to human beings what we want a computer to do." Prof. Donald E. Knuth, 198413

Markdown Structure# Requires.# Configuration.## Common.# Helper Functions.# Statistics.## Hot Key.# GUI.# Protocols.## Info.## Batch.## Message.### Aerospike Message: Header Section.### Aerospike Message: Fields.### Aerospike Message: Operations.### Functions.## Heartbeat.# The Main.14

lit2lua## HeartbeatHeartbeat protocol -------------------------- ---------------------------- Message Header Message Fields -------------------------- ---------------------------- Message Header ------------ ------------- size type ------------ ------------- 046Constantslocallocallocallocal15HB HEADER SZ STARTHB HEADER SZ LENGTHHB HEADER TYPE STARTHB HEADER TYPE LENGTH 0442

lit2lua .Op Value Name Description ------: :--------------------- ----------- 1 AS MSG OP READ Read the value 2 AS MSG OP WRITE Write the value 3 AS MSG OP CDT READ Prospective CDT top-level ops 4 AS MSG OP CDT MODIFY Prospective CDT top-level ops 5 AS MSG OP INCR Add a value to an existing value (only on integers) 6 Unused Reserved 7 Unused Reserved 8 Unused Reserved 9 AS MSG OP APPEND Append a value to an existing value (on strings and blobs ) 10 AS MSG OP PREPEND Prepend a value to an existing value (on strings and blobs) 11 AS MSG OP TOUCH Touch a value (will only increment the generation) 129 AS MSG OP MC INCR Memcache-compatible version of the increment command 130 AS MSG OP MC APPEND Append value to existing value (only on strings) 131 AS MSG OP MC PREPEND Prepend a value to an existing value (only on strings) 132 AS MSG OP MC TOUCH Memcache-compatible touch (does not change generation) To generate the Wireshark Lua plugin from documentation, use: less -f “docs/aerospike.lua.md” lit2lua lua/aerospike.lua16

lit2lua .Table Definitionlocal TYPES OPS {[1] “AS MSG OP READ”,[2] “AS MSG OP WRITE”,[3] “AS MSG AP CDT READ”,[4] “AS MSG OP CDT MODIFY”,[5] “AS MSG OP INCR”,[6] “Unused”,[7] “Unused”,[8] “Unused”,[9] “AS MSG OP APPEND”,[10] “AS MSG OP PREPEND”,[11] “AS MSG OP TOUCH”,[129] “AS MSG OP MC INCR”,[130] “AS MSG OP MC APPEND”,[131] “AS MSG OP MC PREPEND”,[132] “AS MSG OP MC TOUCH”,}17

Protocol Dissection PatternConstantslocal PROTO VERSION START 0local PROTO VERSION LENGTH 1local PROTO TYPE START 1local PROTO TYPE LENGTH 1local PROTO TYPE INFO 1local PROTO TYPE MSG 3local INFO SIZE START 2local INFO SIZE LENGTH 6local INFO DATA START 8 18 --------- ------------- ---------------------------------- version type size --------- ------------- ---------------------------------- 0128

Protocol Dissection Pattern .Create Proto objectslocal aerospike info proto Proto(“Aerospike”,“Aerospike Info Protocol”)local aerospike attribute Proto(“AerospikeAttribute”,“Aerospike Attributes”)local aerospike attribute value Proto(“AerospikeAttributeValue”, “Aerospike Attribute Value pairs”)Proto header fieldslocal header fields {version ProtoField.uint8 (“header.version”, “Version”, base.DEC),type ProtoField.uint8 (“header.type”,“Type”,base.DEC, TYPES PROTO),size ProtoField.uint64 (“header.size”,“Size”,base.DEC),}local header attributes {attribute ProtoField.string(“header.attribute”, “Attribute”),}local header attribute values {attribute ProtoField.string(“header attribute values.attribute”,value ProtoField.string(“header attribute values.value”,}Register the protocol fieldsaeropike info proto.fields header fieldsaerospike attribute.fields header attributesaerospike attribute value.fields header attribute values19“Attribute “),“Value”),

Protocol Dissection Pattern .Functionslocal function dissect aerospike info (tvbuf, tree, size)-- Separate the data by newlinelocal data tvbr tvbuf:range(INFO DATA START, tonumber(size))local data string data tvbr:string()local data start INFO DATA STARTfor line in string.gmatch(data string, "[ \n] ") dolocal d tvbuf:range(data start, string.len(line))local d string d:string()-- if contains attribute-valuesif string.find(d string, "\t") thenlocal parts split tab(d string).enddata start data start string.len(line) 1 -- for \nendend20

Dissector Table# Configurationlocal default settings {aerospike port 3000,heartbeat multicast port 9918,heartbeat mesh port 3002,}# Create Proto objectslocal aerospike proto Proto(“AerospikeProtocol”, “Aerospike Protocol”)local heartbeat proto Proto(“AerospikeHeartbeat”, “Aerospike Heartbeat”)# The Mainlocal function enable default settings.aerospike port, aerospike lt settings.heartbeat mesh port, heartbeat lt settings.heartbeat multicast port, heartbeat proto)endenable dissector()21

Live Coding!Source: https://www.xkcd.com/378/22

Message ProtocolProductionSupportBit-levelDissection Value Name Description ------: :---------------------------------- --------------------------------- 1 AS MSG INFO1 READ Contains a read operation 2 AS MSG INFO1 GET ALL Get all bins data 4 Unused Unused 8 AS MSG INFO1 BATCH New batch protocol 16 AS MSG INFO1 XDR Operation is performed by XDR 32 AS MSG INFO1 GET NO BINS Do not read the bin information 64 AS MSG INFO1 CONSISTENCY LEVEL B0 Read consistency level – bit 0 128 AS MSG INFO1 CONSISTENCY LEVEL B1 Read consistency level – bit 1 23

Heartbeat ProtocolHeartbeat protocol ------------------ ---------------- Message Header Message Fields ------------------ ---------------- Message HeaderNetworkAnalysis ------------ ------------- size type ------------ ------------- 046Header Type Value Name ------: :-------------------- 0 M TYPE FABRIC 1 M TYPE HEARTBEAT V2 2 M TYPE PAXOS 3 M TYPE MIGRATE 4 M TYPE PROXY 5 M TYPE HEARTBEAT 6 M TYPE CLUSTERING 7 M TYPE RW 8 M TYPE INFO 9 M TYPE EXCHANGE 11 M TYPE XDR 15 M TYPE SMD 24

CDT List OperationsModifying list of lists‘values’ bin:[[1523474230000, 39.04],[1523474231001, 39.78],[1523474236006, 40.07],[1523474235005, 41.18],[1523474233003, 40.89],[1523474234004, 40.93]]client.list append(key,‘values’,[1623474234004, 41.23])'values' bin:[[1523474230000, 39.04],[1523474231001, 39.78],[1523474236006, 40.07],[1523474235005, 41.18],[1523474233003, 40.89],[1523474234004, 40.93],[1623474234004, 41.23]]Source: html25

Reassembled TCP SegmentsAnomalyDetectiondissect tcp pdus(tvb, tree, min header size, get len func, dissect func)Source: https://www.wireshark.org/docs/wsdg html chunked/lua module Proto.html26

Hot Key ReportReportGeneration27

Tests: Executionaeropike-wireshark-plugin/tests make clean; onse.pcapng.pdml . [OK]test-msg-write-request.pcapng.pdml . [OK]test-batch.pcapng.pdml . .pdml . [OK]test-heartbeat-mesh.pcapng.pdml . pdml . [OK]test-msg-read-response.pcapng.pdml . [OK]test-msg-read-request.pcapng.pdml . [OK]test-heartbeat-multicast.pcapng.pdml . [OK]test-info-response.pcapng.pdml . [OK]test-info-request.pcapng.pdml . [OK]Generate report using luacov-console .luacov-console ./lualuacov-console -s # --no-colored Summary FileHits Missed lua/aerospike.lua 634 tal634 4892.96%28

DebuggingFunctionDescriptioncritical(text)Critical severitywarn(text)Warninglocal d require ‘debug’message(text)Normalprint ug(text)Debuggingreport failure(text)Message box witherror iconUtility velopment Tipshttps://wiki.wireshark.org/Development/Tips29

Luacheck Accessing undefined variable Line contains only whitespace Setting non-standard global variable Unused variable Unused argument Unused loop variable i Unused function Line is too long Trailing whitespace in a comment luarocks install luacheck luacheck lua/aerospike.luaTotal: 0 warnings / 0 errors in 1 file30