Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

1.0 Introduction
Purpose
Thepurposeofthispaperisdemonstratehowtomonitorandtroubleshootcommonnetworkbasedapplications
usingstandardUNIXutilitiespacketcaptureutilities.AbasicunderstandingoftheTCP/IPstackandEthernet
hardwareisassumed.
PacketCapture/SniffingtheWire
Promiscuousmodeisgenerallyusedtorefertothepracticeofputtinganetworkcardintoasettingsothatit
passesalltrafficitreceivestotheCPUratherthanjustpacketsaddressedtoit.Manyoperatingsystemsrequire
superuserprivilegestooperateinpromiscuousmode.Anonroutingnodeinpromiscuousmodecangenerally
onlymonitortraffictoandfromothernodeswithinthesamecollisiondomain(forEthernetandWirelessLAN)
orring(forTokenRingorFDDI),whichiswhynetworkswitchesareusedtocombatmalicioususeof
promiscuousmode.Aroutermaymonitoralltrafficthatitroutes.Promiscuousmodeiscommonlyusedto
diagnosenetworkconnectivityissues.SomeprogramslikeEthereal,tcpdump,andAirSnort(forwirelessLANs)
makeuseofthisfeaturetoshowtheuserallthedatabeingtransferredoverthenetwork.Someprogramslike
FTPandTelnettransferdataandpasswordsincleartext,withoutencryption,andnetworkscannerscanseethis
data.Therefore,computerusersareencouragedtostayawayfromprogramsliketelnetandusemoresecure
onessuchasSSH.
2.0 libpcap Based Tools
Thelibpcaplibraryisasystemindependentinterfaceforuserlevelpacketcapture.ManyUNIXutilitiesuse
thelibpcapinterfaceastheirunderlyingpacketcaptureengine.Duetotheportabilityofthiscode,allutilities
thatusethelibpcaplibrarysharethesamesyntax.Themostcommonutilitiesthatusethelibpcaplibraryare
tcpdumpandethereal.
2.1ThetcpdumpUtility
ThetcpdumputilityisthemostcommonpacketcaptureutilityforUNIXbasedsystems.Itishighlyversatile
andfilterable.Hereisastandardrunofthecommand:
box:~#tcpdump
tcpdump:WARNING:eth0:noIPv4addressassigned
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth0,linktypeEN10MB(Ethernet),capturesize96bytes

Thereareafewveryhelpfuloptionsoutthere.Inthefollowingexample,theispecifiesanotherEthernet
interfacebesidesthedefault(eth0).Alsoincludedisthenoptionwhichturnsofhostandportresolution.
ThetcpdumputilitywillattempttoresolveIPaddress,whichcanleadtosignificantdelaysinoutputduetothe
latencyofnetworklookups.Thiscanbeaveryhandyoptionwhenattemptingtomonitorhighvolumesof

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

traffic.
box:~#tcpdumpnieth1
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
19:47:22.935554IP192.168.1.105.32783>67.110.253.165.993:P3432387228:3432387265(37)
ack2742259796win63712<nop,nop,timestamp2396751064682926>
19:47:22.967508IP67.110.253.165.993>192.168.1.105.32783:P1:54(53)ack37win1984
<nop,nop,timestamp1064879093239675>

Hereisabreakdownofasinglepacket:
RealTime:
SourceIPAddress:
DirectionofPacketFlow:
DestinationAddress:
TCPFlags:
TCPSourceSYNNumber:
NextTCPSYNNumber
TCPACKNUMBER:
TCPWindowSize:
TCPOptions:

19:47:22.967508
IP67.110.253.165.993
>
192.168.1.105.32783:
P
1:
54(53)#originalSYN(1)+payload(53)=nextSYN(54)
ack37
win1984
<nop,nop,timestamp1064879093239675>

Ifthevolumeoftraffictobemonitoredistoogreatforastandardterminalwindowbuffer,thecapturedpackets
canbewrittentoafileinsteadofSTDOUTwithaw(write)optionandthenreadbackinwithar(read)
option.
box:~#tcpdumpw/tmp/tcp.outnieth1
tcpdump:listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
46packetscaptured
46packetsreceivedbyfilter
0packetsdroppedbykernel
box:~#tcpdumpr/tmp/tcp.outnieth1
readingfromfile/tmp/tcp.out,linktypeEN10MB(Ethernet)
19:56:07.190888IP192.168.1.105.32783>67.110.253.165.993:P3432387731:3432387768(37)
ack2742260475win63712<nop,nop,timestamp2921001065283060>
19:56:07.227315IP67.110.253.165.993>192.168.1.105.32783:P1:54(53)ack37win1984
<nop,nop,timestamp1065403449292100>
<<snip>>

2.2TheetherealUtility
Justliketcpdump,etherealisbasedonthelibpcapinterface.Therearetwomainversionsofethereal.
ThereisthetextversioncalledtetherealandtheGUIbasedversioncalledethereal.Thetextbased
versionisverysimilarinsyntaxtothetcpdumpcommandsyntax.Onceagain,thisisbecausetheyusethesame

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

underlyinglibpcapengine.
box:~#tetherealw/tmp/ethereal.outnieth1
Capturingoneth1
0.327450192.168.1.105>67.110.253.165TLSApplicationData
0.36117567.110.253.165>192.168.1.105TLSApplicationData
0.361220192.168.1.105>67.110.253.165TCP32783>993[ACK]Seq=37Ack=53Win=63712
Len=0TSV=389797TSER=1066380554
0.363460192.168.1.105>67.110.253.165TLSApplicationData
0.41095167.110.253.165>192.168.1.105TLSApplicationData
box:~#tetherealr/tmp/ethereal.out
62.543822192.168.1.105>67.110.253.165TLSApplicationData
72.59333067.110.253.165>192.168.1.105TLSApplicationData
82.593375192.168.1.105>67.110.253.165TCP32783>imaps[ACK]Seq=37Ack=53
Win=63712Len=0TSV=412045TSER=1066603077
92.595989192.168.1.105>67.110.253.165TLSApplicationData

2.3ThedsniffUtility
Unlikethepreviouslymentionedutilities,dsnifftakespacketcaptureonelevelfurther.Usingtheunderlying
libpcapengine,dsnifftakesthepacketscapturedandattemptstoreportsomethingalittlemoreuseful.The
dsniffprogramisoneofmanyutilitiesinthedsniffpackage.Thestandarddsniffcommandwillattemptto
captureandreplayallunencryptedsessionsincluding:FTP,telnet,SMTP,IMAP,andPOP.Thefollowing
exampledemonstrateshowtousedsnifftoaudittelnetandftpsessions:
box:~#dsniffnieth1
dsniff:listeningoneth1

06/01/0520:35:46tcp192.168.1.105.32883>192.168.1.220.21(ftp)
USERdarren
PASSdarren$$$$

06/01/0520:37:53tcp192.168.1.105.32889>192.168.1.220.23(telnet)
darren
darren$$$$
ls
lsl
psef
exit

2.4ThesnortUtility
Thesnortutilityisthemostcommonopensourceintrusiondetectionsystem.Likedsniff,itattemptstomake
senseoutoftraffic.Whereassimplyreportscleartextpayloads,snortattemptstoidentifymalicioustraffic
patternsusingsignaturesofknownmaliciouspackets.Likealltheotherutilities,snortuseslibpcapasthe
underlyingengine.Thefollowingisaverybasicrunofsnort.

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

box:~#snortieth1Ddusnortgsnortc/etc/snort/snort.conf

Thesnortutilitywillrunindaemonmodeinthebackground.Allalertsarewrittentoatextfile
/var/log/snort/alert.Hereisasamplescanfromapossibleintruder:
[root@targus~]#nmapp22sX192.168.1.105

Thisquestionableportscanofthesshport(22)isobfuscatedbythesX(Christmasscan)switch.
Hereistheoutputfromthealertfileonthehostrunningsnort:
box:~#tailf/var/log/snort/alert
[**][1:1228:7]SCANnmapXMAS[**]
[Classification:AttemptedInformationLeak][Priority:2]
06/0120:42:02.813099192.168.1.220:37325>192.168.1.105:22
TCPTTL:47TOS:0x0ID:15194IpLen:20DgmLen:40
**U*P**FSeq:0x82A00B2Ack:0x0Win:0x1000TcpLen:20UrgPtr:0x0
[Xref=>http://www.whitehats.com/info/IDS30]

3.0 Sun Solarissnoop Utility

SnoopisspecifictoSunMicrosystem'sSolarisUNIX.AlthoughthereisaportoftcpdumpforSolaris,thereis
noportofsnoopforLINUX.Muchliketcpdump,itisautilitythatputsyoursystemsinterface(s)in
promiscuousmode.Althoughsimilarindesigngoals,sincesnoopusesit'sownpacketcapturelibrary,the

optionsarealittledifferent.Thefollowingexampledemonstrateshowtorunsnooponanalternateinterface
(d)withnameresolutiondisabled(r).
pilate>snooprdhme0
Usingdevice/dev/hme(promiscuousmode)
66.27.208.74>67.110.253.164TCPD=22S=32897Ack=1172114638Seq=1267082364Len=0
Win=11200Options=<nop,nop,tstamp663391727479722>
67.110.253.164>66.27.208.74TCPD=32897S=22PushAck=1267082364Seq=1172114638
Len=208Win=47824Options=<nop,nop,tstamp727479725663391>

Likelibpcaputilities,snoopalsoenablestheredirectionofpacketstoafileinsteadofSTDOUT.
pilate>snoopo/tmp/snoop.outrdhme0
Usingdevice/dev/hme(promiscuousmode)
14
pilate>snoopi/tmp/snoop.out
10.00000cpe662720874.socal.res.rr.com>pilateTCPD=22S=32897
Ack=1172118318Seq=1267084668Len=0Win=15600Options=<nop,nop,tstamp670581727486912>
20.00005pilate>cpe662720874.socal.res.rr.comTCPD=32897S=22Push
Ack=1267084668Seq=1172118318Len=112Win=47824Options=<nop,nop,tstamp727486915670581>

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

4.0 Using Filter Expressions

Itmaybeeasytoidentifyspecifictrafficstreamsonsmalloridlenetworks.Itwillbemuchharderto
accomplishthisonlargeWANorsaturatednetworks.Theabilitytousefilterexpressionsisextremely
importantinthesecasestocutoutunwantednoisepacketsfromthetrafficinquestion.Fortunately,boththe
libpcapbasedutilitiesandthesnooputilityallusethesamefiltersyntax.Therearemanywaystofiltertraffic
inallutilities,themostcommonfiltersarebyport,protocol,andhost.Thefollowingexampletracksonly
telnettrafficandhost192.168.1.105:
[root@targus~]#tcpdumpnieth0port23andhost192.168.1.105
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth0,linktypeEN10MB(Ethernet),capturesize96bytes
21:13:26.905262IP192.168.1.105.32899>192.168.1.220.telnet:S1903904803:1903904803(0)
win5840<mss1460,sackOK,timestamp7226130,nop,wscale0>
[root@targus~]#tetherealnieth0port23andhost192.168.1.105
Capturingoneth0
0.000000192.168.1.105>192.168.1.220TCP32900>23[SYN]Seq=0Ack=0Win=5840Len=0
MSS=1460TSV=729689TSER=0WS=0

box:~#dsniffnieth1port23andhost192.168.1.105
dsniff:listeningoneth1[port23andhost192.168.1.105]

06/01/0521:11:16tcp192.168.1.105.32901>192.168.1.220.23(telnet)
root
cangetin
pilate>snooprdhme0port53andhost192.168.1.105

Thereareothercaseswhereanadministratormaywanttocaptureallbutcertaintypesoftraffic.Specifically,a
lotofnoisecanbemadeifoneistryingtorunapacketcapturewhileloggedintotheremotehost.Muchof
thetrafficgeneratedwillbethecontroltrafficbacktothathost.Thefollowingexampleshowshowtofilterthe
sshcontroltraffictoandfromthecontrolconnection(192.168.1.105connectedto192.168.1.220asroot)and
allDNStraffic.
[root@targus~]#who
rootpts/2Jun121:30(192.168.1.105)
[root@targus~]#tcpdumpnieth0nothost192.168.1.105andnotport53
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth0,linktypeEN10MB(Ethernet),capturesize96bytes
21:32:07.692524IP216.93.214.50.51606>192.168.1.220.ssh:S2550704294:2550704294(0)
win5840<mss1460,sackOK,timestamp4316081200,nop,wscale2>
21:32:07.692596IP192.168.1.220.ssh>216.93.214.50.51606:S729994889:729994889(0)ack
2550704295win5792<mss1460,sackOK,timestamp111008380431608120,nop,wscale2>
21:32:07.796911IP216.93.214.50.51606>192.168.1.220.ssh:.ack1win1460
<nop,nop,timestamp431608221111008380>

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

5.0 Protocol Layer Problems

5.1.0ARPLayerProblemsNoARPReply
Thepingcommandisoftenusedtotestwhetherornotaremotehosthasaconfigurednetworkstack.Whena
pinghangsornoreplyisreceived,thereisanassumptionthatthehostisnotuporworking.Thismaybetrue,
however,therecouldbemultipleissuesattheARPlayerpreventingaworkinghostfromcommunicatingonthe
network.ThefollowingexampledemonstrateswhatanexpectedARPexchangelookslikeBEFOREaping
commandcancommence.ItdemonstratestheARPREQUESTbythesourcehostfollowedbytheARPREPLY
fromthedestinationhost.
box:~#ping192.168.1.102
PING192.168.1.102(192.168.1.102)56(84)bytesofdata.
64bytesfrom192.168.1.102:icmp_seq=1ttl=128time=5.83ms
box:~#tcpdumpnieth1arp
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
21:43:55.459856arpwhohas192.168.1.102tell192.168.1.105
21:43:55.462638arpreply192.168.1.102isat00:0f:1f:17:ab:a7

Hereisanexampleofafailedpingattempt.NoticethattheARPREQUESTwasneveransweredbythe
destinationhost(192.168.1.107).Thiscouldmeanthatthedestinationhostisnotonline.
box:~#ping192.168.1.107
PING192.168.1.107(192.168.1.107)56(84)bytesofdata.
From192.168.1.105icmp_seq=1DestinationHostUnreachable
From192.168.1.105icmp_seq=2DestinationHostUnreachable
box:~#tcpdumpnieth1arp
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
21:36:48.995766arpwhohas192.168.1.107tell192.168.1.105
21:36:49.992668arpwhohas192.168.1.107tell192.168.1.105
21:36:50.992667arpwhohas192.168.1.107tell192.168.1.105

5.1.1ARPLayerProblemsDuplicateIPAddresses
WhentwohostshavethesameIPAddressassigned,therewillbetwoARPREPLYtotheARPREQUEST.
Thefirstreplyentersthesourcehost'sARPtable.Theproblemisthatthefirstreplymaybethewronghost.The
followingexampledemonstrateshowaWindowsXPandLINUXhostcompeteforthesameIPaddress.
Onthesurface,thepingmakesitappearthatthedestinationhost(expectedtobeLINUX)isupanresponding.
box:~#ping192.168.1.102

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

PING192.168.1.102(192.168.1.102)56(84)bytesofdata.
64bytesfrom192.168.1.102:icmp_seq=1ttl=128time=5.83ms

However,acaptureofARPtrafficshowsthattworepliesweresenttotheoriginalrequest.Thefirstreplywas
fromtheWindowshost.TheLINUXhost'sentrycameafter.
box:~#tcpdumpnieth1arp
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
22:04:35.074216arpwhohas192.168.1.102tell192.168.1.105
22:04:35.078448arpreply192.168.1.102isat00:40:f4:83:48:24#WindowsHost
22:04:35.079562arpreply192.168.1.102isat00:0f:1f:17:ab:a7#LINUXHost

AcheckoftheARPcacheshowsthattheWindowsXPEthernetaddressistheonepopulatedinthecache.
box:~#arpa
targus(192.168.1.220)at00:02:55:74:41:1B[ether]oneth1
?(192.168.1.1)at00:06:25:77:63:8B[ether]oneth1
?(192.168.1.102)at00:40:f4:83:48:24[ether]oneth1

Anattempttousesshtoconnecttotheremotehostfailsbecausethesourcehostisattemptingtoconnecttothe
WindowsXPhostinsteadoftheLINUXhostusingtheWindowsXPEthernetaddress.
box:~#sshv192.168.1.102
OpenSSH_3.8.1p1Debian8.sarge.4,OpenSSL0.9.7e25Oct2004
debug1:Readingconfigurationdata/etc/ssh/ssh_config
debug1:Connectingto192.168.1.102[192.168.1.102]port22.
^C

5.2.0IPProblemsMisconfiguredBroadcastAddress
Thebroadcastaddressisoftenoverlookedwhendeterminingnetworkproblems.Somenetworkcommunications
relyonaproperlyconfiguredbroadcastaddressincluding:NTP,RIPv1,andICMPBroadcastpings.The
followingexampledemonstratesastandardICMPbroadcastpinganassociatedreply.
box:~#ifconfigeth1
eth1Linkencap:EthernetHWaddr00:06:53:E4:8D:B8
inetaddr:192.168.1.105Bcast:192.168.1.255Mask:255.255.255.0
box:~#pingb192.168.1.255
WARNING:pingingbroadcastaddress
PING192.168.1.255(192.168.1.255)56(84)bytesofdata.
64bytesfrom192.168.1.105:icmp_seq=1ttl=64time=0.052ms
64bytesfrom192.168.1.1:icmp_seq=1ttl=150time=3.54ms(DUP!)
64bytesfrom192.168.1.220:icmp_seq=1ttl=64time=4.43ms(DUP!)

Ifthebroadcastaddressisnotconsistentwiththerestofthehostsonthenetwork,noneofthosehostswillreply
tothebroadcastping.Inthefollowingexample,thecorrectsubnetmask255.255.255.0.However,thesource
hosthasamisconfigurednetmaskof255.0.0.0.

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

box:~#ifconfigeth1
eth1Linkencap:EthernetHWaddr00:06:53:E4:8D:B8
inetaddr:192.168.1.105Bcast:192.255.255.255Mask:255.0.0.0

Abroadcastpingreturnsnorepliesfromthenetwork.
box:~#pingb192.168.1.255
PING192.168.1.255(192.168.1.255)56(84)bytesofdata.
From192.168.1.105icmp_seq=1DestinationHostUnreachable

Apacketcapturefurtherconfirmstheproblem.Thecorrectbroadcastis192.168.1.255,howeversincethe
sourcehost'sbroadcastis192.255.255.255,thesourcehostismistakinglytryingtoARPfor192.168.1.255,
thinkingitisarealhost.ThesourcehostwillneverreceiveavalidARPreplyasnohostonthenetworkcan
havea.255init'slastoctet.

box:~#tcpdumpnieth1
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
22:31:01.512750arpwhohas192.168.1.255tell192.168.1.105
22:31:02.512665arpwhohas192.168.1.255tell192.168.1.105

5.2.1IPProblemsMisconfiguredDefaultGateway
WhenaclienthostonaLANcan'tcommunicatewiththeoutsideworld,itcanbeoneof4issues:
nonetworkconnectivity
misconfigured/etc/nsswitch.conf
misconfiguredornonexistent/etc/resolv.confforDNS
misconfiguredorwronggatewayinformation
Thefirst3issuescanbesolvedbyviewingfilesandcheckingphysicallinks.Thereisnorealwaytotellifthe
gatewayentryistrulyroutingpackets.Thefollowingexampledemonstrateshowtomonitorwhetherthe
gatewayisroutingpackets.
TheclienthostisunabletoreachahostontheInternet.
box:~#pingyahoo.com
ping:unknownhostyahoo.com

Theclienthasagatewayconfiguredintheroutingtable.However,thereisnowaytotellwhetherthegatewayis
actuallyrouting.
box:~#netstatrn
KernelIProutingtable
DestinationGatewayGenmaskFlagsMSSWindowirttIface
192.168.1.00.0.0.0255.255.255.0U000eth1
0.0.0.0192.168.1.2200.0.0.0UG000eth1

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

Thefollowingpacketcaptureistakenfromtherouter.Thepacketarecomingfromthesourcehostof
192.168.1.105,buttheinterfaceisNOTshowingthereturnpacket.
[root@targus~]#tcpdumpnieth0notport22
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth0,linktypeEN10MB(Ethernet),capturesize96bytes
09:01:39.063347IP192.168.1.105.32770>4.2.2.2.domain:49762+A?yahoo.com.(27)
09:01:44.075062IP192.168.1.105.32771>4.2.2.1.domain:49762+A?yahoo.com.(27)

Fromtheclientprospective,therouterisonthenetworkasitrepliestoapingrequest.
box:~#ping192.168.1.220
PING192.168.1.220(192.168.1.220)56(84)bytesofdata.
64bytesfrom192.168.1.220:icmp_seq=1ttl=64time=2.69ms
64bytesfrom192.168.1.220:icmp_seq=2ttl=64time=2.95ms

However,theclientisnotreceivinganyrepliesfromit'sDNSrequest.Packetsaregoingtotherouter,however
theyaregettingdropped.
box:~#tcpdumpnieth1notport22
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
09:10:06.367238IP192.168.1.105.32772>4.2.2.2.53:49099+A?yahoo.com.(27)
09:10:11.381598IP192.168.1.105.32773>4.2.2.1.53:49099+A?yahoo.com.(27)

5.2.0TCPProblemsClosedPorts
Therearemultiplereasonswhyaremoteservermaynotrespondtoaclientrequest.Exampleshavealready
beengiventotroubleshootattheEthernet,ARP,andIPlevels.Acommonmistakeistoassumethatsinceahost
isavailableattheIPlevel,itdoesnotmeanthatitisavailableattheTCPlevel.
ThehostisavailableattheIPlevelasperthepingreplies.
box:~#ping192.168.1.220
PING192.168.1.220(192.168.1.220)56(84)bytesofdata.
64bytesfrom192.168.1.220:icmp_seq=1ttl=64time=2.71ms
64bytesfrom192.168.1.220:icmp_seq=2ttl=64time=2.64ms

Thetelnetserviceisnotavailabletotheclient.
box:~#telnet192.168.1.220
Trying192.168.1.220...
telnet:Unabletoconnecttoremotehost:Connectionrefused

2005 Darren Hoch - webmaster [at] litemail.org

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

Takingalookatthepacketcapture,itisclearthatthetelnetserviceisnotrunning.StandardTCPrepliesto
closedportsistosendaresetflagtothesourcehost.
box:~#tcpdumpnieth1port23
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
09:18:12.495679IP192.168.1.105.32780>192.168.1.220.23:S2557844136:2557844136(0)win
5840<mss1460,sackOK,timestamp16439310,nop,wscale0>
09:18:12.498346IP192.168.1.220.23>192.168.1.105.32780:R0:0(0)ack2557844137win0

5.2.1TCPProblemsTCPWrappedServices
TCPWrappershavebeenaroundforquitesometime.Theirpurposeistodohostbasedaccesscontrol.Unlike
closedports,theporttotheserverisopen.Uponconnectiontotheserver,acheckismadetothe
/etc/hosts.allowand/etc/hosts.denyfiles.Iftheclientisallowedtoconnect,thenastandardTCP
connectionismade.Ifnot,theserversendsaTCPresettotheclient.ThefollowingisanexampleofaTCP
wrappedsshservice.
box:~#ssh192.168.1.220
ssh_exchange_identification:Connectionclosedbyremotehost

Alookatthepacketcaptureshowsthatupondeterminingthattheclientisdenied,thesshserverinitiatesa
standardportclosingthroughaseriesofFINpacketexchanges.
box:~#tcpdumpnieth1port22
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
10:24:44.508208IP192.168.1.105.32786>192.168.1.220.22:S2304868053:2304868053(0)win
5840<mss1460,sackOK,timestamp20431320,nop,wscale0>
10:24:44.510770IP192.168.1.220.22>192.168.1.105.32786:S762146323:762146323(0)ack
2304868054win5792<mss1460,sackOK,timestamp444552762043132,nop,wscale2>
10:24:44.510807IP192.168.1.105.32786>192.168.1.220.22:.ack1win5840
<nop,nop,timestamp204313244455276>
10:24:49.526296IP192.168.1.220.22>192.168.1.105.32786:F1:1(0)ack1win1448
<nop,nop,timestamp444602922043132>
10:24:49.526647IP192.168.1.105.32786>192.168.1.220.22:F1:1(0)ack2win5840
<nop,nop,timestamp204363444460292>
10:24:49.529124IP192.168.1.220.22>192.168.1.105.32786:.ack2win1448
<nop,nop,timestamp444602952043634>

5.2.2TCPProblemsPacketFilteredTCPPorts
Whenaportisblockedbyapacketfilter(IPTablesorIPFilterforexample),itmaybeopenbutfilteredattheIP
level.Inthiscase,theclientwillsendmultipleSYNpacketstotheserverandtheserverwillnotrespondsimply
becausethepackethasbeendroppedbythefilter.

2005 Darren Hoch - webmaster [at] litemail.org

10

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

box:~#telnet192.168.1.220
Trying192.168.1.220...
telnet:Unabletoconnecttoremotehost:Noroutetohost

Alookatthepacketcaptureshowsthattheclientsent3TCPSYNpacketstothetelnetport(23)thatwere
simplydroppedbytheserverandnotrepliedtointheclient.
box:~#tcpdumpnieth1port23
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth1,linktypeEN10MB(Ethernet),capturesize96bytes
10:35:02.302120IP192.168.1.105.32787>192.168.1.220.23:S2934062463:2934062463(0)win
5840<mss1460,sackOK,timestamp21049120,nop,wscale0>
10:35:13.847729IP192.168.1.105.32788>192.168.1.220.23:S2951328215:2951328215(0)win
5840<mss1460,sackOK,timestamp21060660,nop,wscale0>
10:35:19.518837IP192.168.1.105.32789>192.168.1.220.23:S2947681121:2947681121(0)win
5840<mss1460,sackOK,timestamp21066330,nop,wscale0>

6.0 Application Layer Problems

6.1.0DHCPProblemsIPAddressalreadyassigned
TheDHCPprotocolisalargelytransparentprotocolonthenetwork.Thefollowingpacketcaptureshowsthe
piecesthatcompriseaDHCPexchangebetweenaclientandaDHCPserver.
[root@targusdhcp]#tetherealnieth1notport22
4.4718020.0.0.0>255.255.255.255DHCPDHCPDiscoverTransactionID0x803fcf50

AftertheinitialDHCPDISCOVERmadebytheclient,theserversendsoutanARPrequesttothenetworktoseeif
anyotherhosthastakentheIPAddressfromtheDHCPpool.
4.47573800:06:25:77:63:8b>ff:ff:ff:ff:ff:ffARPWhohas192.168.1.104?Tell
192.168.1.1

IftherearenoARPreplies,thentheDHCPserverofferstheIPaddresstotheclient.Theclientacceptsthelease
andtheDHCPserveracknowledgesthattheleaseisnowassigned.
5.135171192.168.1.1>255.255.255.255DHCPDHCPOfferTransactionID0x803fcf50
5.1354710.0.0.0>255.255.255.255DHCPDHCPRequestTransactionID0x803fcf50
5.139041192.168.1.1>255.255.255.255DHCPDHCPACKTransactionID0x803fcf50

Fromthispointon,aclientwillcontinuetoaskforwhateveraddressitpreviouslyhad.Iftheclientdisconnects
fromthenetworkandthentriestoreconnect,theoldIPaddresscouldeasilybetakenbyanotherclient.Inthe
followingpacketcapture,theclientrequestsit'soldIPaddressof192.168.1.104.
[root@targusdhcp]#tetherealnieth1notport22
Capturingoneth1

2005 Darren Hoch - webmaster [at] litemail.org

11

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

4.4629230.0.0.0>255.255.255.255DHCPDHCPRequestTransactionID0x6ccda4f

TheDHCPserversentoutanARPrequestandreceivedavalidARPreplyfromanotherhost.
4.46573700:06:25:77:63:8b>ff:ff:ff:ff:ff:ffARPWhohas192.168.1.104?Tell
192.168.1.1
4.46576200:c3:61:f9:42:a7>00:06:25:77:63:8bARP192.168.1.104isat00:c3:61:f9:42:a7
4.467191192.168.1.1>192.168.1.104ICMP[MalformedPacket]

Therefore,theoldrequestedaddressof192.168.1.104isnolongeravailableandtheDHCPserversendsa
NAK(noacknowledgment)packetbacktotheclient,causingtheclienttostartatthebeginningwithaDHCP
DISCOVER.
4.471480192.168.1.1>255.255.255.255DHCPDHCPNAKTransactionID0x6ccda4f

6.1.1DHCPProblemsUsingDHCPforManualNetworkConfiguration
IftherearehardwareorsoftwareversionissuesthatpreventaclientfromnegotiatingaleasefromaDHCP
server,thenaNICcardcanbeconfiguredmanuallywithallthenetworkinginformationfromaDHCPACK
messagefromaDHCPserver.ThefollowingpacketcapturedisplaysverboseoutputofaDHCPACK.Thispacket
includesallinformationneededtosuccessfullyconfigureaNICmanually.
[root@targus~]#ifconfigeth1up
[root@targus~]#ifupeth1
[root@targus~]#tetherealnVieth1portbootpc
Capturingoneth1
<<snip>>
Your(client)IPaddress:192.168.1.100(192.168.1.100)
NextserverIPaddress:192.168.1.1(192.168.1.1)
RelayagentIPaddress:0.0.0.0(0.0.0.0)
ClientMACaddress:00:40:f4:83:48:24(CameoCom_83:48:24)
Serverhostnamenotgiven
Bootfilenamenotgiven
Magiccookie:(OK)
Option53:DHCPMessageType=DHCPACK
Option1:SubnetMask=255.255.255.0
Option3:Router=192.168.1.1
Option6:DomainNameServer
IPAddress:66.75.160.62
IPAddress:66.75.160.41
IPAddress:66.75.160.37
Option15:DomainName="socal.rr.com"
Option51:IPAddressLeaseTime=1day
Option54:ServerIdentifier=192.168.1.1
EndOption
Padding

2005 Darren Hoch - webmaster [at] litemail.org

12

Network Troubleshooting Using Packet Capture Utilities UUASC 6/2/2005

6.1.2SambaProblemsUnabletoConnecttoSambaServer
TheSambaprotocol,oftenusedtonetworkWindowsclientstoUNIXservers,islargelyabroadcastprotocol.
WhenfailingtoconnectaWindowsbasedclienttoaUNIXserver,thereisnousefuldebugginginformation
availableontheWindowshost.ThereisanerrormessagethatsimplystatestheNetworkPathNotFoundor
somethingsimilar.Thefollowingpacketcaptureshowsthattheclientandservercan'tcommunicateoverSamba
duetothedifferencesinbroadcastaddresses.Theclient(192.168.1.102)isattemptingtodiscovertheserver
(192.168.1.220),however,bothhavedifferentbroadcastdomainsandthereforecan'thereeachother'srequests.
[root@targustmp]#tcpdumpnieth0notport22andnothost192.168.1.1
tcpdump:verboseoutputsuppressed,usevorvvforfullprotocoldecode
listeningoneth0,linktypeEN10MB(Ethernet),capturesize96bytes
13:51:58.319893IP192.168.1.102.netbiosns>192.168.1.255.netbiosns:NBTUDPPACKET
(137):QUERY;REQUEST;BROADCAST
13:51:59.069690IP192.168.1.102.netbiosns>192.168.1.255.netbiosns:NBTUDPPACKET
(137):QUERY;REQUEST;BROADCAST
13:51:59.820093IP192.168.1.102.netbiosns>192.168.1.255.netbiosns:NBTUDPPACKET
(137):QUERY;REQUEST;BROADCAST
13:52:02.114031IP192.168.1.220.netbiosns>192.168.255.255.netbiosns:NBTUDPPACKET
(137):QUERY;REQUEST;BROADCAST
13:52:04.113961IP192.168.1.220.netbiosns>192.168.255.255.netbiosns:NBTUDPPACKET
(137):QUERY;REQUEST;BROADCAST
References
Thedsniffprojecthttp://www.monkey.org/~dugsong/dsniff/
Thesnooputilityhttp://www.spitzner.net/snoop.html
Thetcpdump/libpcapprojecthttp://www.tcpdump.org/
Thesnortprojecthttp://www.snort.org
Theetherealprojecthttp://www.ethereal.com/
DHCPProtocolhttp://www.faqs.org/rfcs/rfc1541.html
SMBProtocolhttp://www.faqs.org/rfcs/rfc1001.html

2005 Darren Hoch - webmaster [at] litemail.org

13