Nstrace
Nstrace is a NetScaler packet capture tool. Nstrace dumps packets in the native NetScaler format. These trace files have an extension of .cap and can be analysed with WireShark. You can use specific filters in WireShark as normal to filter through captured data or specify filters using the NetScaler CLI. This allows you to only capture traffic of interest.
start nstrace – Captures all traffic.
stop nstrace – Stops the packet capture.
show nstrace – Shows the status of nstrace and if it is running etc.
Some classic nstrace expressions:
start nstrace -filter “vsvrname == LDAP-LoadBalanced-vServer” – Captures traffic to and from the specified vServer.
start nstrace -size 0 -filter “svcname == xx” – Captures traffic to and from the specified service. -size 0 means all packets are captured regardless of packet size.
start nstrace -filter “DESTIP == 192.168.0.242” -link ENABLE – Captures all traffic to destination IP 192.168.0.242. The -link ENABLE switch is used to capture return traffic from 192.168.0.242.
start nstrace -filter “DESTIP == x.x.x.x” – Captures all traffic to destination IP.
start nstrace -filter “SRCIP == x.x.x.x” – Captures traffic sent from source IP.
start nstrace -filter “DESTPORT == 443” – Captures traffic where the destination port is 443.
start nstrace -filter “SRCIP == x.x.x.x && DESTPORT == xx” – Captures traffic from a specified source IP and a specified destination port.
Some default commands/expressions:
Note: These expressions are the same onces you can use to filter traffic when running nstrace from the GUI. I prefer to use these expressions.
start nstrace -filter CONNECTION.SRCIP.EQ(192.168.0.100) -link ENABLED – Captures all traffic to and from 192.168.0.100.
start nstrace -filter CONNECTION.SRCIP.NE(192.168.0.100) – Captures all source traffic apart from when the source is 192.168.0.100.
start nstrace -filter “CONNECTION.SRCIP.EQ(192.168.0.100) || CONNECTION.DSTPORT.EQ(443)” – Captures all traffic from 192.168.0.100 with the destination port as 443.
start nstrace -filter “CONNECTION.DSTIP.EQ(192.168.0.15) || CONNECTION.DSTIP.EQ(192.168.0.16) || CONNECTION.DSTIP.EQ(192.168.0.17)” -link ENABLED – Captures traffic to and from three IP addresses.
start nstrace -filter CONNECTION.INTF.EQ(“0/1”) – Captures all traffic flowing in and out of interface 0/1.
start nstrace -filter CONNECTION.VLANID.EQ.(205) – Captures all traffic flowing in and out of VLAN 205.
start nstrace -filter CONNECTION.DSTPORT.BETWEEN(80,180) – Captures traffic when the destination port is between 80 and 180.
start nstrace -filter CONNECTION.LB_VSERVER.NAME.EQ(“name”) – Captures traffic to the specified Load Balanced Virtual Server.
start nstrace -filter CONNECTION.SERVICE_TYPE.EQ(SSL) – Captures all SSL traffic.
start nstrace -filter CONNECTION.SERVICE_TYPE.EQ(MONITOR_PING) – Captures all ping requests sent by a monitor including replies.
start nstrace -size 0 -nf 10 -time 120 – Starts an nstrace using circular logging. A maximum of 10 separate logs will be generates, each new log is generated every 120 seconds.
Nstcpdump
Nstcpdump can be used for more low-level troubleshooting. Nstcpdump does not collect as much detailed information as nstrace. Open NetScaler CLI and type shell. You can use filters with nstcpdump but cannot use filters specific to NetScaler resources. The dump output can be viewed directly within the CLI screen.
CTRL + C – Press these keys simultaneously to stop an nstcpdump.
nstcpdump.sh dst host x.x.x.x – Shows traffic sent to the destination host .
nstcpdump.sh -n src host x.x.x.x – Shows traffic from specified host and don’t convert IP addresses to names (-n).
nstcpdump.sh host x.x.x.x – Shows traffic to and from specified host IP.
nstcpdump.sh -c 10 dst host 192.168.0.242 – Outputs the first 10 packets from destination 192.168.0.242.
nstcpdump.sh dst host 192.168.0.242 and port 443 – Outputs traffic destined for 192.168.0.242 on port 443.
nstcpdump.sh src host x.x.x.x and port xx – Outputs traffic from specified source IP and specified port .
nstcpdump.sh -e host x.x.x.x and host x.x.x.x – Shows traffic between two specified host IPs, and prints the MAC address (-e) of the interface sending traffic..
nstcpdump.sh icmp – Outputs all sent and replied echo requests. NetScaler uses ICMP to track up/down status of DNS servers configured on NetScaler for example.
Note: Pings to DNS come from the SNIP, using static routes to force these pings through the NSIP for example will not work.
nstcpdump.sh icmp and dst host x.x.x.x – Outputs all sent echo requests to a specified destination host.
nstcpdump.sh icmp and dst host x.x.x.x -w /var/nstrace/output.cap – Outputs all sent echo request to a specified destination host and saves the output in the specified cap file.
nstcpdump.sh udp and not port 3003 and not port 1985 – Outputs all UDP traffic but excludes traffic on port 3003 and 1985.
nstcpdump.sh port 53 | grep 192.168.1.1 – Outputs all traffic using port 53 from IP 192.168.1.1.
Andreas Fischer
January 4, 2017Hi George! This page is very useful!
I have a question regarding nstrace: why the SSLPLAIN-mode in NetScaler 11.1 is hidden? I read the Citrix documentation, unfortunately version 11.0, where it was described, but I didn’t found it on my 11.1-machine (VPX on an SDX 11500), neither in manual nor in command short help:
NSVPX1_Primary#> start nstrace -nf 21 -time 120 -size 0 -mode
APPFW
C2C
IPV6
MPTCP
NEW_RX
NS_FR_TX
RX
TX
TXB
NSVPX1_Primary#>
So I tried it nevertheless – and it’s functioning!! What does that mean? Will Citrix remove the SSLPLAIN-option in future or is it part of an other option?
Thanks in advance for an answer and greetings from Dresden!
Andreas
George Spiers
January 5, 2017Hi Andreas
Yes, as you noticed SSLPLAIN still works as a command line switch. It is still also available from the GUI under System -> Diagnostics -> Start new trace, and it has been removed from 11.1 documentation.
I don’t know for sure if it will be removed in the future however there is a different (newer) method to decrypt SSL. A new switch was added to late NetScaler v11.0.66.x and above versions. It is called -capsslkeys. So a command such as start nstrace -capsslkeys ENABLED would capture a trace file, and a second file named nstrace.sslkeys. You can import the session keys file in to WireShark to decrypt the captured file.
Brandon
January 12, 2018NetScaler – VIPS/GW Monitors going UP/DOWN
Hi George,
At times at least once or twice a week we see our vips/gslb/monitors flaps from one location (it varies on the site). But any recommendation how to troubleshoot this? We lose our vdi’s when it flaps and sometimes we don’t even notice. But if you notice below it shows Packets dropped in Rx (sw). Anything that stands out? Once it comes back it’s fine and can’t reproduce this issue at all.
> stat interface 1/1
Interface [1/1]:
Interface State UP
Link uptime 6.14:14:43
Link downtime 00:00:11
Throughput Statistics
Rate (/s) Total
Bytes received 435 1966651435
Bytes transmitted 0 21144
Packets received 5 18391487
Packets transmitted 0 368
Jumbo Packets Received 0 0
Jumbo Packets Transmitted 0 0
Tagged Packets Received on Trunk 0 0
Tagged Packets transmitted on Tr 0 0
Packet Statistics
Rate (/s) Total
Multicast packets 1 861758
NetScaler packets 4 17113835
LACP Statistics
Rate (/s) Total
LACPDUs received 0 0
LACPDUs transmitted 0 0
Error Statistics
Rate (/s) Total
Error packets received (hw) 0 0
Error packets transmitted (hw) 0 0
Inbound packets discarded(hw) 0 0
Outbound packets discarded(hw) 0 0
Packets dropped in Rx (sw) 1 658685
Packets dropped in Tx (sw) 0 0
NIC hangs — 0
Status stalls — 0
Transmit stalls — 0
Receive stalls — 0
Error-disables — 0
Duplex mismatches — 0
Link re-initializations — 0
MAC moves registered 0 0
Times NIC became muted — 0
Receive CRC errors 0 0
Done
> stat interface 0/1
Interface [0/1]:
Interface State UP
Link uptime 6.14:15:25
Link downtime 00:00:11
Throughput Statistics
Rate (/s) Total
Bytes received 689763 469703365313
Bytes transmitted 30212 21854551192
Packets received 510 349807958
Packets transmitted 330 251322623
Jumbo Packets Received 0 0
Jumbo Packets Transmitted 0 0
Tagged Packets Received on Trunk 0 0
Tagged Packets transmitted on Tr 0 0
Packet Statistics
Rate (/s) Total
Multicast packets 1 850560
NetScaler packets 509 348938342
LACP Statistics
Rate (/s) Total
LACPDUs received 0 0
LACPDUs transmitted 0 0
Error Statistics
Rate (/s) Total
Error packets received (hw) 0 0
Error packets transmitted (hw) 0 0
Inbound packets discarded(hw) 0 0
Outbound packets discarded(hw) 0 0
Packets dropped in Rx (sw) 0 299686
Packets dropped in Tx (sw) 0 0
NIC hangs — 0
Status stalls — 0
Transmit stalls — 0
Receive stalls — 0
Error-disables — 0
Duplex mismatches — 0
Link re-initializations — 0
MAC moves registered 0 0
Times NIC became muted — 0
Receive CRC errors 0 0
Done
> stat ns | grep Megabits
Megabits received 7 3847569
Megabits transmitted 0 223267
> stat ns | grep Megabits
Megabits received 8 3847622
Megabits transmitted 0 223270
> stat ns | grep Megabits
Megabits received 8 3847622
Megabits transmitted 0 223270
> stat ns | grep Megabits
Megabits received 8 3847622
Megabits transmitted 0 223270
> stat ns | grep Megabits
Megabits received 8 3847622
Megabits transmitted 0 223270
> stat ns | grep Megabits
Megabits received 8 3847622
Megabits transmitted 0 223270
> stat ns | grep Megabits
Megabits received 8 3847622
Megabits transmitted 0 223270
> stat ns | grep Megabits
Megabits received 5 3847656
Megabits transmitted 0 223272
> stat ns | grep Megabits
Megabits received 5 3847656
Megabits transmitted 0 223272
> stat ns | grep Megabits
Megabits received 9 3847716
Megabits transmitted 0 223274
> stat ns | grep Megabits
Megabits received 5 3848922
Megabits transmitted 0 223333
> stat ns | grep Megabits
Megabits received 5 3848922
Megabits transmitted 0 223333
> stat interface 0/1
Interface [0/1]:
Interface State UP
Link uptime 6.14:28:36
Link downtime 00:00:11
Throughput Statistics
Rate (/s) Total
Bytes received 1108259 470353531998
Bytes transmitted 36337 21880328394
Packets received 797 350283270
Packets transmitted 486 251628450
Jumbo Packets Received 0 0
Jumbo Packets Transmitted 0 0
Tagged Packets Received on Trunk 0 0
Tagged Packets transmitted on Tr 0 0
Packet Statistics
Rate (/s) Total
Multicast packets 1 851741
NetScaler packets 796 349412446
LACP Statistics
Rate (/s) Total
LACPDUs received 0 0
LACPDUs transmitted 0 0
Error Statistics
Rate (/s) Total
Error packets received (hw) 0 0
Error packets transmitted (hw) 0 0
Inbound packets discarded(hw) 0 0
Outbound packets discarded(hw) 0 0
Packets dropped in Rx (sw) 0 300103
Packets dropped in Tx (sw) 0 0
NIC hangs — 0
Status stalls — 0
Transmit stalls — 0
Receive stalls — 0
Error-disables — 0
Duplex mismatches — 0
Link re-initializations — 0
MAC moves registered 0 0
Times NIC became muted — 0
Receive CRC errors 0 0
Done
> > show interface
^^^^^^^^^^^
ERROR: No such command
> shell
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
root@# show interface
> show interfaces
1) Interface 0/1 (NetScaler Virtual Interface, VMXNET3) #0
flags=0xc060
MTU=1500, native vlan=1, MAC=00:50:56:be:40:c5, uptime 158h41m54s
LLDP Mode: NONE, LR Priority: 1024
RX: Pkts(350757538) Bytes(471004503134) Errs(0) Drops(300522) Stalls(0)
TX: Pkts(251929029) Bytes(21905417051) Errs(0) Drops(0) Stalls(0)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
Bandwidth thresholds are not set.
2) Interface 1/1 (NetScaler Virtual Interface, VMXNET3) #1
flags=0xc060
MTU=1500, native vlan=249, MAC=00:50:56:be:a6:6d, uptime 158h41m54s
LLDP Mode: NONE, LR Priority: 1024
RX: Pkts(18403708) Bytes(1967730206) Errs(0) Drops(660641) Stalls(0)
TX: Pkts(369) Bytes(21204) Errs(0) Drops(0) Stalls(0)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
Bandwidth thresholds are not set.
3) Interface LO/1 (Netscaler Loopback interface) #2
flags=0x20008020
MTU=1500, native vlan=1, MAC=00:50:56:be:40:c5, uptime 158h42m05s
LLDP Mode: NONE, LR Priority: 1024
RX: Pkts(53362344) Bytes(8051270186) Errs(0) Drops(0) Stalls(0)
TX: Pkts(434220711) Bytes(476838601061) Errs(0) Drops(0) Stalls(0)
Bandwidth thresholds are not set.
Done
> show ver
NetScaler NS12.0: Build 56.20.nc, Date: Dec 19 2017, 06:00:47
Done
>
> show ver
NetScaler NS12.0: Build 56.20.nc, Date: Dec 19 2017, 06:00:47
George Spiers
January 16, 2018What VPX license are you running and how many vCPUs have the VPX(s) been assigned? Is there any consistency to the drops for example they happen at a certain time of the week and do all VIPs drop? Are you running HA and if so make sure failovers are not happening at this time? What is vLAN 249 for, is that where all VIP and SNIPs reside? You have no stalls, mutes, hangs and no sign of NetScaler crashing and packets dropped in Rx does not look abnormal as this can indicate NetScaler dropping packets not destined for any of its owned MAC addresses, if operating in L3 mode.
Brandon
January 17, 2018George,
We are running Enterprise license 25 MB =VPX
Not running HA
2 two sites and gslb enabled
Running 2gig’s of RAM and 2 CPU (1 core per socket)
249 is where all the vips live and vdi subnet live as well
snip is on 247
should i upgrade the VM’s memory and cpu?
Brandon
January 17, 2018Hardware version 7 (ESXI)
George Spiers
January 18, 2018Should be OK unless you are experiencing high traffic at certain times. You could monitor the appliances with NetScaler MAS which can send you alerts if something is wrong such as high CPU during the time when you are experiencing packet loss. Also, if there is a trend to when this happens set up an nstrace at the time to see if you can find what is happening to the packets, or log on to the GUI of NetScaler and look at CPU/Memory consumption. It is also worth checking other appliances at this time such as firewalls in between NetScaler.
You should also generate a tech support file by running “show techsupport” from the CLI and uploading that to cis.citrix.com
Petr Hrubý
July 20, 2018It looks like the nstcpdump.sh runs /netscaler/nsapimgr which captures and saves all (!) packets to /var/nstrace/nstcpdump directory. This temporary file is then used as a source file for standard tcpdump which produces target output. Therefore even if you specify complex capturing filter, the filesystem is full very quickly …
nstcpdump.sh -w scaneng1.cap dst host 10.5.50.76 and ‘tcp[tcpflags] & (tcp-syn) != 0’
reading from file -, link-type EN10MB (Ethernet)
/var: write failed, filesystem is full
Petr Hrubý
July 20, 2018Btw I think that -size 0 means that the whole packet size is captured. If you specify a number, the packet is truncated to the specific size specified by the number. But all packets are always captured.