Troubleshooting Palo Alto Firewalls

Introduction

There are many reasons that a packet may not get through a firewall. After all, a firewall’s job is to restrict which packets are allowed, and which are not.

But sometimes a packet that should be allowed does not get through. So after you do your basic troubleshooting (creating test rules, turning off inspections, packet captures), and still can’t get the packet through, you might find that you’re stuck.

So, in this article, we’ll look at the next level of troubleshooting that you can do – Mostly from the command line.

 

Interfaces

We can check the interface counters for a few things:

  • Are packets being dropped on this interface? See the packets dropped and packets dropped by flow state counters
  • Is traffic being received at all? See bytes/packets received
  • Is traffic being sent out? See bytes/packets transmitted
  • Are there interface errors? See receive/forwarding errors
  • Is there a problem with ARP? See the arp not found counter
  • Is there a security issue? See all the remaining counters
 
admin@Firewall(active)> show counter interface ae1.18
Interface: ae1.18
--------------------------------------------------------------------------------
Logical interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 1029392992089
bytes transmitted 1492332031720
packets received 2546880536
packets transmitted 2700000916
receive errors 0
packets dropped 3689663
packets dropped by flow state check 319
forwarding errors 0
no route 7
arp not found 0
neighbor not found 0
neighbor info pending 0
mac not found 0
packets routed to different zone 0
land attacks 0
ping-of-death attacks 0
teardrop attacks 0
ip spoof attacks 0
mac spoof attacks 0
ICMP fragment 0
layer2 encapsulated packets 0
layer2 decapsulated packets 0
tcp cps 37
udp cps 4
sctp cps 0
other cps 0
--------------------------------------------------------------------------------

 

Route Lookups

Is there a valid entry in the forwarding table to reach your destination? Use the test routing command.

admin@firewall(active)> test routing fib-lookup ip 1.1.1.3 virtual-router default
--------------------------------------------------------------------------------
runtime route lookup
--------------------------------------------------------------------------------
virtual-router: default
destination: 1.1.1.3
result:
via 192.0.2.2 interface ae1.17, source 192.0.2.1, metric 6543
--------------------------------------------------------------------------------

 

Drop Counters

Drop counters is where it gets really interesting. The Palo Alto firewall will keep a count of all drops and what causes them, which we can access with show counter global filter severity drop.

We can then see the different drop types (such as flow_policy_deny for packets that were dropped by a security rule), and see how many packets were dropped.

admin@Firewall(active)> show counter global filter severity drop
Global counters:
Elapsed time since last sampling: 166.47 seconds name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_rcv_err 353 0 drop flow parse Packets dropped: flow stage receive error
flow_rcv_dot1q_tag_err 195 0 drop flow parse Packets dropped: 802.1q tag not configured
flow_no_interface 195 0 drop flow parse Packets dropped: invalid interface
flow_policy_nofwd 212405 0 drop flow session Session setup: no destination zone from forwarding
flow_policy_deny 15184139 5 drop flow session Session setup: denied by policy
flow_policy_nat_land 8829 0 drop flow session Session setup: source NAT IP allocation result in LAND attack
flow_tcp_non_syn_drop 5321357 2 drop flow session Packets dropped: non-SYN TCP without session match
flow_fwd_l3_mcast_drop 1490885 1 drop flow forward Packets dropped: no route for IP multicast
flow_icmp_err_not_passing_thru 1 0 drop flow ipsec ICMP error packet dropped: no IP configured on the interface
flow_fwd_l3_ttl_zero 50426 0 drop flow forward Packets dropped: IP TTL reaches zero
flow_fwd_l3_noarp 40390733 19 drop flow forward Packets dropped: no ARP
flow_fwd_zonechange 1503 0 drop flow forward Packets dropped: forwarded to different zone
flow_parse_l4_hdr 1 0 drop flow parse Packets dropped: TCP (UDP/ICMP/SCTP) packet too short
flow_parse_l4_cksm 41723 0 drop flow parse Packets dropped: TCP/UDP checksum failure
flow_parse_l4_port 352 0 drop flow parse Packets dropped: illegal TCP/UDP port 0
flow_parse_iperror 6 0 drop flow parse Packets dropped: invalid IP address
flow_xmt_platform_encap_err 17 0 drop flow offload Packets dropped: Platform encapsulation error
flow_bind_nack_msg_drop 2 0 drop flow pktproc gtp-u bind-nack msg dropped
flow_predict_convert_policy_deny 18 0 drop flow pktproc A matching predict was not used because of policy denial
flow_action_close 438602 0 drop flow pktproc TCP sessions closed via injecting RST
flow_action_reset 382 0 drop flow pktproc TCP clients reset via responding RST
flow_arp_rcv_err 1340 0 drop flow arp ARP receive error
flow_host_rcv_err 6 0 drop flow mgmt Packets dropped: receive error from control plane
flow_host_decap_err 85 0 drop flow mgmt Packets dropped: decapsulation error from control plane
flow_host_service_deny 306456 0 drop flow mgmt Device management session denied
flow_host_service_unknown 500272 0 drop flow mgmt Session discarded: unknown application to control plane
flow_tunnel_decap_err 27 0 drop flow tunnel Packet dropped: tunnel decapsulation error
flow_tunnel_ipsec_replay_err 21 0 drop flow tunnel Packet dropped: header sequence number is a replay
flow_tunnel_ipsec_wrong_spi 4 0 drop flow tunnel Packet dropped: IPsec SA for spi in packet not found
flow_tunnel_natt_nomatch 13 0 drop flow tunnel Packet dropped: IPSec NATT packet without SPI match
flow_host_slowpath_drop 1053987 0 drop flow tunnel ESP/AH host bound packet comes before tunnel finishes installation
flow_gre_tunnel_decap_not_found 39 0 drop flow tunnel GRE Tunnel IPs don't match configuration
flow_fpga_rcv_err 359 0 drop flow offload Packets dropped: receive error from offload processor
flow_fpga_ingress_exception_err 9502477 2 drop flow offload Packets dropped: receive ingress exception error from offload processor
flow_fpga_egress_exception_err 578 0 drop flow offload Packets dropped: receive egress exception error from offload processor
appid_lookup_invalid_flow 3 0 drop appid pktproc Packets dropped: invalid session state
url_request_pkt_drop 173484 0 drop url pktproc The number of packets get dropped because of waiting for url category request
--------------------------------------------------------------------------------
Total counters shown: 37
--------------------------------------------------------------------------------

 

Of course, we’ll need to filter this information a bit. We can add more than one filter to the command. severity drop is the filter we used in the previous command.

Add delta yes as an additional filter to see the drop counters since the last time that you ran the command.

This makes it easier to see if counters are increasing.

admin@Firewall(active)> show counter global filter severity drop delta yes
Global counters:
Elapsed time since last sampling: 89.376 seconds name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_policy_deny 513 5 drop flow session Session setup: denied by policy
flow_tcp_non_syn_drop 240 2 drop flow session Packets dropped: non-SYN TCP without session match
flow_fwd_l3_mcast_drop 104 1 drop flow forward Packets dropped: no route for IP multicast
flow_fwd_l3_ttl_zero 8 0 drop flow forward Packets dropped: IP TTL reaches zero
flow_fwd_l3_noarp 1950 21 drop flow forward Packets dropped: no ARP
flow_action_close 32 0 drop flow pktproc TCP sessions closed via injecting RST
flow_host_service_deny 24 0 drop flow mgmt Device management session denied
flow_host_service_unknown 11 0 drop flow mgmt Session discarded: unknown application to control plane
flow_fpga_ingress_exception_err 205 2 drop flow offload Packets dropped: receive ingress exception error from offload processor
url_request_pkt_drop 54 0 drop url pktproc The number of packets get dropped because of waiting for url category request
--------------------------------------------------------------------------------
Total counters shown: 10
--------------------------------------------------------------------------------

 

To filter it further, you can configure a packet filter in the GUI (under packet captures), and filter based on packet-filter yes.

This will narrow it down to only traffic we’re interested in.

admin@Firewall(active)> show counter global filter severity drop packet-filter yes
Global counters:
Elapsed time since last sampling: 10.385 seconds name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_rcv_dot1q_tag_err 182 0 drop flow parse Packets dropped: 802.1q tag not configured
flow_no_interface 182 0 drop flow parse Packets dropped: invalid interface
flow_policy_nofwd 155402 0 drop flow session Session setup: no destination zone from forwarding
flow_policy_deny 4912229 0 drop flow session Session setup: denied by policy
flow_policy_nat_land 3565 0 drop flow session Session setup: source NAT IP allocation result in LAND attack
flow_tcp_non_syn_drop 2234727 0 drop flow session Packets dropped: non-SYN TCP without session match
flow_fwd_l3_mcast_drop 27693 0 drop flow forward Packets dropped: no route for IP multicast
flow_fwd_l3_ttl_zero 17151 0 drop flow forward Packets dropped: IP TTL reaches zero
flow_fwd_l3_noarp 12261869 0 drop flow forward Packets dropped: no ARP
flow_fwd_zonechange 1270 0 drop flow forward Packets dropped: forwarded to different zone
flow_xmt_platform_encap_err 17 0 drop flow offload Packets dropped: Platform encapsulation error
flow_action_close 151655 0 drop flow pktproc TCP sessions closed via injecting RST
flow_action_reset 229 0 drop flow pktproc TCP clients reset via responding RST
flow_host_rcv_err 6 0 drop flow mgmt Packets dropped: receive error from control plane
flow_host_decap_err 66 0 drop flow mgmt Packets dropped: decapsulation error from control plane
flow_host_service_deny 52052 0 drop flow mgmt Device management session denied
flow_host_service_unknown 162114 0 drop flow mgmt Session discarded: unknown application to control plane
flow_tunnel_decap_err 11 0 drop flow tunnel Packet dropped: tunnel decapsulation error
flow_tunnel_ipsec_replay_err 10 0 drop flow tunnel Packet dropped: header sequence number is a replay
flow_tunnel_ipsec_wrong_spi 1 0 drop flow tunnel Packet dropped: IPsec SA for spi in packet not found
flow_tunnel_natt_nomatch 5 0 drop flow tunnel Packet dropped: IPSec NATT packet without SPI match
flow_host_slowpath_drop 1053987 0 drop flow tunnel ESP/AH host bound packet comes before tunnel finishes installation
flow_gre_tunnel_decap_not_found 12 0 drop flow tunnel GRE Tunnel IPs don't match configuration
url_request_pkt_drop 51336 0 drop url pktproc The number of packets get dropped because of waiting for url category request
--------------------------------------------------------------------------------
Total counters shown: 24
--------------------------------------------------------------------------------

 

Clearing Sessions

Sometimes sessions can get stuck open for some reason, and won’t be evaluated by firewall rules or packet captures.

You can look for open sessions with show session all and then filter by destination IP address. We can use source, destination, or both.

In this example, we can see three RDP sessions open:

-----
admin@Firewall(active)> show session all filter destination 10.16.8.31
--------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
2015202 ms-rdp ACTIVE FLOW 10.16.201.251[9075]/VPN/6 (10.16.201.251[9075])
vsys1 10.16.8.31[3389]/LAN (10.16.8.31[3389])
2374790 ms-rdp ACTIVE FLOW 10.16.201.251[9076]/VPN/6 (10.16.201.251[9076])
vsys1 10.16.8.31[3389]/LAN (10.16.8.31[3389])
2041822 ms-rdp ACTIVE FLOW 10.16.201.251[58834]/VPN/17 (10.16.201.251[58834])
vsys1 10.16.8.31[3389]/LAN (10.16.8.31[3389])

 

We can then look at more detail if we want to. If NAT were used, we could also check which NAT rules is being hit.

This shows us the Client-to-server (c2s) side of the flow, and the Server-to-Client (s2c) side.

admin@Firewall(active)> show session id 2015202
Session 2015202
c2s flow:
source: 10.16.201.251 [VPN]
dst: 10.16.8.31
proto: 6
sport: 9075 dport: 3389
state: INIT type: FLOW
src user: networkdirection\admin
dst user: networkdirection\admin
s2c flow:
source: 10.16.8.31 [LAN]
dst: 10.16.201.251
proto: 6
sport: 3389 dport: 9075
state: INIT type: FLOW
src user: networkdirection\admin
dst user: networkdirection\admin
start time : Mon Nov 23 10:44:27 2020
timeout : 15 sec
total byte count(c2s) : 1772
total byte count(s2c) : 1688
layer7 packet count(c2s) : 8
layer7 packet count(s2c) : 6
vsys : vsys1
application : ms-rdp
rule : VPN Access
service timeout override(index) : False
session to be logged at end : False
session in session ager : False
session updated by HA peer : False
layer7 processing : enabled
URL filtering enabled : True
URL category : any
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : False
session terminate tunnel : False
captive portal session : False
ingress interface : ae1.34
egress interface : ae1.18
session QoS rule : N/A (class 4)
tracker stage firewall : TCP RST - client
tracker stage l7proc : ctd proc changed
end-reason : tcp-rst-from-client

 

And finally, we can clear the session if needed:

admin@firewall(active)> clear session id 2015202
session 2015202 cleared

 

 

References

Palo Alto KB – How to Troubleshoot Using Counters via the CLI

Palo Alto KB – Packet Drop Counters in Show Interface Ethernet … Display

Palo Alto KB – Packets Dropped: Forwarded to a Different Zone