Cisco 3850 fails to send dot1x authentications after Denali upgrade

This isn’t a Cisco ISE bug but it could affect ISE deployments. A customer had recently deployed several Cisco 3850s with Multigigabit at their headquarters. Initially, the switches were deployed with IOS XE 3.7.5. We tested the Cisco ISE configuration on those switches (Monitor Mode) and everything worked properly. The customer was able to authenticate users and endpoints using 802.1x and MAB as well as profile connected devices.

The customer ran into an issue with the IOS XE 3.7.5 code that caused link flaps. Another engineer engaged TAC to troubleshoot. When they discovered it was a bug that only affected 3850s with Multigigabit, the recommended fix was to upgrade to Denali (16.3.x). One stack was upgraded to 16.3.5b in order to verify that the link flap issue was resolved. We needed to verify that all authentication functions continued to work properly because of the IOS upgrade.

It was discovered that 802.1x authentication was no longer working. Checking the ISE RADIUS live logs showed that only MAB was coming through. There was no 802.1x authentication request coming from the switch. The switch logs showed the following:

Jan 22 09:22:23 CST: %SESSION_MGR-5-FAIL:Switch 1 R0/0: smd:  Authorization failed or unapplied for client (38C9.860E.E745) on Interface GigabitEthernet1/0/6 AuditSessionID 0A4A05640000060F1E764F7C

This was repeated every time the client disconnected and reconnected. A quick search in Bug Tracker found bug ID CSCvg07470. This bug mostly matched our issue even though MAB was working. The bug was logged against a 3850 running 16.3.3. We opened a TAC case to see if the customer is seeing the same bug in 16.3.5b. It was determined that the customer was in fact hitting that bug. Unfortunately, according to the TAC engineer, there is no current Denali release that corrects this issue.

Because of the link flap issue being resolved in 16.3.5b, the rest of the switches will need to receive the upgrade. The good news is that the customer is only running Cisco ISE in Monitor Mode so no endpoint/user access is affected. The bad news is that the roll out to Low Impact and Closed mode will be delayed until a fix for this new bug is put into place.

Update 2018/01/29: It turns out that the issue I was running into is a configuration issue. The TAC engineer double checked my configuration and found that I was missing “dot1x pae authenticator” from the port config. This was not a required command when the switch was running 3.7.5 and C3PL was enabled but it is required in Denali.

So there you go. Lesson learned. I’ve made sure my configuration templates were updated accordingly.

Share this post:

4 comments

  1. Hi,
    Thank you for the post but still have the issue with Fuji 16.8.1a even if the port is configured with “dot1x pae authenticator”

    interface GigabitEthernet1/0/6
    switchport mode access
    switchport nonegotiate
    switchport voice vlan 200
    ip arp inspection limit rate 20
    authentication host-mode multi-domain
    authentication open
    authentication order dot1x mab
    authentication priority dot1x mab
    authentication port-control auto
    authentication periodic
    authentication timer reauthenticate 65535
    authentication violation protect
    mab
    dot1x pae authenticator
    dot1x timeout quiet-period 5
    dot1x timeout tx-period 3
    storm-control broadcast level 1.00
    storm-control action shutdown
    spanning-tree portfast
    spanning-tree bpduguard enable
    spanning-tree guard root
    ip dhcp snooping limit rate 10

    1. I haven’t had a chance to play around with a switch running 16.8, yet. It looks like you are using the old style port configuration. Are you doing C3PL/ISBN 2.0 or ISBN 1.0? You could try the Denali C3PL configuration template I have listed here.

  2. Hi Brad,
    Thank you for your answer, I will try to select the commands that fit to my scenario.
    The difference here is that I am using NPS, the authentication is working but what’s bother me is that the switch always send the same message (the same one in the post) either for successful authentication or failed one, note that the devices (mab/dot1x) authenticate without problem.
    # sh auth sess int g1/0/3 details
    Interface: GigabitEthernet1/0/3
    IIF-ID: 0x1188C1CC
    MAC Address: 2880.45d8.0e68
    IPv6 Address: Unknown
    IPv4 Address: 192.168.213.114
    User-Name: domain\usertest
    Status: Authorized
    Domain: DATA
    Oper host mode: multi-domain
    Oper control dir: both
    Session timeout: 65535s (local), Remaining: 51020s
    Timeout action: Reauthenticate
    Common Session ID: 0A0E640D000000E8B8BE0E9F
    Acct Session ID: Unknown
    Handle: 0x1f000053
    Current Policy: POLICY_Gi1/0/3
    Server Policies:
    Vlan Group: Vlan: 213
    Method status list:
    Method State
    dot1x Authc Success
    —————————————-
    Interface: GigabitEthernet1/0/3
    IIF-ID: 0x1405DCC6
    MAC Address: f8a5.15c0.271a
    IPv6 Address: Unknown
    IPv4 Address: 192.168.200.45
    User-Name: f8a515c0271a
    Status: Authorized
    Domain: VOICE
    Oper host mode: multi-domain
    Oper control dir: both
    Session timeout: 65535s (local), Remaining: 47177s
    Timeout action: Reauthenticate
    Common Session ID: 0A0E640D000000B2B49B40A1
    Acct Session ID: Unknown
    Handle: 0x6e000033
    Current Policy: POLICY_Gi1/0/3
    Server Policies:
    Vlan Group: Vlan: 200
    Method status list:
    Method State
    dot1x Stopped
    mab Authc Success

Leave a Reply to Network Eng Cancel reply

Your email address will not be published. Required fields are marked *