How I perform Cisco ISE deployment upgrades

Cisco has their way (ISE 2.4 upgrade guide) of performing an ISE deployment upgrade using the CLI or GUI. Here is the way I’ve been doing them since 1.x and I’ve had a lot of success. There is no need to restore backups from previous versions (unless something bad happens but that’s different). For this example, let’s assume we have the following deployment:

  • 2 physical servers (35×5) running the PAN and MNT personas
  • 2 physical servers (35×5) running as PSNs
  • All nodes running Cisco ISE 2.3 to be upgraded to 2.6
  • All nodes have the CIMC connection
    • Use this in order to mount the ISO in a remote KVM session.

The steps I follow are always the same.

  1. Follow all the steps found in the Cisco guide to prepare for the upgrade.
    • This is very important. The URT will save you a lot of headache from a failed upgrade.
    • The certificate and private keys are vital for the next steps when the nodes are reloaded (not upgraded).
    • Make a note of the services/portals the SSL certificates are assigned to so you can assign them again later.
    • Make a note how the node is configured (Administration > System > Deployment > [Node]). Especially how the Profiling Configuration is set up if this is a PSN. You’ll need this later.
  2. Prepare the upgrade on the secondary PAN/primary MNT node following the CLI upgrade guide.
  3. Upgrade the secondary PAN/primary MNT node.
    • IMPORTANT: Do NOT run the upgrade via SSH. Either physically plug into the server with a keyboard and monitor or connect to the virtual KVM via the CIMC. Losing the SSH connection during the upgrade (before a reboot) will cause the upgrade to fail and there is no rollback. Your chances of this happening are very high according to Murphy’s Law so don’t take the chance.
    • When an upgrade is performed, a “new” deployment is formed because you can only have nodes running the same version of ISE in a deployment.
    • Running an upgrade will automatically remove it from the previous deployment.
    • This node will become the primary PAN/primary MNT in the new deployment.
    • All configuration (ie Policy Sets) is retained during the upgrade so the new primary PAN/primary MNT will have your configs.
    • Verify it is connected to your AD in External Identity Sources (if it was previously). If it has lost its connection, rejoin the domain.
    • If these are separate nodes (i.e. fully distributed deployment), I still upgrade the secondary PAN and primary MnT nodes and then continue.
  4. Once the node finishes upgrading and is fully online (verify, verify, verify), it’s time to reload the other nodes starting with one of the PSNs.
    • This will be a fresh install and not an upgrade for the remaining nodes.
  5. Connect to the node’s CIMC and utilize the virtual KVM to mount the Cisco ISE installation ISO.
    • Before rebooting the node in order to boot from the ISO, you should remove it from the old deployment. Not necessary if it is the last node to be reloaded.
    • You do not need to wait for the services on the now stand-alone node to fully come back online before starting the next step.
  6. Boot the node from the ISO and perform a full install from the ISO you mounted.
    • This will give you a clean server to later join to the new deployment running ISE 2.4.
    • Set up the node with the exact settings (IP, hostname, etc.) it had before the reload.
    • Remember backing up the SSL certificate and private key? This is why. You can import those back into the server instead of needing to purchase a new SSL certificate. Be sure to import the trusted root certificates first.
  7. Once the node is fully back online and operational, join it to the new deployment from the admin GUI of the upgraded PAN/MNT node.
    • Make sure you set up the node with the same settings as noted in Step 1.
    • If this is a PSN, you will need to reconfigure the profiling configuration after you joins the deployment.
    • As soon as it synchronizes, join the node to your AD!
  8. Verify the node is working properly (ie authenticating sessions if it is a PSN) and the certificates are assigned to the proper roles/portals.
    • If this is not the final node, the old deployment should still be up so you can use that to verify it matches or you can use the notes you made in step 1.
  9. If everything checks out, perform steps 5-8 for the 2nd PSN.
  10. Once the 2nd PSN is back online and running smoothly, you should only have one node left in the old deployment. It’s time to follow steps 5-8 to reload it and bring it into the new deployment.
    • It will be brought into the deployment as a secondary PAN/secondary MNT. That’s fine for now. The next step will change that.
    • If you had a node group in the old deployment, add the PSNs back to the same group. You may have to recreate the group name.
  11. Everything up and running? Sweet! Time to reload the node we upgraded in step 3 in order to have it with a clean slate. But first, you need to promote the secondary PAN/secondary MNT to primary PAN/primary MNT.
    • Keep in mind that changing the admin persona will result in a services reload on both PAN nodes. This will take approximately 10 minutes.
  12. Only after you verify synchronization is complete (all green check marks on the Deployment page) do you move on to removing the secondary PAN/secondary MNT from the deployment.
  13. Repeat steps 5-8 for the now stand-alone node.
    • The node will be joined to the deployment as secondary PAN/secondary MNT. Since this was the original secondary PAN/primary MNT, you only need to change the MNT persona to Primary.
  14. Once the secondary PAN/primary MNT finishes synchronizing, install the latest patch!

That’s it. Basically the same order you find in the Cisco upgrade guide except you are only upgrading one node and reloading all the others.

Extra steps? Sure.
Does it take a lot longer? It doesn’t add much time to the process (if any).
The GUI upgrade is easier! Yep.

So why do I do it this way? Because it cleans up the server of any extra files that are not necessary. Maybe it’s log files that are not needed or config files that are no longer required but don’t get cleaned up by upgrades. If you are worried about losing logs, you shouldn’t be if you made the backups as outlined in the upgrade preparation guide. RADIUS/TACAS+ logs should not be lost either since they were retained when upgrading the first node. This is also why it is important to verify the nodes fully synchronize before moving on to reload the next node so that the logs/configs are retained.

Virtual environments are even better. A lot of times the customer has enough room in their environment to pre-stage the nodes. They will load the OVA/ISO to the setup prompt so we can shut a node down and immediately start the setup process on the new node. I’ve even seen some people set up a VM node with a different IP so they only have to change the IP settings and then bring the new node into the deployment. Even if they can’t pre-stage a node, the reload usually goes quickly.

The same premise outlined above is used for any deployment type. Only the secondary PAN and primary MnT is upgraded, all others are reloaded, and then the upgraded nodes are reloaded. Be sure to follow the Cisco upgrade guide for the properly sequence!

Share this post:

2 comments

  1. would you usually take out PSNs from the load balancer so that NADs don’t use it for auth until the upgrade is complete (point few test NADs to the upgraded PSNs manually) ? Otherwise I guess there is a risk of outage for users if the upgraded half of the deployment is either facing issues due to bugs or AD not being joined, etc.

    1. Yes, that is a possible solution to limit the amount of down time experienced. It takes coordination between people doing the upgrade and the load balancer admins but that’s usually not a big deal. The same can go for the ASA/Firepower for VPN connections to limit VPN downtime while a node is up but not authenticating.

Leave a Reply

Your email address will not be published. Required fields are marked *