Smiley's Plan and Response
Amanda Lloyd
Director of IT Infrastructure & Operations
Amanda led Smiley’s response to the CrowdStrike outage, coordinating the team's actions, implementing technical solutions, and ensuring continuous communication with clients. Amanda explained: “We created the plan, and I asked my team to take action. They did not hesitate. When IT needed help, we asked, and no one hesitated. Having a team you trust, where everyone has each other's backs in a crisis, made this day feel like less of a crisis."
The Smiley team came together to face challenges head-on, ensuring the banks were able to open and conduct business as usual.
Comprehensive Communication Strategy
The Smiley team implemented a comprehensive communication strategy to keep their clients informed during the crisis–including phone calls, emails, and updates through their automated system.
Holly Roberts, the on-call IT System Administrator, first realized something was wrong when she received numerous alerts of system outages. She immediately went into action and began assembling the IT leadership team. Additionally, Smiley informed clients about the outage via their IVR (Interactive Voice Response) system (their automated phone system that routes calls to the appropriate person or department) and offered a callback option so clients wouldn’t have to wait on hold.
Timeline of the Response
Swift Problem Assessment and Action Plan
Upon identifying the issue, the team followed their business continuity recovery plan, prioritizing mission-critical systems such as web banking and SIBanking core applications. Initial assessments revealed that about a third of their servers were down. The team started by fixing the critical business servers first, as they were crucial for the banks' operations. They managed to bring ten critical servers online within the first 45 minutes.
Technical Solutions Implemented
Smiley followed CrowdStrike’s instructions to manually fix the problem by rebooting in safe mode and deleting the problematic file. They pushed out a script that automatically removed the problematic file, simplifying the process for the end users. The team employed remote access tools to manage both virtual and physical servers, allowing them to address issues without needing physical access to the hardware.
Thanks to their well-coordinated plan and technical expertise, Smiley successfully navigated the CrowdStrike outage, ensuring minimal disruption to their clients' operations.
