8 Stories from IT Pros Who Got 3,500 Users Remote in 48 Hours
This is an excerpt from the free case study: How to Prepare Your Company to Go Remote: 29 Lessons from IT Professionals Who Rapidly Scaled their Remote Infrastructure
Last week, millions of people worldwide started working from home — either voluntarily or as directed by local, state, or national governments — in an effort to limit the spread of the COVID-19 virus.
For some companies, the mass exodus was as simple as telling everyone to grab their laptop and monitors on the way out the door. For other organizations, sending everyone home presented challenges to the IT professionals tasked with preparing users, infrastructure, and hardware for the switch to a fully remote workforce.
We asked six IT professionals in the United States and South Africa how it went getting their people remote. Here are a few of the challenges they faced and their solutions.
More VPNs, More Problems
Previously when a user was going to be working remotely, we would work with them to set up the VPN and the Remote Desktop Connection. This was not a big deal since we only ever had to do one user at a time. If there was something difficult to understand in the documentation, they could simply come to us and we would explain it. When your entire office is going remote this becomes a much bigger hassle. The best way to avoid this is to have up-to-date documentation.
Lessons learned: Make sure your VPN instructions cover different operating systems. In some cases you will need to include documentation for tablets and even smart phones. Take lots and lots of pictures — and test everything thoroughly. Last week, we found a lot of mistakes in our documentation that presented users with erroneous instructions.
Pay attention to the VPN settings. In our case we are using Windows Server VPN "Routing and Remote Access" service. The default settings for Windows when creating a new VPN connection were sufficient. On some devices we had to specify PPTP as the VPN protocol.
Contributor: Nick in Gainesville, Florida
Disaster Recovery Software is Worth the Investment
I am fortunate to work at a large company that has multiple layers of disaster recovery and redundancy, so everyone going remote was not an issue for me. But I have colleagues that work at companies where remote work has been catered for, but not at scale. They mostly struggled with licensing constraints, limited concurrent VPN connections, and not enough remote resources.
Solution: Try getting in touch with your vendors and see if you can get licenses and seats provisioned during this tricky time, even if they are temporary. It doesn't hurt to ask, and if you can be creative, then you might be able to get your team back online sooner rather than later. If this is not an option for you, then coordinating access with your colleagues could be a solution. Draft an access schedule and share the resources that you do have.
Contributor: Graeme in Durban, South Africa
Slow Internet is Worse Than No Internet
No, you're not imagining this one. The problem with the current social distancing models is that everyone is stuck at home. If you are lucky enough to have a redundant or secondary internet connection for such occasions, then you are still not out of the woods. Everyone on your ISP's network is hitting the internet hard. My internet connection is a rock-solid LTE service that generally downloads and uploads at around 50mbps. This week it has gradually gotten slower and slower, and I am currently getting just over 2mbps as I write this.
I am lucky to live in a country where the mobile internet market has some healthy competition. I have a secondary SIM card on a rival network that seems to be taking much less strain during the day but is crippled in the evenings. I have been manually swapping between the two, and so far, so good!
Contributor: Graeme in Durban, South Africa
When Your SSH Cheat Sheet Has the Command to Access the Cheat Sheet
I have a confession to make: I don't memorize shell commands. Anytime I need to SSH into a server, I go to my handy-dandy list of commands kept on the share drive. Anything that requires the review of a system diagram, or a particular workflow is typically kept on a shared drive too. That is why it was so frustrating to learn I could not access my share drive from home. After hours of troubleshooting, I stumbled across a windows command called gpupdate /force.
This helpful command forces the group policy to update, thus synchronizing your user access to the Active Directory policies enforced on the server. If you are trying to access your share drive from home, but are unable to do so, I would recommend the following steps:
Go to windows —> run and type "cmd", then hit enter. This will bring up the windows command prompt. After doing so, type gpupdate /force and hit enter. The terminal will then provide a message that says, "Updating policy…" once this is complete, the share drive should be accessible.
As an aside, if you want an in-depth description of what gpupdate does, simply type gpupdate / in a different CMD window. This command will provide you with all the parameters and their descriptions.
Contributor: Erik in Fishers, Indiana
No Laptops? Buy Some.
The primary issue I have run into is that most employees were not geared for remote work (i.e. laptops, VPN access, and MFA accounts). We had a vast amount of employees working from desktops only and we do not have a VDI environment for employees. So, we had to order hundreds of laptops so that employees could take them. In addition, we were also stuck getting all users into the Active Directory group that allowed VPN access, as well as creating multi-factor authentication accounts for all users who never work remotely.
I believe the lesson here is that it would be most cost efficient to just have all employees work from laptops as their workstation (with a dock for multiple monitors), and create MFA accounts for all employees when they onboard — regardless of whether they need it or not, so that it doesn't cause a major backlog.
Contributor: Seth in Chattanooga, Tennessee
How to Set Up a Computer via FaceTime
Sending users home with a desktop and monitor because of shortages of laptops was a struggle, particularly for the ones with less than average computer skills. This is the user who struggles to open e-mail and print documents. Now they're being sent home with a computer and big box full of monitors, weird-looking cables, a keyboard, and a mouse. The IT staff struggles to help them over the phone with navigating all the cables and ports. To alleviate this issue, many users reverted to using FaceTime or other video call software to provide support and get them online.
Contributor: John in Jacksonville, Florida
Hardware Problems: Wireless Cards, Internet Access
The availability of spare equipment proved to be another challenge. IT personnel handed out all spare laptops immediately, and then started sending users home with their desktop and monitors. This posed another issue: most desktops do not have a built-in wireless card. We had to go out and buy USB wireless cards at local retail stores.
The second issue was home internet access. Broadband internet access in the rural United States is limited. To alleviate this issue, mobile hotspots were distributed to employees who needed it.
Contributor: John in Jacksonville, Florida
It's Time to Update Your Documentation
There was one common theme among all the responses.
While often overlooked, documentation is vital to the success of any IT operation. When it comes to documentation there can be several target audiences. These can be broken down into four categories: management, peer, support, and end-user.
Management documentation provides the proverbial "10,000-foot view" of the environment; whereas, peer documentation is akin to architectural blueprints and standard operating procedures.
Normally, the least focused on, both support documentation and end-user documentation really shine during a crisis. In particular, documentation for the end users will help diminish the effort required on three fronts.
- Ensure users know what to do (or at least those who follow the documentation)
- Thus cutting down on support calls
- And lastly enables the service desk to assist users consistently.
Done correctly, support documentation can elevate your service desk, allowing your engineers to focus on managing the crisis, not just responding to it.
Contributor: Carlos in Raleigh, North Carolina
Share Your Story with CBT Nuggets
While the decision to go remote may come from the top, it's the IT professionals in the trenches making it happen. They're the ones who spend the time on the phone with users setting up laptops, troubleshooting home networks, and updating documentation.
Hopefully, you've taken note of the things that went wrong (or that went well) so you can fix (or replicate) them for next time — or least know what challenges you may face.
If you have a story about prepping your systems and users to go remote, share them with us at @CBTNuggets or our Facebook page.
We'll keep adding to this list.