Sweating the Work

On the first day of the Malaysian Movement Control Order (MCO) on 18th March 2020, our network team were glued to their seats. The whole company would be connecting to the office remotely and our team weren’t sure if the network would hold. The good news  ­- like the Year 2000 crisis – it was an anticlimax. The Juris Technologies (JurisTech) network infrastructure handled the entire load without breaking a sweat.

Of course, the Year 2000 crisis never became a real problem because there was a massive amount of work behind the scenes. Our Business Continuity Plan (BCP) and network teams also did a lot of preliminary work that made handling the MCO manageable. Here are some of the lessons learnt.

Adequate BCP Network Infrastructure

Long before the COVID-19 pandemic, in 2018, JurisTech upgraded their networking infrastructure to be able to handle 4 times more staff than it currently had then. JurisTech also installed broadband links to the Internet from two different ISP vendors. We regularly back up our essential documents to Amazon S3 and have a backup data centre in Kuala Lumpur in case our main office data centre goes down.

For security reasons, we also separated the network into DMZ (Demilitarised Zone), Development, and Sales & Admin zones. In this way, we are able to open the VPN to users and give them different access permissions into these zones based on their roles.

Preparations for COVID-19

In late February 2020, JurisTech management could see the COVID-19 crisis getting worse, and preparations were made. Hand sanitisers were made freely available and tracking of visitors was enforced.

The network team was also busy. Previously, VPN to office was only granted upon request. Now, we started to install VPN on every notebook, and created VPN accounts for everyone. We also started a policy where every staff member had to take their notebook home at the end of the day.

Special Attention was given to Admin and HR as they are used to working in the office and we had to plan the transition to remote work, carefully. Both our Accounting and HR software had been migrated to web-based versions last year, so it was easy to access the office Accounting and HR software using web browsers. Contingency plans on who was supposed to take home printers and projectors were also made.

Our call centre software is web based, but we had to handle rerouting of phone calls to helpdesk. We solved this by tuning our call centre software and ACD (automatic call distributor) to reroute calls to our helpdesk staff’s smartphones. We tested this a week before the MCO was announced.

Although Zoom is more popular in our company for video conferencing, we had a limited number of paid Pro accounts. So, plans to use Skype for video conferencing were made, as Skype is free and is installed by default as part of Windows 10.

Movement Control Order

When the Movement Control Order from March 18 – 31 this year was announced on 16th March, we were all very stunned. Then, our preparations kicked in. On 17th March, we had a 9am meeting with all the Business Unit heads to detail out the emergency plans. We also backed up key files and development virtual machines (VMs) into a separate laptop in case the VPN failed. Fortunately, this laptop was never used as our VPN worked so well.

One issue common to VPNs is that all traffic by default goes through the VPN. This means if our staff is watching a YouTube video while connected to the VPN, the video would go through our office network, taking up our precious bandwidth. We solved this by setting the VPN client to route only our office network domains through the VPN, while any other traffic would bypass the VPN and go direct through our staff’s normal broadband link.

We also found that the antivirus installed on our laptops would sometimes be too aggressive in scanning shared drives, taking up gigabytes of our precious network bandwidth! This was solved by switching from file sharing to using FTP/SFTP for most file access.

Another optimisation is that we tuned our network to perform load balancing on our two links to the Internet.  The faster link would take 80% of the load, while our slower backup link would take 20%. Lastly, we moved our backups to Amazon S3 over the weekend to minimise bandwidth issues.

Keeping People Together

JurisTech practices Scrum, and our teams have a daily standup meeting already. We switched to using Discord for our daily standups, which is a fantastic combination of voice and text chat, video conferencing, and groups. Originally developed for gamers, it has become extremely popular as a replacement for Slack and Skype.

For our Kanban boards, we switched from using Trello to ClickUp. ClickUp is better because it allows you to convert a comment into a task to be assigned to someone, allows multiple assignees, and has other features suited for large teams working on large projects.

We continue to use Skype, but now that Zoom has removed the 40-minute meeting restriction for free accounts for this period, we now conduct more Zoom meetings.

Lessons Learnt

1. Prepare early. Two years ago, we were committed to upgrading our network infrastructure by investing in better firewalls and VPN software. We started planning for a possible MCO in late February, and focused especially on helping units that had additional requirements such as HR and Admin.
2. Don’t forget network security. Once you open up VPN access, security becomes more important. We tightened our network security and started collecting network statistics to track VPN usage to prevent abuse of VPN.
3. Tune the network and VPN. We restricted shared drive access because of aggressive antivirus software. We set router rules so YouTube/Skype/Zoom/surfing traffic does not go through our internal network. We rerouted phone calls to our staff’s smartphones.
4. Estimating bandwidth usage for VPN. All our servers are running on VMware vSphere. We totalled the average network utilisation of all our VMware servers and used that. This was pretty accurate.
5. Not everyone will log in at the same time. During working hours, about half of our staff log in at any one time. Network usage was higher on the first few days of MCO, probably because people were downloading files just in case the network failed. Today, the 6th day of MCO, each VPN uses less than 1Mbps on average.
6. Keep communicating! Working from home, it is important to increase the frequency of group contact. We switched from a daily standup to twice-daily standups. Meetings are conducted using Zoom/Skype with screen sharing. Teams collaborate using WhatsApp, ClickUp (Kanban boards), and Discord (Chat/Conferencing).

Credits: Photo by Dimitri Karastelev on Unsplash