Connected Drones: 3 Powerful Lessons We Can All Take Away

Connected Drone Powe by Azure

Lesson 1: Cloud-Based Solutions Empower Trailblazers Wherever They Are

eSmart Systems is a young dynamic company in Norway that made the early strategic choice of basing their products and services completely on Microsoft Azure. Using Azure, gave them an immediate global reach in a way unthinkable just a few years earlier. Their mission is to bring big data analytics to utilities and smart cities, and one of their focus areas is electric utilities and smart grids. eSmart has been delivering analytics solutions to power utilities for years, but the specific use case I want to share with you in this blog is called Connected Drone. It is a story that combines drones with intelligent software to prevent power blackouts, or as eSmart puts it “making Azure intelligence mobile”.

To see Connected Drone in action, please watch this video.

Big Data Analytics to Utilites and Cities

The economic impact of blackouts is massive, and the scale of power grids is huge. According to Grid Resilience Report, US Department of Energy, 2011, the average annual cost of power outages caused by severe weather in the USA is anywhere between $18-33 billion. Current inspection methods include dangerous, low-flying helicopters, or people climbing pole-by-pole, and even in some cases, dogs sniffing for rot. Once the inspection is done, electric companies have data, and lots of it. Typically, they manually go through it, picture-by-picture, looking for faults. Such manual process over vast amounts of data is obviously slow, error-prone and very expensive.

Initiate Plan Execute Analyze Azure

Lesson 2: Established Industry + Deep Intelligence + Cloud = New Opportunities!

eSmart began developing Connected Drone out of a strong conviction that drones combined with cloud intelligence could bring a positive disruptive change to the power industry. The objective of Connected Drone is to support and automate the inspection and monitoring of power grid infrastructure instead of the currently an expensive, risky, and extremely time consuming activity performed by ground crews and helicopters. To do this, they use Deep Learning to analyze multiple data feeds streamed from the drones. Their analytics software recognizes individual objects, such as insulators on power poles, and directly links the new information with the component registry, so that inspectors can quickly become aware of potential problems. Connected Drone covers all phases of the inspection process: (1) initiation of a mission, (2) planning of the flight, (3) execution of the mission, and finally (4) the analysis of collected data.

eSmart applies a range of deep learning technologies to analyze data from Connected Drone, from the very deep Faster R-CNN to Single Shot Multibox Detectors and more, deployed on Azure GPUs.

Neural Nets for Object Recognition

Below you can see some examples of analysis of both still images and video streams and how the system recognizes the different components of power poles.

Drone Still Image 1

Drone Still Image 2

Lesson 3: When Faced with Challenges… Unleash Your Creativity!

In order to really appreciate the ingenuity that went into Connected Drone project, you need to get a taste of the challenges that eSmart faced:

  1. Recognition of Small Objects. The first challenge was the recognition of small objects. Images for power line inspections are typically high-resolution. Object detectors that are available typically work with low resolution images, since they were developed to recognize big objects such as cars, people, airplanes, etc., (e.g., ImageNet). Detecting small differences between small objects from a small image is simply not possible. eSmart solution was to have a multi-step process where low resolution object detectors first identify generic objects, and then this information is used to crop the high resolution image and apply an object classifier on it.Small Object Recognition
  2. Class Imbalance. The second big challenge was that of class imbalance. eSmart had many different components that they needed to identify. What you see in the picture below is just a small sample. Some components are more common than others, and it is not hard to imagine all the potential problems this creates when trying to create a balanced classifier.Class Imbalance

3. Few Fault Examples. A similar problem is the lack of sufficient examples of the faults that analytics software needed to recognize using Connected Drone. Below are a few examples of some fault types. It is unrealistic to expect to have tens of thousands examples of each fault situation, so eSmart needed to think of something new and innovative to address this challenge as well.

Few Fault Examples

A Creative Solution

The solution they came up with is to complement real images with synthetic images. They’ve built detailed 3D models which are rendered and can be used to generate images, on demand, by varying camera angles, lighting conditions, and backgrounds. In this way, they could both balance the object classes, and generate additional fault examples as needed. This is a very creative way of doing machine teaching to address the limitations and scarcity of high-quality training data.

Mix Real and Synthetic Images

So How Does This All Actually Work?

Below is an example from a real drone operation eSmart did in September. The red lines are the customers power grid, the blue is the 6 kilometer line that Connected Drone had inspected.

Real Drone Operation Activity 1

Closing in on the inspected line (picture below), you can see 6 observations that AI made.

Real Drone Operation Activity 2

The screenshot below shows that there might be a problem with one of the insulators.

Real Drone Operation Activity 3

The inspectors can then go through all the inspected data, pole-by-pole, and create a diagnosis. With Connected Drone on Azure, eSmart was able to turn the massive amount of inaccessible data into useful information that energy companies can act on.

Real Drone Operation Activity 4

Today, connected drones fly and literally collect an inventory of assets for electric companies. The ultimate solution that eSmart envisions is taking the skills of people working on power grids and train AI to suggest the areas and what should be fixed and the cost that it would take to fix it. eSmart is working on “what-if” analysis using the data from their Connected Grid platform and the time-series data. The outcome will be something like a time-travel “movie”, where you can go backwards and diagnose what happened (that part is relatively easy), or you can go forward (using their Connected Grid algorithms) and predict what will happen. Using the time-series travel, and allowing users to change the conditions will enable customers to see how it impacts the ‘what-if’ scenarios and where the electric companies should invest. Furthermore, eSmart is looking closer is expanding the usage of sensors. Connected Drone project is just scratching the surface of what’s possible.

Bonus Lesson: Empower Others to Join You on the Same Journey!

The story doesn’t end here… One of the interesting things was how eSmart created a community around the Connected Drone project.

Involving Kids Through Gamification (to Tag Insulators Smile)

Who knew that power grids could be gamified? eSmart proved that it can be. In order to train the models, they needed a lot of training data, so they made an AI Training Service with a website front-end to crowdsource data tagging with a ranking list as a motivational factor (similar to the one in computer games). Local kids, football teams and school classes are raising money for their tournaments and trips by tagging insulators. (Surprisingly, grown-ups weren’t as enthusiastic). In addition to having fun, there is a strong incentive that they are doing something important for the infrastructure of their country and for their teams.

Involving Kids Through Gamification

Helping Government Agencies (through “Black Box” for Drones Innovation)

In Norway, drone business has recently exploded. There are more than 1,000 operators. Currently, Norway’s equivalent of FAA, called Civil Aviation Authority Norway (CAA), is severely under-staffed. eSmart has been closely working together with Norwegian CAA in developing safe and risk-free drone missions. Today, when drones fly, they typically don’t store statistical data about the flight. It’s one of the reasons why longer flights and more advanced operations are not allowed.

eSmart Solution is building a virtual “Black Box” for drones to make risk analysis more automatic. A virtual black box collects sufficient information about the flight and using that data they can perform risk analysis and share it with CAA. As eSmart puts it “We are not only in the business of helping electricity companies, but also CAA”. The basic idea of the black box is to actually stream data into Azure and then perform risk analysis on it there. In some rural areas, however, connectivity may actually be lacking, so in those cases they may store it on the drone itself and then perform analysis post-mission.

Collaborating with Drone Vendors to Design More Robust Drones (via Predictive Maintenance)

eSmart is also working on predictive maintenance for drones (something similar in spirit to this). They are actively involved with Robot Aviation (their drone partner) on addressing early warnings, e.g., when the temperature of a drone goes up over some period of time, maybe it’s not a good idea to fly. Furthermore, Robot Aviation doesn’t just have one type of drone – it’s a fleet of drones, from helicopters to small drones. Since the weather in Norway is often cold (as one Norwegian proverb says: “There Is No Such Thing as Bad Weather, Only Inadequate Clothing”), they need drones that can fly in cold weather and in remote areas and under bad weather conditions (many drones aren’t fit for this).

Other Industries?

While today, Connected Drone project only does inspections of power grids, it’s not a big leap to imagine other applications using the same concepts (of training models, ability to recognize objects, safe planning of missions) and architecture, for instance, to inspect railroads, infrastructure and gas pipelines.

It is fair to say that Connected Drone project exemplifies how Deep Learning on Azure GPUs is redefining what a cloud solution can do.