Modern Datacenter Architecture Patterns–Hybrid Networking
This document provides an overview of Microsoft Azure networking capabilities for hybrid environments. Microsoft Azure Virtual Networking enables you to create logically isolated networks in Azure and securely connect them to your on-premises datacenter over the Internet or using a private network connection. In addition, individual client machines can connect to an isolated Azure network using an IPsec VPN connection.
Table of Contents
1.2.1 Connecting On-Premises Networks to Azure
1.2.2 Extending Hybrid Networks Across Azure Virtual Networks
1.2.3 Extending Hybrid Networks Across Azure Datacenter Regions
1.2.4 Extending Hybrid Networks Across Azure Subscriptions
1.2.5 Providing developers remote access to the hybrid network
1.2.6 Additional recommended requirements
Prepared by:
Kunal Bhatia – Microsoft
Robert Larson – Microsoft
Michael Lubanski – Microsoft
Tom Shinder – Microsoft
Michael Withrow – Microsoft
Cloud Platform Integration Framework Overview and Patterns:
Cloud Platform Integration Framework – Overview and Architecture
Modern Datacenter Architecture Patterns-Multi-Site Data Tier
Modern Datacenter Architecture Patterns - Offsite Batch Processing Tier
Modern Datacenter Architecture Patterns-Global Load Balanced Web Tier
1 Overview
The Hybrid Networking design pattern details the Azure features and services
required to deliver network functionality that can provide predictable performance and high availability across geographic boundaries. A full list of Microsoft Azure regions and the services available within each is provided within the Microsoft Azure documentation site. This document provides an overview of Microsoft Azure networking capabilities for hybrid environments. Microsoft Azure Virtual Networking enables you to create logically isolated networks in Azure and securely connect them to your on-premises datacenter over the Internet or using a private network connection. In addition, individual client machines can connect to an isolated Azure network using an IPsec VPN connection.
The Hybrid Networking architecture pattern includes the following focus areas:
- Connecting on premises networks to Azure
- Extending Azure virtual networks across regions
- Extending Azure virtual networks between subscriptions
- Providing developers remote network access
1.1 Technology Background
In order to understand the Hybrid Networking Architectural Pattern, a basic knowledge of Microsoft Azure networking is required. The following table defines terms that will be used in the Hybrid Networking architectural pattern.
Technology |
Description |
An address space that can be established in Azure that can be subdivided to logically separate virtual machines and cloud services. |
|
One or more address spaces that represent on-premises routable subnets that you want to communicate with across a gateway |
|
A defined set of addresses from the Azure virtual network address pool that you want to logically separate for other addresses within the same network ID |
|
Server that provides domain name resolution to machines in Azure virtual networks. One or more DNS servers can be added to an Azure subscription and then assigned to a virtual network. The DNS server will be assigned to the virtual machines on the virtual network through DHCP during the boot process. |
|
This is a virtual VPN gateway device that is created on request in an Azure virtual network. It is designed to support routable connectivity between virtual networks, on premises locations, and single remote access VPN client machines. The gateway comes in three configurations: Dynamic, Static and Dedicated. Gateway uses IPSec and pre-shared keys to establish connectivity and encrypt data in motion |
|
This is a physical gateway that is established at a customer on premises location. In addition, this must be a supported device to be used with Microsoft Azure. |
|
This is the capability to have one or more virtual networks in a routable configuration. This is accomplished using IPSec based gateways and pre-shared keys. To accomplish vNet-vNet routing across the Azure backbone, you must configure vNet-to-vNet multi-hop routing. |
|
vNet Multi-hop Routing |
Virtual Network multi-hop routing requires the Azure Gateway device to be configured with vNet-to-Local Network Site routing where the local network site definition has all of the routable subnets on the other side of the connection. |
Daisy Chain |
A virtual network multi-hop routing configuration where the virtual networks connect in an end-to-end fashion
|
Hub and Spoke |
A virtual network multi-hop routing configuration where the virtual networks connect to a central hub virtual network and multiple hubs connect together for expansion.
|
Full Mesh |
A Virtual network routing configuration where the virtual networks connect to every other virtual network over a single hop connection. While this allows routing access to every subnet, it limits scalability due to the ten virtual network connection limit for a gateway
|
Azure Virtual Network to on premises connection utilizing Azure Dynamic or Static Routing Gateway and On Premises Gateway to establish a routable communications path between subnets on both sides. The connection is an IPsec tunnel over the Internet using pre-shared keys to establish the connection and perform encryption. |
|
Client computer to Azure virtual network connection utilizing an Azure Gateway and a VPN client. The connection is established as an IPSec VPN using certificates. |
|
Azure Virtual Network to on premises connection utilizing an Azure Dedicated Routing Gateway and a private network path to the on premises environment. No traffic flows across the Internet. Implementation of the ExpressRoute connection can be over a multi-protocol label switching (MPLS) circuit or dedicated connection. |
1.2 Pattern Requirements
The following are general requirements of the Hybrid Networking Infrastructure architectural pattern:
- An active Microsoft Azure account
- An active Microsoft Azure Subscription within the account
- An active Internet connection to access the Microsoft Azure portal or the service management API to configure Microsoft Azure subscription networking
- IP address pool for establishing Virtual Networks in Microsoft Azure that is non-overlapping with on premises
- IP Subnets from the IP Address pool for point-to-point connectivity: A non-overlapping IP address block (/28 subnet) to configure IP addressing for the Site-to-Site or ExpressRoute circuits
Once these requirements are satisfied, the following actions can be performed to provide network access to the organization’s Azure subscription:
- Connecting on-premises networks to Azure
- Extending on-premises networks across Azure Virtual Networks
- Extending networks across Azure datacenter regions
- Extending networks across Azure subscriptions
- Providing developers remote access to the Azure network
The following sections will describe each of the scenarios outlined above.
1.2.1 Connecting On-Premises Networks to Azure
Establishing a hybrid networking connection to Microsoft Azure can be accomplished using two approaches. The first involves utilizing an IPsec based Virtual Private Network (VPN) between Microsoft Azure and an on-premises network infrastructure and is often referred to as a “Site-to-Site” (S2S) connection. This approach uses the Internet as the communication medium and therefore is subject to random changes in latency and bandwidth availability.
The second approach is to use a dedicated network connection between Microsoft Azure and an on-premises network infrastructure and is referred to as an ExpressRoute connection. This approach uses dedicated private circuits or private network clouds for the connection. Unlike site-to-site connections, these private network paths do not utilize the Internet and are not subject to the random changes in latency and bandwidth of an Internet based connection.
IPsec VPN Connectivity
Site-to-Site communications utilize an IPsec tunnel established between two gateway devices (one hosted within Azure and one on-premises). The IPsec tunnel uses pre-shared key approach for establishing the connection and encrypting the data flowing over the connection. This allows data to flow across the Internet in a secure manner. The gateway within Azure is a virtual gateway implemented as a small virtual machine. When the gateway is created, you must select either a dynamic routing or a static routing gateway. Note that the on-premises gateway device must be implemented using either a supported model or compatible with a supported model. Details about the supported on premises gateways can be found in the article About VPN Devices for Virtual Network.
To establish the Site-to-Site connection between the two gateways, you must provide a pre-shared key and configure both gateways to utilize the key. The pre-shared key can be generated by the Azure portal, or it can be created and set on the connections using PowerShell. A single gateway may now establish multiple site-to-site connections to different on-premises locations. This enables redundancy for site-to-site communications. In this configuration, a separate on-premises gateway is required to establish the alternate connection.
ExpressRoute Connectivity
ExpressRoute allows direct site-to-site connectivity between an on-premises datacenter and Microsoft Azure, thus bypassing traditional, shared network connections through the Internet. It allows for flexibility of choice in the network performance desired. ExpressRoute allows two types of private connectivity to Azure - Through an ExpressRoute location (Internet Exchange Provider facility or IXP) or directly from an existing MPLS VPN network provided by a network service provider (NSP).
The following summarizes the requirements of the Hybrid Networking Infrastructure architectural pattern for connecting on-premises networks to Microsoft Azure:
S2S Internet connectivity
- Azure routing gateway (Static or dynamic)
- On-premises routing gateway device that is on the Microsoft approved list of gateway devices for Site-to-Site connections.
- Potential secondary on-premises routing devices if the pattern requires multiple on-premises site connections
- Pre-shared key for each connection path for establishing connection and encryption of data
ExpressRoute Connectivity
- Routing gateway device that is on the Microsoft approved list of gateway devices for ExpressRoute connections.
- An existing business relationship with an Internet Exchange Provider (IXP) or a Network Service Provider (NSP) who are on Microsoft's supported list of vendors for facilitating ExpressRoute connectivity
- Connectivity to an IXP: Network connectivity from customer datacenter to a supported IXP Datacenter (e.g. Equinix), along with an Ethernet cross-connect to IXP's Layer 2 peering infrastructure
- Connectivity to an NSP: MPLS Layer 3 VPN connectivity through at least one of customer sites/datacenters to a supported NSP network (e.g. - AT&T NetBond)
1.2.2 Extending Hybrid Networks Across Azure Virtual Networks
Hybrid networks can be established from an on-premises environment to Azure virtual networks. By default, these connections only allow the on-premises network to talk to the virtual network that the connection is established with (the first hop virtual network). In order to extend the connection from the first hop virtual network to additional virtual networks in Azure, you make configuration changes to establish multi-hop routing.
Establishing a hybrid network connection across Azure Virtual Networks requires an Azure gateway to be established and connected to the on-premises gateway. In the Site-to-Site scenario, an Azure static or dynamic routing gateway is required. If multi-hop virtual network routing is desired, then the dynamic routing Azure gateways must be utilized. In the ExpressRoute scenario, an Azure dedicated gateway is required. It is not possible to combine multi-hop routing and ExpressRoute connections currently. To obtain access to multiple Azure virtual networks in the ExpressRoute scenario, a separate ExpressRoute connection must be established with each virtual network.
1.2.3 Extending Hybrid Networks Across Azure Datacenter Regions
Hybrid network connections can span Azure datacenter regions to allow traffic to route between regions. This can be accomplished either using virtual networks in different regions connected using IPsec tunnels or ExpressRoute connections independently established to virtual networks in separate regions. These two methods cannot be combined.
Establishing a hybrid network connection to Azure Virtual Networks in different regions requires an Azure gateway be established in each virtual network. These gateways will then connect to each other to establish the link between Azure Virtual Networks. An on-premises gateway is required to establish a hybrid network connection to an on-premises datacenter. In the Site-to-Site scenario, an Azure dynamic routing gateway and multi-hop routing is required for cross region connections. In the ExpressRoute scenario, an Azure dedicated gateway is required. For ExpressRoute to obtain access to Azure virtual networks in different regions, a separate ExpressRoute connection must be established with each virtual network.
1.2.4 Extending Hybrid Networks Across Azure Subscriptions
Hybrid network connections can span Azure subscriptions to allow traffic to route between virtual networks in each subscription. This can only be accomplished using multi-hop routing of virtual networks in different subscriptions connected using IPsec tunnels. ExpressRoute connections currently are not supported across subscriptions.
Establishing a hybrid network connection to Azure Virtual Networks in different subscriptions requires an Azure Gateway be established in each virtual network and connected. In the Site-to-Site scenario, an Azure dynamic routing gateway and multi-hop routing is required.
1.2.5 Providing developers remote access to the hybrid network
In scenarios where a user does not have access to the corporate network where a Site-to-Site or ExpressRoute hybrid network connection can provide routable access to Azure virtual networks, it is possible to establish a temporary connection to the hybrid network by creating a Point-to-Site (P2S) connection directly from the a Windows-based system to an Azure virtual network. To accomplish this, an Azure virtual network must be configured to support Point-to-Site connections (remote access VPN client connections), an address range must be established for the machines connecting, a certificate must be created and uploaded for IPsec encryption, and the client machine must download a VPN client to establish the connection.
1.2.6 Additional recommended requirements
The following are not required to establish hybrid networking, but are recommended for an operational environment:
- Accessible DNS servers that can be specified in Microsoft Azure for automatic assignment to virtual machines placed on virtual networks
- An Enterprise Certificate Authority to generate trusted certificates for Point-to-Site connections
2 Architecture Pattern
The hybrid networking architecture pattern is complex due to the possible number of scenarios that can be created. This architectural pattern will focus on the following four scenarios:
- Site-to-Site hybrid networking with Multi-hop virtual network routing within a single subscription and region
- Site-to-Site hybrid networking with multi-hop virtual network routing across subscriptions and regions
- ExpressRoute hybrid networking using MPLS connectivity
- ExpressRoute hybrid networking using IXP connectivity
Site-to-Site Hybrid Networking with Multi-Hop Virtual Network Routing within a Single Subscription and Region
Site-to-Site Hybrid Networking with Multi-Hop Virtual Network Routing across Subscriptions and Regions
ExpressRoute Hybrid Networking using MPLS connectivity
This option uses a Network Service Provider and is for customers who already use MPLS VPN services through a supported telecommunications provider. In this option, Azure becomes another "branch office" extension to the customer's existing MPLS network. This option is relatively simple to deploy as the customer's carrier is responsible for configuring and managing the ExpressRoute connection. This option is suitable for customers who already use MPLS VPN services through a supported Telco. In this option, Azure becomes another "branch office" extension to the customer's existing MPLS network. This option is relatively simple to deploy as the customer's carrier is responsible for configuring and managing the ExpressRoute connection. This is illustrated in the diagram below:
ExpressRoute Hybrid Networking using IXP Connectivity
This option uses a Network Service Provider and allows customers to directly peer with Azure by collocating customer premises equipment at an IXP (e.g. - Equinix). This option provides more granular routing control to the customer and also enables higher bandwidth connections to Azure. It is suitable for enterprises that require a high-throughput connection to Azure or run low latency hybrid workloads, Media services, storage/backup in Azure. This option requires the customer to have in-house capabilities for configuring and managing this network infrastructure. This option is illustrated in the diagram below:
2.1 Pattern Dependencies
Being an infrastructure pattern, there are no dependencies on other patterns, however dependencies with the customer’s on-premises network environment and the network service provider used within the organization.
2.2 Azure Services
The following Azure Services are required for this architecture pattern as outlined above:
- Virtual Networks
- Virtual Network Gateways
2.3 Pattern Considerations
For each component of Microsoft Azure, a series of subscription and service limits are defined by the service. These limits are subject to change and are published at the following link. Microsoft Azure limits fall into the categories of default and maximum limits. Default limits are those which exist on every Azure subscription and can be increased through a request to Microsoft support whereas maximum limits define the upper boundary of a given service or capability within Azure. Limits can be raised by contacting Microsoft support as outlined through the Azure portal as outlined in this article, however the request cannot exceed the maximum limits outlined for each Azure service listed above. The following considerations must be taken into account during architectural planning of the Hybrid Networking pattern:
- Currently there are hard and soft limits on the number of Virtual Networks per Azure subscription. The default limit is 10, while the maximum limit is 100.
- Azure currently supports up to 3000 customer routes per ExpressRoute BGP peering.
- Quality of Service (QoS):
Customer traffic inbound to Azure: The NSP may provide QoS treatment of traffic traversing over its network towards Azure; this is dependent on the service Offering procured from the carrier.
Customer traffic outbound from Azure: This traffic is provided default QoS treatment as Microsoft routers today do not mark DSCP/CoS (Class of Service) on this traffic.
- A virtual network in Azure can have a maximum of a single routing gateway.
- A routing gateway can only be in a single state at a time (static or dynamic).
- A dynamic routing gateway in Azure has maximum of 10 connections (site-to-site or vNet-to-vNet).
- A Point-to-Site VPN requires dynamic routing gateway.
- A Site-to-Site VPN requires a dynamic or static routing gateway.
- An ExpressRoute connection requires a dedicated routing gateway.
- By default a vNet-to-vNet connection can only route across a single vNet.
- ExpressRoute connection cannot utilize vNet-to-vNet routing to pass traffic across the Azure backbone.
3 Interfaces and End Points
As outlined earlier, the Hybrid Networking pattern utilizes different types of end points to establish connectivity between the on-premises network and the Azure network infrastructure for each subscription. For the purposes of hybrid networking, these endpoints can be divided into physical and logical end points.
The following table outlines these end points by type:
| End Point Type | Description |
| Physical | IXP based connectivity - Ethernet cross connect setup between customer & Microsoft equipment collocated in IXP facility NSP based connectivity - MPLS based connectivity to Azure is extended from within existing customer WAN |
| Logical | BGP session # 1 for connectivity to Azure Compute services within a Vnet - VM's (IaaS) and Cloud Services (PaaS) BGP session # 2 for connectivity to Azure Public Services - SQL, Storage, Media Services etc |
4 Availability and Resiliency
Microsoft provides clearly defined Service Level Agreements (SLAs) for each service provided within Azure. Each architectural pattern is comprised of one or more Azure services and details about each individual Azure service can be found on the Microsoft Azure Service Level Agreement website. For the Hybrid Networking architectural pattern, the Azure services required carry the following SLAs:
Azure Service |
Service Level Agreement |
Virtual Network Gateway |
99.9% |
Virtual Networks |
99.9% |
ExpressRoute |
99.9% |
The composite Service Level Agreement (SLA) of the Hybrid Networking architectural pattern is 99.9% based on the services outlined above. The details behind the SLA can differ based on the type of connectivity utilized. For ExpressRoute connections, the composite SLA is comprised of the ExpressRoute SLA offered by Microsoft combined with the underlying infrastructure which is provided by a third party service provider. For Site-to-Site connections, the composite SLA is comprised of the VPN SLA offered by Microsoft is around the Virtual Gateway and not the connection itself.
5 Scale and Performance
A key consideration for the performance and scale of the Hybrid Networking architectural pattern is highly dependent on the network connection itself, which is defined by the type of connection used. In general, ExpressRoute provides higher scale and performance (higher throughput, lower latency, lower jitter) in comparison to an Internet-based IPSec VPN connection (site-to-site or point-to-site). The differences in the type of connection and expected speeds are outlined in the table below:
Type of Connection |
Speed |
Network Service Provider |
10Mbps, 50Mbps, 100Mbps, 500Mbps & 1Gbps |
Exchange Provider |
200Mbps, 500Mbps, 1Gbps & 10Gbps |
Site-to-Site VPN |
Default: 100Mbps, or High Performance (200 Mbps) |
Point-to-Site VPN |
100Mbps |
6 Cost
An important consideration when deploying any Solution within Microsoft Azure is the cost of ownership. Costs related to on-premises cloud environments typically consist of up-front investments in compute, storage and network resources, while costs related to public cloud environments such as Azure are based on the granular consumption of the services and resources found within them. Costs can be broken down into two main categories: cost factors and cost drivers.
Cost factors consist of the specific Microsoft Azure services which have a unit consumption cost and are required to compose a given architectural pattern. Cost drivers are a series of configuration decisions for these services within a given architectural pattern that can increase or decrease costs. Microsoft Azure costs are divided by the specific service or capability hosted within Azure and continually updated to keep pace with the market demand. Costs for each service are published publicly on the Microsoft Azure pricing calculator. It is recommended that costs be reviewed regularly during the design, implementation and operation of this and other architectural patterns.
6.1 Cost Factors
Cost factors within the Hybrid Networking architectural pattern include the choice in ExpressRoute providers and connectivity options (Site-to-Site vs. ExpressRoute). For Site-to-Site connections (VPN) there is a minimal charge for the Gateway itself. The connection runs over the open Internet. Customers can establish the ExpressRoute Service in two ways – Exchange Provider or Network Service Provider. The associated charges for both these options differ accordingly. ExpressRoute service using an Exchange Provider is charged based on a monthly dual-port fee for two physical Ethernet ports (on two routers).
Inbound transfer (e.g. - Customer to Azure) is included. A finite outbound transfer (e.g. - Azure to Customer) is included with the port speed selected. Any outbound transfer beyond that is charged based on bandwidth consumed. Alternatively, ExpressRoute using a Network Service Provider is charged based on a monthly dual-port fee for two physical Ethernet ports (on two routers), and there are no additional costs related to all inbound and outbound data transfer.
6.2 Cost Drivers
As stated earlier, cost drivers consist of the configurable options of the Azure services required when implementing an architectural pattern which can impact the overall cost of the Solution. These configuration choices can have both a positive or negative impact on the cost of ownership of a given Solution within Azure, however they may also potentially impact the overall performance and availability of the Solution depending on the selections made by the organization. Cost drivers can be categorized by their level of impact (high, medium and low).
Cost drivers for the Hybrid Network architectural pattern are summarized in the table below.
| Level of Impact | Cost Driver | Description |
| High | Connectivity from customer location to Azure Cost of egress of data using IPsec VP connection to Azure | While considering the cost of connecting to Azure using ExpressRoute, it must be kept in mind that Network service providers/Exchange providers may levy additional charge/fees for their services in addition to the ExpressRoute change levied by Microsoft An IPsec VPN based connection to Azure incurs charges only for egress of customer data outbound from Azure. There are no cost impacts for ingress (inbound) of data into Azure, irrespective of the amount of data transferred |
| Medium | Data transfer over ExpressRoute within US/Europe Azure regions Data transfer over vNet-to-vNet connection across Azure zones | Once a customer has established an ExpressRoute connection to a Microsoft location within US/Europe, data can be sent to/from any other Azure region within that continent. For example, a US customer can send or receive data to/from any Azure region within US without additional charges from Microsoft. This however, is not applicable to Asia-Pacific In a VNet-to-VNet Scenario, an egress charge will be assessed if those VNet’s are in different Azure zones |
| Low | Cancellation of ExpressRoute Service | Prorated billing is applied in case the ExpressRoute service is cancelled during a month such that the billing would only be for the hours used and the actual data transfer incurred |
7 Operations
Cloud Platform Integration Framework (CPIF) extends the operational and management functions of Microsoft Azure, System Center and Windows Server to support managed cloud workloads. As outlined in CPIF, Microsoft Azure architectural patterns support deployment, business continuity and disaster recovery, monitoring and maintenance as part of the operations of the Hybrid Networking architectural pattern.
Deployment of this pattern can be achieved through multiple methods. In addition to using the Management portal, ExpressRoute deployment currently requires PowerShell-based configuration tasks which need to be performed through the Service Management REST API. The following three broad steps that need to be carried out in a proper order for a circuit to be fully provisioned. Detailed configuration steps can be found in the article Configure an ExpressRoute Connection through an Exchange Provider.
- Creation of ExpressRoute circuits
- Route configuration
- Linking Virtual network to ExpressRoute circuit
These steps are illustrated in the diagram below:
Figure 1: ExpressRoute Deployment Workflow for Exchange Provider based connection
Azure Virtual Networks can also be created through the portal and Azure PowerShell. When dealing with VNet-to-VNet configuration and multi-site VPN configurations, customers will be required to author a .netcfg xml configuration file for deployment.
Monitoring of this pattern and associated resources depends on the connection type. An ExpressRoute circuit can be managed and monitored through the Service Provider portal. A Site-to-Site VPN connection can be managed and monitored through the Service Portal. From this view a customer will only see the External interfaces from a connection perspective. With a Site-to-Site VPN connection the only side which is visible in the portal is the Azure side of the connection. In order to look at verbose logging of the connection access to the on-premises router will be required. For VNet-to-VNet monitoring, the gateways and the connection can be monitored solely in the Azure Portal.
Depending on the nature of the task involved, maintenance of an ExpressRoute connection can be performed using a combination of Azure Management Portal, Service Provider portal and/or PowerShell through the Service Management API.
From a business continuity and disaster recovery perspective, the availability constructs outlined earlier should help make sure that solutions deployed using the Hybrid Networking architectural pattern are designed to support continuity of operations, however it is important to understand how this differs between ExpressRoute and Site-to-Site VPN connections.
ExpressRoute connections are typically terminated at both ends using two physical Ethernet ports on two separate routers. This allows for a pair of simultaneously active physical connections to carry data traffic to/from an ExpressRoute location. While Microsoft offers a 99.9% uptime SLA on the ExpressRoute service itself, it is up to the customer to appropriately architect the network connectivity to an ExpressRoute location. This SLA will not be applicable if the circuit is not built in an active-active configuration. Site-to-Site VPN connections are associated to a single Azure Virtual Gateway that is redundant, but that connection is not redundant. In the event of a lost connection on a Gateway a customer will have to establish another connection.
Operational changes to an ExpressRoute circuit generally require modifications to both Microsoft Azure and the underlying Service provider’s infrastructure. While Microsoft allows customers to perform operational changes through the Service Management REST API, the service provider generally offer a portal through which operational tasks can be carried out. Operational changes to a Site-to-Site VPN Connection are very limited. Once the network is in use, most changes are restricted. If you need to delete a gateway or change a gateway and there are virtual machines on the Virtual Network. They will have to be removed in order to make that modification.
8 Architecture Anti-Patterns
Establishing a Hybrid Network to Microsoft Azure where multi-hop routing will be used in a Site-to-Site VPN scenario is not
recommend. It is not recommended to utilize the “full mesh” approach (outlined above) to perform multi-hop virtual network routing. When multi-hop virtual network routing is configured using the full mesh routing approach, the 10 virtual network connections per gateway will limit the scale of the hybrid networking infrastructure in Azure to a maximum of 11 virtual networks.
The ExpressRoute connection is meant to be a Layer 3 (routed) connection between the customer and Azure. As such, no Layer 2 connectivity is allowed over an ExpressRoute circuit.
