Networking Blog articles https://gorovian.000webhostapp.com/?exam=t5/networking-blog/bg-p/NetworkingBlog Networking Blog articles Wed, 20 Oct 2021 08:22:31 GMT NetworkingBlog 2021-10-20T08:22:31Z Deploying HTTP/3 on Windows Server at Scale https://gorovian.000webhostapp.com/?exam=t5/networking-blog/deploying-http-3-on-windows-server-at-scale/ba-p/2839394 <P>Windows Server 2022 was <A href="https://gorovian.000webhostapp.com/?exam=t5/containers/windows-server-2022-now-generally-available/ba-p/2689973" target="_blank" rel="noopener">released for general availability</A> last month. Since then, in cooperation with the Microsoft 365 team, we have started deploying the latest Windows Server on Exchange Online service front door servers globally, with a primary goal of adding support for HTTP/3 to <A href="#" target="_blank" rel="noopener">https://outlook.office.com</A>. We have only scaled the deployment up to 20% of capacity of front-end servers so far, but the data we are getting back is looking great!</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="nibanks_4-1634066443511.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/316871i231333DA71B4CC13/image-size/large?v=v2&amp;px=999" role="button" title="nibanks_4-1634066443511.png" alt="nibanks_4-1634066443511.png" /></span></P> <P>&nbsp;</P> <P>Total requests per second (RPS) have steadily increased in-step with the increased deployment to Exchange Online service front doors. Now that deployment is at 20% of capacity, we are seeing RPS spike nearly to 50k per second. Throughout the deployment we have been tracking the last-mile request latencies. The last-mile request latency is the time spent between the client and the front-end server; essentially request time minus any back-end communication and processing. Exchange Online service front doors support small request, small response workloads for various SPAs (Single Page Applications) such as Outlook on the Web where responsiveness is a key differentiator for user experience.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="nibanks_5-1634066443517.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/316872i7E2ABA0A9A2CE56C/image-size/large?v=v2&amp;px=999" role="button" title="nibanks_5-1634066443517.png" alt="nibanks_5-1634066443517.png" /></span></P> <P>&nbsp;</P> <P>The latency data coming back has held firm throughout the deployment. As seen in the last-mile HTTP request latency data above, HTTP/3 is giving huge gains for Microsoft 365. An 8% reduction from the baseline at the 50<SUP>th</SUP> percentile is not bad, but over a 60% reduction at P99.9 is huge!</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="nibanks_6-1634066443519.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/316870i23BDB2316D9604E1/image-size/large?v=v2&amp;px=999" role="button" title="nibanks_6-1634066443519.png" alt="nibanks_6-1634066443519.png" /></span></P> <H2>Why is HTTP/3 Better?</H2> <P>You will immediately ask why HTTP/3 has so much lower latency. Well, there are a lot of factors at play, most of which come from replacing TCP/TLS layers with QUIC.</P> <UL> <LI>QUIC reduces the number of round trips in the handshake by combining the transport (QUIC) and the security (TLS) handshakes together. One less round-trip means the HTTP request can be started (and completed) that much faster.</LI> <LI>QUIC reduces/removes cross-request (stream) head of line blocking by: <UL> <LI>Only retransmitting stream data that was lost in a QUIC packet. This ensures streams without packet loss continue unaffected. Parallel HTTP requests no longer must wait on earlier requests just because they lost a packet. Only the request that lost a packet suffers the extra latency, while all others continue unaffected.</LI> <LI>Using first-in, first-out (FIFO) framing of sent stream data instead of round-robin. This ensures completion of a request is not delayed by the payload of any other request. In previous versions of HTTP, Windows used a round-robin approach to be “fair” to all requests. In HTTP/3 we moved to the FIFO model to reduce overall latency, completing the requests as fast as possible, as they come.</LI> <LI>Encrypting on a per-packet boundary (instead of 16KB per-TLS message). HTTP over TCP/TLS encrypts data in larger chunks, to reduce CPU costs, but this also has the effect of delaying the decryption of the entire chunk when a single packet is lost. QUIC pays the marginally higher CPU cost to make all packets independent.</LI> </UL> </LI> <LI>QUIC builds/improves on TCP loss recovery by: <UL> <LI>Eliminating the retransmission ambiguity in TCP by not reusing packet numbers.</LI> <LI>Updating probe timeout (PTO) logic for quickly transmitting on suspected loss. Much of this logic exists in TCP, but QUIC takes the latest learnings and puts them all together for the best results.</LI> <LI>Removing the selective acknowledgement (SACK) limitation of TCP (max of 3 SACK ranges) to allow for QUIC to accurately acknowledge received packets in the face of sporadic, non-continuous packet loss. This results in QUIC utilizing available bandwidth more efficiently because it does not unnecessarily retransmit data the peer already received.</LI> </UL> </LI> <LI>QUIC uses pacing when sending packets. <UL> <LI>This reduces the burst sizes of traffic onto the network, generally reducing packet loss.</LI> </UL> </LI> </UL> <P>To sum all this up, QUIC makes parallel work independent, improves the speed and accuracy of loss detection, and generally tries to be a better citizen on the network. As you can see from the data above, this results in HTTP/3 proving huge gains in responsiveness for Microsoft 365!</P> <P>&nbsp;</P> <P>Deployment of HTTP/3 support in Exchange Online is the latest effort undertaken by the Microsoft 365 team to improve end-user experience via network-level engineering. Customers who have aligned with the Microsoft 365 <A href="#" target="_blank" rel="noopener">Principles of Network Connectivity</A> will experience the most benefit from this improvement and others to come. If you are a Microsoft 365 tenant administrator, you can find guidance on how to improve your connectivity in the <A href="#" target="_blank" rel="noopener">Microsoft 365 admin center</A>.</P> <P>&nbsp;</P> <H2>Trying Out HTTP/3 on Your Server</H2> <P>If you are currently on an older Windows Server using HTTP and want to try out HTTP/3 please take a look <A href="https://gorovian.000webhostapp.com/?exam=t5/networking-blog/enabling-http-3-support-on-windows-server-2022/ba-p/2676880" target="_blank" rel="noopener">here</A>. If you are using .NET instead, you can try <A href="#" target="_blank" rel="noopener">HTTP/3 with the ASP.NET Kestrel web server</A>, but be aware that it is still in preview.</P> Tue, 12 Oct 2021 19:41:30 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/deploying-http-3-on-windows-server-at-scale/ba-p/2839394 nibanks 2021-10-12T19:41:30Z Network ATC: Common Preview Questions https://gorovian.000webhostapp.com/?exam=t5/networking-blog/network-atc-common-preview-questions/ba-p/2780086 <P>Network ATC has received some great feedback during its time in preview. We’ve heard that you love how it:</P> <UL> <LI>Simplifies deployment across the entire cluster</LI> <LI>Implements the latest Microsoft validated best practices</LI> <LI>Keeps all cluster node’s configuration in-synchronization</LI> <LI>And remediates misconfigurations mistakenly configured by an administrator</LI> </UL> <P>In discussions with our preview customers, we’ve also heard some common questions. This blog will discuss some of the most common questions we’ve heard along with our recommendations to each.</P> <P>&nbsp;</P> <P>But first, let's recap one important distinction about Network ATC. Network ATC doesn’t change what you deploy, just how you deploy it. For the intents that Network ATC manages (Management, Compute, Storage) you should no longer think about a virtual switch, host vNICs, adapter properties, etc. Instead, think about your outcomes (intents) and Network ATC take care of the rest.</P> <P>&nbsp;</P> <P>This is an important change: Whether you’re a seasoned pro with years of experience writing your own PowerShell scripts, custom tweaks, and modifications, or a complete n00b that is just nervous about not having all that information – the outcome will be the same – Network ATC will deploy everything needed to match the intent of the adapters you specified. In other words, every deployment comes with the collective knowledge of the MVPs, Partners, and customers as well as Microsoft’s own networking configurations validated in our test environments EVERY DAY.</P> <P>&nbsp;</P> <P>With that understanding, let’s dive into the article and discuss some of the questions we received. If you have more, check what we’ve <A title="Example Network Intents" href="#" target="_blank" rel="noopener">documented already</A>, add them in the comments below, or send a question on <A title="twitter" href="#" target="_blank" rel="noopener">twitter</A>!</P> <P>&nbsp;</P> <H1>Can Network ATC NICs also be used for Backup Networks?</H1> <P>&nbsp;</P> <P>Yes! Network ATC configures the host adapters for the intent types that you indicate (management, compute, storage). However, this doesn’t mean that these are the <STRONG><EM>only</EM></STRONG> uses for these NICs. You could add VMs (and subsequently virtual NICs) to a compute switch, you could layer on SDN, AKS-HCI, or more. This includes a custom NIC(s) to connect to your backup network.</P> <P>&nbsp;</P> <P>If the intents you chose created a virtual switch (any combination of intents besides only -Storage), you would use the following cmdlet to attach a new virtual NIC to the virtual switch.</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="powershell">Add-VMNetworkAdapter -ManagementOS -Name Backup01 -SwitchName 'SwitchName'</LI-CODE> <P>&nbsp;</P> <H1>Can I use Network ATC with a Stretch Cluster?</H1> <P>&nbsp;</P> <P>Certainly! As mentioned above, you can add additional configurations (including virtual NICs) to the solution deployed by Network ATC. For reference, we’ll use the stretch cluster configuration <A title="networking documentation for stretch cluster." href="#" target="_blank" rel="noopener">from our networking documentation for stretch cluster.</A>&nbsp;.</P> <P>&nbsp;</P> <P>At the time of writing, there is one nuance to using stretch. As with all intents, all nodes in the cluster are expected to look exactly the same. Therefore, if you specify the storage intent, both sites must use the same storage vlans. The same goes for the Management intent. For example, if you use the command:</P> <P>&nbsp;</P> <LI-CODE lang="powershell">Add-NetIntent -Name IntentName -Storage -Compute -AdapterName pNIC01, pNIC02 -ClusterName Cluster01</LI-CODE> <P>&nbsp;</P> <P>All hosts in the cluster (in both sites) will use the storage VLANs of 711 (pNIC01), and 712 (pNIC02). This doesn’t mean that you need to stretch VLANs if your cluster spans physical datacenters. Since this storage traffic is limited to the local site, it doesn’t need to span sites. Subsequently, you would have unique VLANs at each site despite having the same numbers.</P> <P>&nbsp;</P> <P>Next, you would manually add-on the stretch cluster configuration as follows:</P> <P>&nbsp;</P> <LI-CODE lang="powershell">Add-VMNetworkAdapter -ManagementOS -Name vStretch1 -SwitchName 'SwitchName' Add-VMNetworkAdapter -ManagementOS -Name vStretch2 -SwitchName 'SwitchName' Set-VMNetworkAdapterTeamMapping -ManagementOS -VMNetworkAdapterName vStretch1 -PhysicalNetAdapterName pNIC01 Set-VMNetworkAdapterTeamMapping -ManagementOS -VMNetworkAdapterName vStretch2 -PhysicalNetAdapterName pNIC02 Disable-NetAdapterRDMA -Name 'vEthernet (vStretch1)', 'vEthernet (vStretch2)' # Next, configure your IP Addresses and VLANs for the host vNICs </LI-CODE> <P>&nbsp;</P> <P>In the future, we hope to address all stretch configuration scenarios in Network ATC.</P> <P>&nbsp;</P> <H1>Can I use Network ATC with a switchless configuration?</H1> <P>Of course! Switchless is used as a simplified storage topology that reduces the cost and complexity of smaller solutions (e.g. 2-3 nodes); for more information please see <A href="#" target="_blank" rel="noopener">here</A>.</P> <P>&nbsp;</P> <P><STRONG>Two-node switchless configurations:</STRONG></P> <P>For two node configurations there are no changes required. Just ensure that the adapter names connected to one another are identical (see picture below). Consistent naming of adapters is always required for Network ATC and two-node switchless is no exception.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanCuomo_0-1632434165444.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/312531iC5B23EC3E782E9CE/image-size/large?v=v2&amp;px=999" role="button" title="DanCuomo_0-1632434165444.png" alt="DanCuomo_0-1632434165444.png" /></span></P> <P>&nbsp;</P> <P>The following command would deploy a two-node switchless configuration:</P> <P>&nbsp;</P> <LI-CODE lang="powershell">Add-NetIntent -Name IntentName -Storage -AdapterName pNIC01, pNIC02 -ClusterName Cluster01</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Three-node switchless configurations:</STRONG></P> <P>Three node configurations can also be deployed with Network ATC but require that you use the same VLANs for all links. Since this is a switchless configuration, using the same VLANs will not pose any stability problems despite not being a commonly recommended approach. In the future, we hope to have full support in Network ATC for this deployment model.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanCuomo_1-1632434186020.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/312532i6AF7D1FD673F7BC4/image-size/large?v=v2&amp;px=999" role="button" title="DanCuomo_1-1632434186020.png" alt="DanCuomo_1-1632434186020.png" /></span></P> <P>&nbsp;</P> <P>To deploy this switchless configuration, you add the StorageVLAN parameter to override the <A href="#" target="_blank" rel="noopener">default storage VLANs</A>.</P> <LI-CODE lang="powershell">Add-NetIntent -Name IntentName -Storage -StorageVLAN 711 -AdapterName pNIC01, pNIC02 -ClusterName Cluster01</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <H1>Can I prevent Network ATC from deploying three Traffic Classes?</H1> <P>&nbsp;</P> <P>No. Network ATC will only deploy a reliable and supported configuration. For Azure Stack HCI, this means three traffic classes are required including one for cluster heartbeats, one for RDMA (SMBDirect), and one for the Default traffic class (e.g. all other traffic including VMs).</P> <P>Without this, it’s highly likely that some traffic is “drowned out” by other traffic which could lead to application (e.g. VM) starvation or worse Storage Spaces Direct and Cluster crashes.</P> <P>&nbsp;</P> <H1>Can I prevent Network ATC from deploying Data Center Bridging?</H1> <P>&nbsp;</P> <P>If you specify the storage intent, DCB will always be configured and there’s no downside even if you’re not going to use it. If you do need to use it (e.g., “lossy” RDMA implementations), it will be there with the defined defaults and a portion of your work (the hosts) is already configured.</P> <P>&nbsp;</P> <P>Switchless configurations gain a significant advantage with this. DCB requires configuration at each endpoint where the data travels. In switchless configurations, the adapters in the intent are inclusive of all the endpoints. As a result, Network ATC enables strong Service Level Agreements (bandwidth guarantees) for network traffic.</P> <P>&nbsp;</P> <H1>What intent type does “Live Migration” use?</H1> <P>&nbsp;</P> <P>Live migration supports three different modes (Compression, TCP, and SMBDirect/RDMA). SMB Direct is always the recommendation whenever available and since all Azure Stack HCI systems have RDMA capable adapters, it’s likely <A href="https://gorovian.000webhostapp.com/?exam=t5/failover-clustering/optimizing-hyper-v-live-migrations-on-an-hyperconverged/ba-p/396609" target="_blank" rel="noopener">that you will configure live migration to use SMB</A>.</P> <P>&nbsp;</P> <P>So long as you do this, live migration will use adapters configured for the storage intent type.&nbsp;Microsoft <A title="Network ATC Definitions" href="#" target="_blank" rel="noopener">defines</A> the storage intent type as:</P> <UL> <LI><EM>Storage - adapters are used for SMB traffic including Storage Spaces Direct</EM></LI> </UL> <P>No additional configuration is required.</P> <P>&nbsp;</P> <H1>Can I change the vSwitch and host vNIC names?</H1> <P>&nbsp;</P> <P>No. Fundamentally, Network ATC is intending to “level-up” host configuration to a point where the unique host configuration artifacts aren’t something to concern yourself with. If we refer back to the introduction section of this article:</P> <P>&nbsp;</P> <P class="lia-indent-padding-left-30px"><EM>"…Network ATC doesn’t change what you deploy, just how you deploy it. You should no longer think about a virtual switch, host vNICs, adapter properties, etc. Instead, think about your outcomes and Network ATC take care of the rest."</EM></P> <P>&nbsp;</P> <P>When you provide intent information to Network ATC, it’s Network ATC’s job to keep them straight and ensure the names are unique, properly configured, etc.</P> <P>&nbsp;</P> <H1>What doesn’t Network ATC configure?</H1> <P>&nbsp;</P> <P>Note: The following list is specific to the current version (21H2) of Network ATC which is currently in preview and may change following this release. Please ensure you watch release notes, as we’ll aim to continually add new capabilities to Network ATC.</P> <P>&nbsp;</P> <P><STRONG>IP Addresses for Storage Adapters</STRONG>: In the 21H2 release, Storage Adapters are not automatically IP-Addressed following the intent configuration. You must manually, or through DHCP if available on the appropriate subnet, provide IP-Addresses for any created vNICs. In the next release, we’ll provide an automatic IP address capability.</P> <P>&nbsp;</P> <P><STRONG>Cluster Network Names</STRONG>: Network ATC does not configure or modify the existing cluster networks</P> <P>&nbsp;</P> <P><STRONG>Live Migration chosen networks</STRONG>: Live migration is not forced to use the storage intent network however, as mentioned above, it’s highly likely that it will be chosen naturally.</P> <P>&nbsp;</P> <P><STRONG>SMB Bandwidth Limits</STRONG>: This is now automatically configured by clustering.</P> <P>&nbsp;</P> <P><STRONG>Physical Network Switches</STRONG>: Network ATC is focused on ensuring the host configuration is consistent across all nodes in the cluster. You’ll still need to ensure that your network switches are configured properly. At the recent Azure Stack HCI Days 2021 event, we announced a new capability called Network HUD that will look to ensure that the system is functioning (operationally) as expected.</P> <P>&nbsp;</P> <H1>Summary</H1> <P>Network ATC provides several advantages including:</P> <UL> <LI>simplified deployment across the entire cluster</LI> <LI>ensuring the cluster is deployed with the current Microsoft validated best practices</LI> <LI>maintaining a consistent, cluster-wide configuration</LI> <LI>and remediation of misconfigurations (configuration drift)</LI> </UL> <P>It also allows you to stop focusing on virtual switches, team mappings, adapter properties; rather it lets you focus on your outcomes (intent).</P> <P>&nbsp;</P> <P>It’s great to see so many users of Network ATC in preview and we’re excited that you’ve chosen to give us your feedback. Please keep doing so as we strive to improve the product and ultimately your experience on Azure Stack HCI. Remember that Network ATC comes to production clusters as soon as 21H2 release is available – for all Azure Stack HCI subscribers.</P> <P>&nbsp;</P> <P>Dan “Network Automation Technology Champ” Cuomo</P> Mon, 27 Sep 2021 16:07:53 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/network-atc-common-preview-questions/ba-p/2780086 Dan Cuomo 2021-09-27T16:07:53Z What's QUIC? https://gorovian.000webhostapp.com/?exam=t5/networking-blog/what-s-quic/ba-p/2683367 <P>James Kehr here with the Windows Networking Escalation Engineering team. Today’s topic is the newly published QUIC protocol. This is a quick discussion, all puns intended, about why QUIC is important to the modern internet.</P> <P>&nbsp;</P> <H1>Back in my day…</H1> <P>&nbsp;</P> <P>Back in the old days, about the mid-1990’s, when the Internet was new and the Wild Wild Web was a better fit for WWW than the World Wide Web, Internet connections were really slow. Youngsters can’t appreciate just how amazingly slow the Internet was back then.</P> <P>&nbsp;</P> <P>A simple web page could take upwards of 1-2 minutes to load. Good video streaming, forget about it! High resolution video was originally 640x480, what we now call 480p or SD video, but most early video was 320x240. Even at the lower resolution you’d start the video, then immediately pause it so the video could buffer. Go refill your beverage, talk with friends, read a book, and about 5-10 minutes later you could start the video. And, if you were lucky, it wouldn’t pause to buffer before the video was done.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="JamesKehr_0-1629897112807.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/305641i8BF914054F99E121/image-size/large?v=v2&amp;px=999" role="button" title="JamesKehr_0-1629897112807.png" alt="JamesKehr_0-1629897112807.png" /></span></P> <P>&nbsp;</P> <P>This is normally the part when us old timers do the traditional joke of “And we liked it! We loved it!” but that would be a lie. We hated it. It was horrible. We were Internet addicts and we wanted to surf the net and not have to wait.</P> <P>&nbsp;</P> <P>Students of computer history will note that around this time a huge surge in computer advancements began. Processor and network speeds improved dramatically. Displays got larger with higher resolutions. New Internet connection technologies sprung up everywhere. Then the smartphone happened, and a second wave of innovation hit. To put this in perspective, the current&nbsp;<A title="CPU Benchmark - Qualcomm Snapdragon 888" href="#" target="_blank" rel="noopener">flagship</A> <A title="CPU Benchmark - Apple A14X Bionc" href="#" target="_blank" rel="noopener">smartphones</A>&nbsp;are about ten times (10x) more powerful than the <A title="Pingdom - The incredible growth of supercomputing performance, 1995 – 2010" href="#" target="_blank" rel="noopener">fastest super computer of 1995</A>.</P> <P>&nbsp;</P> <P>The driving force behind most of this innovation was, you guessed it, a fast and stable Internet experience.&nbsp;People today want to open a site and have the page loaded by the time their thumb clears the smartphone screen or the cursor clears the browser window. Our video needs to stream immediately in 4K with no buffering and no pixelation.</P> <P>&nbsp;</P> <P>Despite the massive leaps in technologies that have enabled the modern Internet experience, we have been using the same major Internet protocol since the beginning, our tried-and-true friend, TCP.</P> <P>&nbsp;</P> <H1>The Old Guard and the New Guard</H1> <P>&nbsp;</P> <P>Let me start by saying that TCP, or Transmission Control Protocol, isn’t go anywhere any time soon. It is, and will be, the dominant network protocol in the world for years to come. Which begs the question, why replace it?</P> <P>&nbsp;</P> <P>TCP was first developed in 1974 for the precursor to the Internet, ARPAnet. A little math will reveal that TCP will have its 50<SUP>th</SUP> birthday soon. ARPAnet was mainly used by military and educational institutions across closed circuits until ARPAnet became the commercial Internet in the early 1990’s. The TCP/IP suite of protocols used by ARPAnet were carried over to the Internet. TCP, as the resilient transport protocol, became the de facto standard for most Internet activities. Without TCP the Internet would not have worked quite right.</P> <P>&nbsp;</P> <P>This is normally where I say something like TCP is bad and needs to be replaced, but there isn’t anything necessarily bad about TCP because it’s a great protocol. The problem is that we can’t change TCP to make it better and meet the needs of modern Internet services that need to be crazy fast. The TCP protocol is simply too embedded to make any significant changes to it without the risk of breaking millions of devices.</P> <P>&nbsp;</P> <P>A lot of ideas have been thrown around to improve the Internet experience over the years but none of them stuck until QUIC, the Quick UDP Internet Connection protocol, was developed at Google using a unique concept. Rather than developing a completely new protocol that would require a massive upgrade to the Internet backbone the QUIC protocol extends an existing protocol, UDP. I say extend because QUIC is a transport layer protocol, not an application protocol, even though QUIC is transmitted inside the UDP segment. Think of it like this:</P> <P>&nbsp;</P> <P>UDP + QUIC = the transport layer</P> <P>&nbsp;</P> <P>QUIC uses UDP for ports and connectionless transport, then adds the resiliency of TCP, the security of TLS 1.3, sprinkles in a dash of commands and version control from protocols like SMB, and then mixes in a set of new protocol concepts and efficiencies to create something entirely unique in the protocol world.</P> <P>&nbsp;</P> <H1>Why Should I Care?</H1> <P>&nbsp;</P> <P>Big tech companies are excited about QUIC because it adds several changes that will help make Internet services better, especially edge services where every millisecond matters. By extension, this makes the Internet experience better for everyone. Here are some of the improvements in QUIC:</P> <P>&nbsp;</P> <H2>Encryption</H2> <P>&nbsp;</P> <P>QUIC 1.0 requires TLS 1.3 based encryption for all data. This makes data over QUIC inherently secure regardless of service.</P> <P>&nbsp;</P> <H2>Lower Connection Latency</H2> <P>&nbsp;</P> <P>QUIC doesn’t change the laws of physics, but it doesn’t have to wait for two handshakes (TCP then TLS) to complete a secure network connection. Connection setup takes fewer packets to complete than TCP + TLS and can be resilient after close. This means you start getting data faster the first time you connect to a service, and potentially faster the second time.</P> <P>&nbsp;</P> <H2>Connection Reuse</H2> <P>&nbsp;</P> <P>QUIC can reuse a session in two ways: streams and session tickets. A single QUIC session can have multiple simultaneous data streams. A server can also grant clients a session ticket that can be used to reconnect to a server without going through a full handshake. This reduces the number of client-server connects and allows fast, secure reconnections.</P> <P>&nbsp;</P> <H2>Connection Migration</H2> <P>&nbsp;</P> <P>This is one of the coolest features, in my opinion, as it allows a QUIC connection to survive an IP change. Let’s say you have a laptop with WiFi and LAN, and switch from a wired LAN connection to WiFi. With TCP the connections must be closed, and new connections opened using the WiFi IP address. With QUIC, the client can provide evidence to the server to prove who they are and continue with the existing connection on the new IP as if nothing changed. Making the mobile user experience more seamless.</P> <P>&nbsp;</P> <H2>Security</H2> <P>&nbsp;</P> <P>In addition to encryption, QUIC is built to prevent or lessen the impact of things like Denial of Service (DoS), replay, reflection, spoofing, and other types of attacks. QUIC can’t eliminate all attacks, but it does try to make it harder to successfully attack.</P> <P>&nbsp;</P> <H2>Version Control</H2> <P>&nbsp;</P> <P>QUIC can change and be adapted to meet new Internet needs because it has version control. QUIC 1.0 was published in May 2021 via RFC 9000, plus RFCs 8999, 9001, and 9002. Future versions of QUIC are free to change the protocol as needed. Computers can support multiple simultaneous versions of QUIC. This allows new version to fix and improve the QUIC protocol while supporting older implementations during a transition period. Over time the old versions can be dropped to keep QUIC secure and up to date for decades to come.</P> <P>&nbsp;</P> <H2>Extension Frames</H2> <P>&nbsp;</P> <P>People and companies can extend QUIC to meet their own needs. This is handled via QUIC extension frames, which can be public and potentially be added to a future version of QUIC, or private and used only for internal services. There are few rules that must be followed, but otherwise people are free to extend QUIC via custom frames as they see fit.</P> <P>&nbsp;</P> <H1>The All-Important Caveats…</H1> <P>&nbsp;</P> <P>Not everything is sunshine and flowers in QUIC land. QUIC 1.0 is brand spanking new, less than 4 months old at the time of writing. It’s going to take time for most devices to support QUIC. Modern browsers already support QUIC. The newest versions of Windows, Window 10 21Hx, Windows 11, and Windows Server 2022, have native QUIC support. Some older versions of Windows should see MsQuic support in early 2022. Apple has native QUIC support starting with Big Sur. Linux and FreeBSD currently need a QUIC driver installed or implemented in the user application, but future versions may have native support.</P> <P>&nbsp;</P> <P>Older OS builds are the problem here and may not have any QUIC capabilities outside the browser. And if there’s one thing that IT has taught me, many people are generally <A title="StatCounter - Desktop Windows Version Market Share Worldwide" href="#" target="_blank" rel="noopener">slow to upgrade</A>.</P> <P>&nbsp;</P> <P>Then there’s the learning curve when developing for QUIC. The QUIC protocol does things a bit differently than TCP and plain UDP. Learning the best way to leverage QUIC streams to your advantage will take some time and effort.</P> <P>&nbsp;</P> <P>Finally, there’s the traffic shaping problem. Some networks and ISPs will prefer TCP traffic over UDP, and rightly so. TCP is currently the de facto transport protocol on the Internet. This can, however, cause problems with QUIC performance in the short term when certain network congestion conditions are reached. A heavily congested network may drop UDP traffic to pass TCP traffic. This will change as backbone networks become more aware of QUIC. In the short term, there might be performance issues during peak hours.</P> <P>&nbsp;</P> <P>Does this give you license to ignore QUIC? It shouldn’t. Microsoft, Apple, Google, Amazon, Facebook, Cloudflare, and <A title="IETF - QUIC in the Internet industry" href="#" target="_blank" rel="noopener">other big tech companies</A> are already using QUIC because of its advantages over TCP. The tech world is ready for a QUIC switch, so you should be thinking about it, too.</P> <P>&nbsp;</P> <H1>MsQuic</H1> <P>&nbsp;</P> <P>No article would be complete without a small plug for your own product. Microsoft’s implementation of QUIC is called <A title="Github - MsQuic" href="#" target="_blank" rel="noopener">MsQuic</A>. It is an open source, cross platform project available to the world. Windows and Linux are fully supported as of version 1.5.0, with MacOS currently in the alpha stage. There is support in MsQuic for the <A title="Windows Blog - Announcing Windows Server vNext Preview Build 20201" href="#" target="_blank" rel="noopener">UDP offloading features</A> announced for Windows Server 2022, which will be available in Windows 11, as well. The UDP features do require a compatible network adapter. MsQuic <A title="Microsoft Networking Blog - Making MsQuic Blazing Fast" href="https://gorovian.000webhostapp.com/?exam=t5/networking-blog/making-msquic-blazing-fast/ba-p/2268963" target="_blank" rel="noopener">performance</A> is pretty impressive, too.</P> <P>&nbsp;</P> <P>I hope that helps you understand some of the pros and cons of QUIC, and why people are so excited about it. Keep an eye on the <A title="Microsoft Networking Blog on Tech Communities" href="https://gorovian.000webhostapp.com/?exam=t5/networking-blog/bg-p/NetworkingBlog" target="_blank" rel="noopener">Networking</A> and <A title="Storage at Microsoft Blog on Tech Communities" href="https://gorovian.000webhostapp.com/?exam=t5/storage-at-microsoft/bg-p/FileCAB" target="_blank" rel="noopener">Storage at Microsoft</A> Tech Communities for more content about QUIC and SMB over QUIC in the future.</P> Wed, 25 Aug 2021 18:30:20 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/what-s-quic/ba-p/2683367 JamesKehr 2021-08-25T18:30:20Z Enabling HTTP/3 support on Windows Server 2022 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/enabling-http-3-support-on-windows-server-2022/ba-p/2676880 <P><EM>Credit and thanks to Matthew Cox and Daniel Ring for implementation work</EM></P> <P>&nbsp;</P> <P>One of the new features Windows Server 2022 brings is native support for hosting HTTP/3 services. In this post, we will discuss how to enable it and how it can benefit web services.</P> <P>&nbsp;</P> <P>HTTP/3 is a major overhaul of HTTP with performance and security in mind. It uses QUIC as a transport (our HTTP server, http.sys, is using <A href="#" target="_blank" rel="noopener">msquic</A>) to gain the benefits of eliminated head of line blocking at the transport layer. This is a significant improvement over HTTP/2 which eliminated head of line blocking only at the HTTP layer with streams that allowed a single HTTP/2 connection to replace a set of HTTP/1.1 connections. HTTP/3 also benefits from many lessons learned in HTTP/2, such as simplifying the protocol by removing prioritization.</P> <P>&nbsp;</P> <P>The HTTP/3 standard is nearly complete; its <A href="#" target="_blank" rel="noopener">final publication</A> as an RFC is only waiting on formal process at this point. It is already supported by major browsers which means web services are ready to benefit from deploying it.</P> <P>&nbsp;</P> <P>One thing to note before proceeding: these instructions presume there were no changes made to the list of enabled TLS cipher suites on the Windows Server 2022 installation. If this is not the case, consult RFC9001 (“<A href="#" target="_blank" rel="noopener">Using TLS to Secure QUIC</A>”) and ensure there are some cipher suites in common between the server and its expected clients. HTTP/3 is built on QUIC which requires TLS 1.3. Turning off TLS 1.3 or disabling TLS 1.3 cipher suites will result in HTTP/3 deployment failures. See “<A href="#" target="_blank" rel="noopener">TLS Cipher Suites in Windows Server 2022</A>” to learn how to add cipher suites and which ones are enabled by default.</P> <P>&nbsp;</P> <P>HTTP/3 support is opt-in on Windows Server 2022 via a registry key named “EnableHttp3” with value 1 at “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\HTTP\Parameters”. Running this command from an elevated prompt will create the key:</P> <P>&nbsp;</P> <PRE>reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\HTTP\Parameters" /v EnableHttp3 /t REG_DWORD /d 1 /f</PRE> <P>&nbsp;</P> <P>Once this key is set, either restart the http.sys service or reboot Windows to apply the setting.</P> <P>&nbsp;</P> <P>It is likely the web service will need to advertise it is available over HTTP/3 as well using “Alt-Svc” headers in HTTP/2 responses (though this can also be done using HTTP/2 ALTSVC frames). This allows clients who connect over HTTP/2 to learn the service’s HTTP/3 endpoint and use that going forward. This is done by sending an HTTP/3 ALPN (“Application-layer Protocol Negotiation”) identifier with HTTP/2 responses advertising a specific version of HTTP/3 to use for future requests. Sending the ALTSVC frame &nbsp;can be done by http.sys. That can be enabled by setting the “EnableAltSvc” registry key with the command below. To apply the setting, restart http.sys or reboot Windows.</P> <P>&nbsp;</P> <PRE>"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\HTTP\Parameters" /v EnableAltSvc /t REG_DWORD /d 1 /f</PRE> <P>&nbsp;</P> <P>If all goes well, the service will start serving content over HTTP/3. To ensure this is happening, use Edge to verify the protocol used to serve the web request. Right click on the page, select “Inspect”, then select the “Network” tab. If only “h2” is being used in the “Protocol” column instead of “h3”, try refreshing the page to ensure the ALPN is being honored (the first request will use HTTP/2 which will then advertise HTTP/3 support to Edge for future requests).</P> Thu, 26 Aug 2021 15:40:57 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/enabling-http-3-support-on-windows-server-2022/ba-p/2676880 tojens 2021-08-26T15:40:57Z Windows Insiders gain new DNS over HTTPS controls https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-insiders-gain-new-dns-over-https-controls/ba-p/2494644 <P><EM>Credit and thanks to Alexandru Jercaianu, Vladimir Cernov, and Sam Yun for implementation work</EM></P> <P><EM>&nbsp;</EM></P> <P>Over the last year, we have been improving the DNS over HTTPS (DoH) functionality in the Windows DNS client. Now we are pleased to introduce you to the different features now available through the Windows Insider program.</P> <P>&nbsp;</P> <P>To start with, we want to note that the registry key controls documented in our <A href="https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-insiders-can-now-test-dns-over-https/ba-p/1381282" target="_blank" rel="noopener">original DoH testing blog post</A> are no longer applicable. As stated there, those instructions were time limited to the initial DoH test rollout. If you did ever set that key, please delete it then reboot your machine before proceeding with the rest of this blog post.</P> <P>&nbsp;</P> <P>Next, we will be reviewing the new configuration behavior, how Windows will know if a DNS server supports DoH, and what our next steps are in advancing encrypted DNS discovery.</P> <P>&nbsp;</P> <H2>UI</H2> <P>The first control you should try out is the new UI fields in the Settings app, <A href="#" target="_blank" rel="noopener">originally announced on the Insider blog</A>. When Windows knows a given DNS server’s IP address has a corresponding DoH server, it will unlock a dropdown that lets you decide whether to require encryption always be used, use encryption but fall back to plain-text DNS when encryption fails, or not to use encryption (the default value).</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tojens_0-1624918166884.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/292154iE9457EAC2D187477/image-size/medium?v=v2&amp;px=400" role="button" title="tojens_0-1624918166884.png" alt="tojens_0-1624918166884.png" /></span></P> <P>&nbsp;</P> <H2>GPO</H2> <P>For enterprise administrators, we have provided a new GPO for controlling DoH behavior. This will allow the use of DoH to be allowed, required, or prohibited system-wide.</P> <UL> <LI>Allowed will defer the use of DoH to local settings available in the UI per network adapter.</LI> <LI>Required will prevent the use of configured DNS servers if they do not support DoH and will disable fallback to plain-text DNS.</LI> <LI>Prohibited will prevent any local DoH settings from taking effect, ensuring Windows functions as it did before the DoH client using plain-text DNS only.</LI> </UL> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tojens_1-1624918166897.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/292155i9188FB2CB0E3B796/image-size/medium?v=v2&amp;px=400" role="button" title="tojens_1-1624918166897.png" alt="tojens_1-1624918166897.png" /></span></P> <P>&nbsp;</P> <H2>NRPT</H2> <P>The <A href="#" target="_blank" rel="noopener">Name Resolution Policy Table</A> (NRPT) allows administrators to specify rules for name resolution by namespace. For example, you can create an NRPT rule that specifies all queries for “*.microsoft.com” must be sent to a specific DNS server.</P> <P>&nbsp;</P> <P>If Windows knows that a DNS server provided in an NRPT rule supports DoH (see the next section for how this works), then the traffic affected by the NRPT rule will inherit the benefits of using DoH. This allows admins who want to use DoH for some namespaces and not others to configure that behavior.</P> <P>&nbsp;</P> <H2>Knowing a server supports DoH</H2> <P>All these mechanisms rely on Windows already knowing a given DNS server IP address supports DoH. We ship a few definitions of known DoH servers in Windows:</P> <P>&nbsp;</P> <TABLE> <TBODY> <TR> <TD> <P><STRONG>Server Owner</STRONG></P> </TD> <TD> <P><STRONG>Server IP addresses</STRONG></P> </TD> </TR> <TR> <TD> <P><STRONG>Cloudflare</STRONG></P> </TD> <TD> <P>1.1.1.1</P> <P>1.0.0.1</P> <P>2606:4700:4700::1111</P> <P>2606:4700:4700::1001</P> </TD> </TR> <TR> <TD> <P><STRONG>Google</STRONG></P> </TD> <TD> <P>8.8.8.8</P> <P>8.8.4.4</P> <P>2001:4860:4860::8888</P> <P>2001:4860:4860::8844</P> </TD> </TR> <TR> <TD> <P><STRONG>Quad9</STRONG></P> </TD> <TD> <P>9.9.9.9</P> <P>149.112.112.112</P> <P>2620:fe::fe</P> <P>2620:fe::fe:9</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>Other definitions need to be added using the netsh command. To start with, you can check to see what DoH server definitions we already know by retrieving them:</P> <H6>&nbsp;</H6> <H6>Using netsh</H6> <PRE>netsh dns show encryption</PRE> <H6>Using PowerShell</H6> <PRE>Get-DnsClientDohServerAddress</PRE> <P>&nbsp;</P> <P>Then you can add another server definition to the list and ensure it never falls back to plain-text DNS:</P> <H6>&nbsp;</H6> <H6>Using netsh</H6> <PRE>netsh dns add encryption server=&lt;resolver-IP-address&gt; dohtemplate=&lt;resolver-DoH-template&gt; autoupgrade=yes udpfallback=no</PRE> <H6>Using PowerShell</H6> <PRE>Add-DnsClientDohServerAddress -ServerAddress '&lt;resolver-IP-address&gt;' -DohTemplate '&lt;resolver-DoH-template&gt;' -AllowFallbackToUdp $False -AutoUpgrade $True</PRE> <P>&nbsp;</P> <P>If you prefer to allow fallback so that when encryption fails you can still make DNS queries, you can run the same commands with the fallback flag toggled to add a new server:</P> <H6>&nbsp;</H6> <H6>Using netsh</H6> <PRE>netsh dns add encryption server=&lt;resolver-IP-address&gt; dohtemplate=&lt;resolver-DoH-template&gt; autoupgrade=yes udpfallback=yes</PRE> <H6>Using PowerShell</H6> <PRE>Add-DnsClientDohServerAddress -ServerAddress '&lt;resolver-IP-address&gt;' -DohTemplate '&lt;resolver-DoH-template&gt;' -AllowFallbackToUdp $True -AutoUpgrade $True</PRE> <P>&nbsp;</P> <P>The `-AutoUpgrade` and `-AllowFallbackToUdp` flags together represent the values present in the Setting app per-server dropdown. If for some reason you want to add these DoH server definitions but leave them to use unencrypted DNS for now, you can set the `-AutoUpgrade` flag to false instead of true as in the examples above.</P> <P>&nbsp;</P> <P>If you want to edit an existing list entry rather than adding a new one, you can use the `Set-DnsClientDohServerAddress` cmdlet in place of the `Add-DnsClientDohServerAddress` cmdlet.</P> <P>&nbsp;</P> <P>It would be easier for users and administrators if we allowed a DoH server to have its IP address determined by resolving its domain name. However, we have chosen not to allow that. Supporting this would mean that before a DoH connection could we established, we would have to first send a plain-text DNS query to bootstrap it. This means a node on the network path could maliciously modify or block the DoH server name query. Right now, the only way we can avoid this is to have Windows know in advance the mapping between IP addresses and DoH templates.</P> <P>&nbsp;</P> <H2>Coming up next</H2> <P>Going forward, we want to be able to directly discover DoH server configuration from the DNS server. This would mean DoH servers could be used without having to include it in Windows or manually configure the IP address to DoH template mapping. We are currently contributing to two proposals in the IETF ADD WG to enable this: <A href="#" target="_blank" rel="noopener">Discovery of Designated Resolvers</A> (DDR) and <A href="#" target="_blank" rel="noopener">Discovery of Network-designated Resolvers</A> (DNR). We look forward to updating you with our first tests in supporting DoH discovery!</P> Wed, 30 Jun 2021 15:14:06 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-insiders-gain-new-dns-over-https-controls/ba-p/2494644 tojens 2021-06-30T15:14:06Z Network ATC in Preview on Azure Stack HCI https://gorovian.000webhostapp.com/?exam=t5/networking-blog/network-atc-in-preview-on-azure-stack-hci/ba-p/2393233 <P>As you may be aware, Microsoft <A href="https://gorovian.000webhostapp.com/?exam=t5/azure-stack-blog/starting-a-new-era-for-azure-stack-hci-and-a-sneak-peek-at-what/ba-p/1975029" target="_blank" rel="noopener">announced the general availability</A> of the Azure-connected <STRONG>H</STRONG>yper-<STRONG>C</STRONG>onverged <STRONG>I</STRONG>nfrastructure, Azure Stack HCI. Previously Azure Stack HCI was built off Windows Server which is a great general-purpose operating system that allows you to run your virtualized workloads. The new and improved Azure Stack HCI OS however is a purpose-built, cloud-connected infrastructure, intended to run your Azure Stack HCI workloads in the modern data center (for more information, start <A href="https://gorovian.000webhostapp.com/?exam=t5/azure-stack-blog/starting-a-new-era-for-azure-stack-hci-and-a-sneak-peek-at-what/ba-p/1975029" target="_blank" rel="noopener">here</A>, then go <A href="#" target="_blank" rel="noopener">here</A>, then see what’s coming next over <A href="#" target="_blank" rel="noopener">here</A>).</P> <P>&nbsp;</P> <P>Azure Stack HCI is a subscription service that, like Office 365 or Windows 10, continually get free updates. The next update available to Azure Stack HCI subscribers will be 21H2 which is in preview right now! With this update comes a new feature called Network ATC, which simplifies the deployment and management of networking on your HCI hosts.</P> <P>&nbsp;</P> <P>If you’ve deployed Azure Stack HCI previously, you know that network deployment can pose a significant challenge. You might be asking yourself:</P> <UL> <LI>How do I configure or optimize my adapter?</LI> <LI>Did I configure the virtual switch, VMMQ, RDMA, etc. correctly?</LI> <LI>Are all nodes in the cluster the same?</LI> <LI>Are we following the best practice deployment models?</LI> <LI>(And if something goes wrong) What changed!?</LI> </UL> <P>So, what does Network ATC actually set out to solve? Network ATC can help:</P> <UL> <LI><STRONG>Reduce</STRONG> host networking deployment <STRONG>time</STRONG>, <STRONG>complexity</STRONG>, and<STRONG> errors</STRONG></LI> <LI><STRONG>Deploy</STRONG> the latest Microsoft validated and supported <STRONG>best practices</STRONG></LI> <LI><STRONG>Ensure</STRONG> configuration <STRONG>consistency across the cluster</STRONG></LI> <LI><STRONG>Eliminate</STRONG> configuration <STRONG>drift</STRONG></LI> </UL> <P>Network ATC does this through some new concepts, namely “intent-based” deployment. If you tell Network ATC how you want to use an adapter, it will translate, deploy, and manage the needed configuration across all nodes in the cluster. For more details, please see our <A href="#" target="_blank" rel="noopener">Network ATC preview documentation</A>.</P> <P>&nbsp;</P> <P>Let’s take a quick look at Network ATC in action. In this video, we deploy the host networking configuration across an 8-node cluster, each with two physical adapters (16 total) – <STRONG>with a single command</STRONG>. By the end, these two physical adapters are ready to run Storage Spaces Direct (storage intent) and provide the compute infrastructure (compute intent) needed run your virtual machines all in under 5 minutes.</P> <P><LI-VIDEO vid="https://youtu.be/Z8UO6EGnh0k" align="center" size="medium" width="400" height="225" uploading="false" thumbnail="https://i.ytimg.com/vi/Z8UO6EGnh0k/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <P>One of the greatest benefits of Network ATC is that it remediates configuration drift. Have you ever wondered “who changed that?” or said, “we must have missed this node.” You’ll never worry about this again with Network ATC at the helm. Expanding the cluster to add new nodes? Just install the service on the new node, join the cluster and rest assured that in a few minutes, the expected configuration will be deployed.</P> <P>&nbsp;</P> <P>As you can see, Network ATC greatly reduces the deployment time, complexity, and errors with host networking for Azure Stack HCI as it manages the lifecycle of the cluster. Whether you’re building out a new cluster, expanding the cluster, or just want the peace-of-mind that the network configuration is in steady-state, Network ATC can make this a breeze.</P> <P>&nbsp;</P> <P>Please take a look at our preview documentation, give Network ATC a try, and as always, let us know what you think! Next, enjoy your newfound free time now that Network ATC is managing the host networking!</P> <P>&nbsp;</P> <P>Thanks for reading</P> <P>Dan “Network ATC helps me sleep at night” Cuomo</P> Sat, 29 May 2021 16:16:12 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/network-atc-in-preview-on-azure-stack-hci/ba-p/2393233 Dan Cuomo 2021-05-29T16:16:12Z Algorithmic improvements boost TCP performance on the Internet https://gorovian.000webhostapp.com/?exam=t5/networking-blog/algorithmic-improvements-boost-tcp-performance-on-the-internet/ba-p/2347061 <P class="lia-align-justify"><SPAN data-contrast="auto">Improved network performance over the&nbsp;Internet&nbsp;is&nbsp;essential for edge devices connecting to the cloud.&nbsp;Last mile performance impacts user perceived latencies and is an area of focus&nbsp;for&nbsp;our online services like&nbsp;M365, SharePoint, and Bing.&nbsp;Although&nbsp;the next generation transport&nbsp;QUIC is on the horizon,&nbsp;TCP is the dominant transport protocol today. Improvements made to TCP’s&nbsp;performance directly improve response times&nbsp;and download/upload speeds.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">The Internet last mile and&nbsp;wide&nbsp;area&nbsp;networks (WAN)&nbsp;are characterized by&nbsp;high latency&nbsp;and&nbsp;a long tail of&nbsp;networks&nbsp;which suffer from&nbsp;packet loss and reordering.&nbsp;Higher latency, packet loss, jitter, and reordering, all impact TCP’s performance.&nbsp;Over&nbsp;the past&nbsp;few&nbsp;years, we have&nbsp;invested heavily in&nbsp;improving&nbsp;TCP&nbsp;WAN performance and&nbsp;engaged&nbsp;with&nbsp;the&nbsp;IETF standards&nbsp;community&nbsp;to help advance the state of the art.&nbsp;In this blog we&nbsp;will walk through&nbsp;our journey and show how we&nbsp;made&nbsp;big&nbsp;strides in&nbsp;improving&nbsp;performance&nbsp;between Windows Server 2016 and&nbsp;the upcoming&nbsp;Windows Server 2022.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><STRONG>Introduction&nbsp;</STRONG></H2> <P class="lia-align-justify"><SPAN data-contrast="auto">There are two important building blocks of TCP&nbsp;which govern its performance&nbsp;over the Internet:&nbsp;Congestion&nbsp;Control and&nbsp;Loss&nbsp;Recovery. The&nbsp;goal of&nbsp;congestion control is to determine the&nbsp;amount of data that can be&nbsp;safely&nbsp;injected into the network&nbsp;to maintain good performance and&nbsp;minimize&nbsp;congestion. Slow Start is the initial stage of congestion control where TCP ramps up&nbsp;its speed&nbsp;quickly until a congestion signal&nbsp;(packet loss, ECN, etc.)&nbsp;occurs.&nbsp;The&nbsp;steady state&nbsp;Congestion Avoidance stage&nbsp;follows Slow Start&nbsp;where different TCP congestion control algorithms use different approaches to adjust the amount of data in-flight.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">Loss&nbsp;Recovery is the process to&nbsp;detect and&nbsp;recover&nbsp;from packet&nbsp;loss during transmission.&nbsp;TCP can&nbsp;infer&nbsp;that&nbsp;a segment is lost by looking at the&nbsp;ACK&nbsp;feedback from the receiver,&nbsp;and retransmit&nbsp;any&nbsp;segments&nbsp;inferred&nbsp;lost. When loss recovery fails, TCP&nbsp;uses&nbsp;retransmission timeout (RTO, usually 300ms in WAN scenarios) as the last resort to retransmit the lost segments.&nbsp;When the RTO&nbsp;timer&nbsp;fires, TCP&nbsp;returns&nbsp;to&nbsp;&nbsp;Slow&nbsp;Start from the first unacknowledged segment.&nbsp;This&nbsp;long wait period&nbsp;and the subsequent congestion response&nbsp;significantly&nbsp;impacts&nbsp;performance,&nbsp;so&nbsp;optimizing&nbsp;Loss Recovery algorithms&nbsp;enhances&nbsp;throughput and&nbsp;reduces&nbsp;latency.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none">Improving Slow Start:&nbsp;HyStart++</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P class="lia-align-justify"><SPAN class="TextRun SCXW33734733 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SCXW33734733 BCX8">We<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">determined that the traditional slow start algorithm</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>is overshooting the optimum rate and likely to hit a</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">n</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>RTO during slow start</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>due to massive packet loss</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">.<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">We explor</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">ed</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>the use of</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>an algorithm<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun ContextualSpellingAndGrammarErrorV2 SCXW33734733 BCX8">called</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN></SPAN></SPAN><A class="Hyperlink SCXW33734733 BCX8" href="#" target="_blank" rel="noreferrer noopener"><SPAN class="TextRun Underlined SCXW33734733 BCX8" data-contrast="none"><SPAN class="NormalTextRun SCXW33734733 BCX8" data-ccp-charstyle="Hyperlink">Hy</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8" data-ccp-charstyle="Hyperlink">S</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8" data-ccp-charstyle="Hyperlink">tart</SPAN></SPAN></A><SPAN class="TextRun SCXW33734733 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>to mitigate this problem.<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">Hy</SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">S</SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">tart</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>triggers an exit from<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">Slow Start</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>when the connection latency is observed to increase.</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">However, we found that sometimes false positives cause a</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>premature<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">exit from slow start</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">,</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>limiting performance.<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">We<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">developed<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">a variant of</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">Hy</SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">S</SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">tart</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>to mitigate premature Slow Start exit in networks with delay jitter</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">:<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">w</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">hen<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8 DefaultHighlightTransition">Hy</SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8 DefaultHighlightTransition">S</SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8 DefaultHighlightTransition">tart</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">is triggered</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">,<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">rather than going to the Congestion Avoidance stage we use<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">LSS (Limited Slow Start)</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">, an increase algorithm that is less aggressive than Slow Start but more aggressive than<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun CommentStart SCXW33734733 BCX8 CommentHighlightPipeRest CommentHighlightRest">Congestion Avoidance</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8 CommentHighlightRest">.</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8 CommentHighlightRest"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8 CommentHighlightPipeRest">W</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">e<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">have</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>published<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">our ongoing work</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">on the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SpellingErrorV2 SCXW33734733 BCX8">HyStart</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>algorithm<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">as</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>an IET</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">F</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">d</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">raft</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8"><SPAN>&nbsp;</SPAN>adopted by the TCPM working group</SPAN><SPAN class="NormalTextRun SCXW33734733 BCX8">:<SPAN>&nbsp;</SPAN></SPAN></SPAN><A class="Hyperlink SCXW33734733 BCX8" href="#" target="_blank" rel="noreferrer noopener"><SPAN class="TextRun Underlined SCXW33734733 BCX8" data-contrast="none"><SPAN class="NormalTextRun SCXW33734733 BCX8" data-ccp-charstyle="Hyperlink">HyStart++: Modified Slow Start for TCP (ietf.org)</SPAN></SPAN></A><SPAN class="TextRun SCXW33734733 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SCXW33734733 BCX8">.</SPAN></SPAN><SPAN class="EOP SCXW33734733 BCX8" data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none">Loss recovery performance:&nbsp;Proportional&nbsp;Rate&nbsp;Reduction</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P class="lia-align-justify"><SPAN class="TextRun SCXW173536440 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SpellingErrorV2 SCXW173536440 BCX8">HyStart</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">help</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">s prevent the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">overshoot problem<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">so that</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">we enter loss recovery in Slow Start with fewer packet losses. However,<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun CommentStart CommentHighlightPipeRest CommentHighlightRest SCXW173536440 BCX8">loss recovery</SPAN><SPAN class="NormalTextRun CommentHighlightPipeRest SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN>itself might also incur packet losses</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN>if we<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">retransmit</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">in large</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">bursts</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">.</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">Proportional Rate Reduction (</SPAN></SPAN><A class="Hyperlink SCXW173536440 BCX8" href="#" target="_blank" rel="noreferrer noopener"><SPAN class="TextRun Underlined SCXW173536440 BCX8" data-contrast="none"><SPAN class="NormalTextRun SCXW173536440 BCX8" data-ccp-charstyle="Hyperlink">PRR</SPAN></SPAN></A><SPAN class="TextRun Underlined SCXW173536440 BCX8" data-contrast="none"><SPAN class="NormalTextRun SCXW173536440 BCX8" data-ccp-charstyle="Hyperlink">)</SPAN></SPAN><SPAN class="TextRun SCXW173536440 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun ContextualSpellingAndGrammarErrorV2 SCXW173536440 BCX8">is</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN>a loss recovery algorithm which</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">accurately<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">adjusts the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">number of bytes in</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">flight</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN>through</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">out</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">the entire loss<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">recovery<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">period<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">such that at the end of recovery it will be as close as possible to the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">congestion window</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">. We<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">enabled</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN>PRR<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">by default<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">in<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">Windows<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">10<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">May 2019 Update</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8"><SPAN>&nbsp;</SPAN>(19H1)</SPAN><SPAN class="NormalTextRun SCXW173536440 BCX8">.&nbsp;</SPAN></SPAN><SPAN class="EOP SCXW173536440 BCX8" data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none">Re-implementing&nbsp;TCP RACK: Time-based loss recovery</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P class="lia-align-justify"><SPAN data-contrast="auto">After implementing&nbsp;PRR and&nbsp;HyStart, we&nbsp;still&nbsp;noticed&nbsp;that&nbsp;we tend to consistently hit an RTO during loss recovery&nbsp;if&nbsp;many&nbsp;packets&nbsp;are lost&nbsp;in one congestion&nbsp;window.&nbsp;After looking at the traces, we figured out that it’s lost retransmits that&nbsp;cause TCP to&nbsp;time&nbsp;out. The&nbsp;</SPAN><A href="#" target="_blank"><SPAN data-contrast="none">RACK</SPAN></A><SPAN data-contrast="auto">&nbsp;implementation&nbsp;shipped in&nbsp;Server 2016&nbsp;is&nbsp;unable to recover lost retransmits.&nbsp;A fully RFC-compliant&nbsp;RACK&nbsp;implementation&nbsp;(which can recover lost retransmits)&nbsp;requires per-segment&nbsp;state&nbsp;tracking&nbsp;but&nbsp;in&nbsp;Server 2016,&nbsp;per-segment&nbsp;state&nbsp;is&nbsp;not stored.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">In&nbsp;Server 2016,&nbsp;we built&nbsp;a&nbsp;simple&nbsp;circular-array based data structure to track the send time of&nbsp;blocks of data in one congestion window.&nbsp;The RACK implementation we had with this data structure has many limitations,&nbsp;including&nbsp;being&nbsp;unable to&nbsp;recover&nbsp;lost retransmits.&nbsp;During the development of&nbsp;Windows 10&nbsp;May 2020 Update, we&nbsp;built&nbsp;per-segment&nbsp;state tracking&nbsp;for TCP&nbsp;and in&nbsp;Server&nbsp;2022, we shipped&nbsp;a&nbsp;new RACK implementation which&nbsp;can&nbsp;recover lost retransmits.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">(Note that&nbsp;Tail Loss Probe (TLP)&nbsp;which is&nbsp;part of RACK/TLP RFC&nbsp;and helps recover faster from tail losses&nbsp;is also implemented and&nbsp;enabled&nbsp;by default since Windows Server 2016.)</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none">Improving resilience&nbsp;to network reordering</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P class="lia-align-justify"><SPAN class="TextRun SCXW87811367 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SCXW87811367 BCX8">Last year,</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>Dropbox and Samsung reported<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">to us that</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>Windows<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">TCP<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">had poor<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">upload<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">perform</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">ance</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>in their networks</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>due to<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">network reordering.<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">We bumped up the priority of reordering resilience<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">i</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">n<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">Windows<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">version currently<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">under</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>development</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">, w</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">e</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>have completed<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">our</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">RACK</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">implementation</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">which</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>is<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">now<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">fully compliant with the RFC.</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">Dropbox and Samsung</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">confirmed that<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">they no longer observed<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">upload performance<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">problem</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">s</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">with this new implementation</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">.</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">You can find how we collaborated with<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">Dropbox<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun ContextualSpellingAndGrammarErrorV2 SCXW87811367 BCX8">engineers</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN></SPAN></SPAN><A class="Hyperlink SCXW87811367 BCX8" href="#" target="_blank" rel="noreferrer noopener"><SPAN class="TextRun Underlined SCXW87811367 BCX8" data-contrast="none"><SPAN class="NormalTextRun SCXW87811367 BCX8" data-ccp-charstyle="Hyperlink">here</SPAN></SPAN></A><SPAN class="TextRun SCXW87811367 BCX8" data-contrast="auto"><SPAN class="NormalTextRun SCXW87811367 BCX8">.</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8"><SPAN>&nbsp;</SPAN>In our<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">automated<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">WAN performance<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">tests</SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">, we also found that the<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">throughput<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">in reordering test cases<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">improved<SPAN>&nbsp;</SPAN></SPAN><SPAN class="NormalTextRun SCXW87811367 BCX8">more than 10x.</SPAN></SPAN><SPAN class="EOP SCXW87811367 BCX8" data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none">Benchmarks</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P class="lia-align-justify"><SPAN data-contrast="auto">To measure the performance improvements, we set up&nbsp;a&nbsp;WAN environment&nbsp;by&nbsp;creating two NICs&nbsp;on&nbsp;a&nbsp;machine&nbsp;and connecting the two NICs with an emulated link where bandwidth, round trip time,&nbsp;random loss, reordering and jitter can be emulated.&nbsp;We&nbsp;did performance benchmarks&nbsp;on this testbed&nbsp;for Server 2016, Server 2019 and&nbsp;Server 2022&nbsp;using an A/B&nbsp;testing framework&nbsp;we previously built&nbsp;where you&nbsp;can&nbsp;easily automate&nbsp;testing and&nbsp;data analysis.&nbsp;We used the&nbsp;current&nbsp;Windows build&nbsp;21359&nbsp;for&nbsp;Server 2022&nbsp;in the benchmarks since we plan to backport all TCP perf improvement changes to Server 2022 soon.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">Let’s&nbsp;look&nbsp;at non-reordering scenarios first. We emulated 100Mbps bandwidth and tested the three OS&nbsp;versions under four different round trip times&nbsp;(25ms, 50ms, 100ms,&nbsp;200ms)&nbsp;and two different flow sizes&nbsp;(32MB,&nbsp;128MB).&nbsp;The bottleneck buffer size was set to 1 BDP.&nbsp;The results are averaged&nbsp;over&nbsp;10 iterations.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="huanyi_0-1620772775483.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/280125iBD65055368595128/image-size/large?v=v2&amp;px=999" role="button" title="huanyi_0-1620772775483.png" alt="huanyi_0-1620772775483.png" /></span></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">Server&nbsp;2022&nbsp;is the&nbsp;clear&nbsp;winner in all categories&nbsp;because&nbsp;RACK significantly&nbsp;reduces&nbsp;RTOs occurring&nbsp;during loss recovery.&nbsp;Goodput is improved&nbsp;by&nbsp;up to 60%&nbsp;(200ms case).&nbsp;Server 2019&nbsp;did&nbsp;well in relatively high latency cases (&gt;= 50ms). However,&nbsp;for&nbsp;25ms RTT, Server&nbsp;2016 outperformed&nbsp;Server&nbsp;2019.&nbsp;After digging into the traces, we noticed that&nbsp;the&nbsp;Server 2016 receive window tuning algorithm is more conservative than the one in Server 2019 and&nbsp;it happened to throttle the sender,&nbsp;indirectly preventing&nbsp;the&nbsp;overshoot problem.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P class="lia-align-justify">&nbsp;</P> <P class="lia-align-justify"><SPAN data-contrast="auto">Now let’s look at reordering&nbsp;scenarios.&nbsp;Here’s how we emulate network reordering: we set a&nbsp;probability of reordering per packet. Once a packet is chosen&nbsp;to be&nbsp;reordered, it’s delayed by a specified amount of time instead of the configured RTT.&nbsp;We tested 1% reordering rate and 5ms reordering delay. Server 2016 and Server 2019&nbsp;achieved&nbsp;extremely low goodput due to lack of&nbsp;reordering resilience.&nbsp;In&nbsp;Server 2022, the new RACK implementation&nbsp;avoided most unnecessary&nbsp;loss recoveries and achieved reasonable performance.&nbsp;We can see&nbsp;goodput is up over 40x&nbsp;in the 128MB with&nbsp;200ms RTT case.&nbsp;In&nbsp;the&nbsp;other&nbsp;cases, we are seeing&nbsp;at least&nbsp;5x&nbsp;goodput improvement.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="reo.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/280146i0AB70691990AE11D/image-size/large?v=v2&amp;px=999" role="button" title="reo.png" alt="reo.png" /></span></SPAN></H2> <P class="lia-align-justify">&nbsp;</P> <H2 class="lia-align-justify" aria-level="2"><SPAN data-contrast="none">Next Steps</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></H2> <P class="lia-align-justify"><SPAN data-contrast="auto">We have come a long way i</SPAN><SPAN>n&nbsp;</SPAN><SPAN>i</SPAN><SPAN data-contrast="auto">mproving&nbsp;Windows&nbsp;TCP performance&nbsp;on the Internet.&nbsp;However, there are still several issues&nbsp;that&nbsp;we will&nbsp;need to&nbsp;solve in future releases.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <UL class="lia-align-justify"> <LI data-leveltext="" data-font="Symbol" data-listid="34" aria-setsize="-1" data-aria-posinset="1" data-aria-level="1"><SPAN data-contrast="auto">We are unable to&nbsp;measure specific&nbsp;performance improvements&nbsp;from PRR&nbsp;in&nbsp;the&nbsp;A/B tests. This&nbsp;needs more investigation.</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="34" aria-setsize="-1" data-aria-posinset="2" data-aria-level="1"><SPAN data-contrast="auto">We&nbsp;have found&nbsp;issues with&nbsp;HyStart++&nbsp;in networks with jitter.&nbsp;So&nbsp;we&nbsp;are working on making the algorithm&nbsp;more&nbsp;resilient to&nbsp;jitter.&nbsp;</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> <LI data-leveltext="" data-font="Symbol" data-listid="34" aria-setsize="-1" data-aria-posinset="3" data-aria-level="1"><SPAN data-contrast="auto">The reassembly queue limit&nbsp;(the&nbsp;max&nbsp;number of&nbsp;discontiguous&nbsp;data blocks&nbsp;allowed&nbsp;in receive queue),&nbsp;turns out to be another factor that affects our WAN performance.&nbsp;After&nbsp;this limit is reached, the receiver&nbsp;discards any&nbsp;subsequent&nbsp;out of order data segments&nbsp;until in-order data fills&nbsp;the gaps.&nbsp;When&nbsp;these&nbsp;segments are discarded, the receiver can only send back&nbsp;SACKs&nbsp;not&nbsp;carrying new information and make the sender stall.</SPAN><SPAN data-ccp-props="{&quot;134233279&quot;:true,&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></LI> </UL> <P class="lia-align-justify"><SPAN data-contrast="auto">-- Windows TCP&nbsp;Dev Team&nbsp;(Matt Olson, Praveen&nbsp;Balasubramanian, Yi Huang)</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> Wed, 12 May 2021 02:56:54 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/algorithmic-improvements-boost-tcp-performance-on-the-internet/ba-p/2347061 huanyi 2021-05-12T02:56:54Z Making MsQuic Blazing Fast https://gorovian.000webhostapp.com/?exam=t5/networking-blog/making-msquic-blazing-fast/ba-p/2268963 <P>It’s been a year since we open sourced <A href="#" target="_blank" rel="noopener">MsQuic</A> and a lot has happened since then, both in the industry (QUIC v1 in the <A href="#" target="_blank" rel="noopener">final stages</A>) and in MsQuic. As far as MsQuic goes, we’ve been hard at work adding new features, improving stability and more; but improving performance has been one of our primary ongoing efforts. MsQuic recently passed the 1000<SUP>th</SUP> commit mark, with nearly 200 of those for <A href="#" target="_blank" rel="noopener">PRs related to performance</A> work. We’ve improved single connection upload speeds from 1.67 Gbps in July 2020 to as high as 7.99 Gbps with the latest builds*.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="nibanks_1-1618423631950.png" style="width: 980px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/272890i2D88A8B728F19DDD/image-dimensions/980x501?v=v2" width="980" height="501" role="button" title="nibanks_1-1618423631950.png" alt="nibanks_1-1618423631950.png" /></span></P> <P><FONT size="2">* Windows Preview OS builds; User-mode using <A href="#" target="_blank" rel="noopener">Schannel</A>; and server-class hardware with <A href="#" target="_blank" rel="noopener">USO</A>.</FONT><BR /><FONT size="2">** x-axis above reflects the number of Git commits back from HEAD.</FONT></P> <P>&nbsp;</P> <H1>Defining Performance</H1> <P>&nbsp;</P> <P>“Performance” means a lot of different things to different people. When we talk with Windows file sharing (SMB), it’s always about single connection, bulk throughput. <EM>How many gigabits per second can you upload or download?</EM> With HTTP, more often it’s about the maximum number of requests per second (RPS) a server can handle, or the per-request latency values. <EM>How many microseconds of latency do you add to a request?</EM> For a general purpose QUIC solution, all of these are important to us. But even these different scenarios can have ambiguity in their definition. That’s why we’re working to <A href="#" target="_blank" rel="noopener">standardize</A> the process by which we measure the various performance scenarios. Not only does this provide a very clear message of what exactly is being measured and how, but it has also allowed for us to do cross-implementation performance testing. Four other implementations (that we know of) have implemented the “perf” protocol we’ve defined.</P> <P>&nbsp;</P> <H1>Performance-First Design</H1> <P>&nbsp;</P> <P>As already mentioned above, performance has been a primary focus of our efforts. Since the very start of our work on QUIC, we’ve had both HTTP and SMB scenarios driving pretty much every design decision we’ve made. It comes down to the following: <EM>The design must be both performant for a single operation and highly parallelizable for many.</EM> For SMB, a few connections must be able to achieve extremely high throughput. On the other hand, HTTP needs to support tens of thousands of parallel connections/requests with very low latency.</P> <P>&nbsp;</P> <P>This design initially led to significant improvements at the UDP layer. We added support for UDP <A href="#" target="_blank" rel="noopener">send segmentation</A> and <A href="#" target="_blank" rel="noopener">receive coalescing</A>. Together, these interfaces allow a user mode app to batch UDP payloads into large contiguous buffers that only need to traverse the networking stack once per batch, opposed to once per datagram. This greatly increased bulk throughput of UDP datagrams for user mode.</P> <P>&nbsp;</P> <P>These design requirements have led to some significant complexity internal to MsQuic as well. The QUIC protocol and the UDP (and below) work are separated onto their own threads. In scenarios with a small number of connections, these threads generally spread to separate processors, allowing for higher throughput. In scenarios with a large number of connections, effectively saturating all the processors with work, we do additional work improves parallelization.</P> <P>&nbsp;</P> <P>Those are just a few of the (bigger) impacts our performance-driven design has had on MsQuic architecture. This design process has affected every part of MsQuic from the API down to the platform abstraction layer.</P> <P>&nbsp;</P> <H1>Making Performance Testing Integral to CI</H1> <P>&nbsp;</P> <P>Claiming a performant design means nothing without data to back it up. Additionally, we found that occasional, mostly manual, performance testing led to even more issues. First off, to be able to make reasonable comparisons of performance results, we needed to reduce the number of factors that might affect the results. We found that having a manual process added a lot of variability to the results because of the significant setup and tool complexity. Removing the “middleman” was super important, but frequent testing has been even more important. If we only tested once a month, it was next to impossible to identify the cause of any regressions found in the latest results; let alone prevent them from happening in the first place. That inevitably led to a significant amount of wasted time trying to track down the problem. All the while, we had regressed performance for anyone using the code in the meantime.</P> <P>&nbsp;</P> <P>For these reasons, we’ve invested significant resources into making performance testing a first-class citizen in our <A href="#" target="_blank" rel="noopener">CI automation</A>. We run the full performance suite of tests for every single PR, every commit to main, and for every release branch. If a pull request affects performance, we know before it’s even merged into main. If it regresses performance, it’s not merged. With this system in place, we have pretty much guaranteed performance in main will only go up. This has also allowed us to confidently take external contributions to the code without fear of any regressions.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="nibanks_0-1618423531227.png" style="width: 1006px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/272887i15FB87C2CAC1FB0C/image-dimensions/1006x507?v=v2" width="1006" height="507" role="button" title="nibanks_0-1618423531227.png" alt="nibanks_0-1618423531227.png" /></span></P> <P>&nbsp;</P> <P>Another significant part of this automation is generating our <A href="#" target="_blank" rel="noopener">Performance Dashboard</A>. Every run of our CI pipeline for commits to main generates a new data point and automatically updates the data on the dashboard. The main page is designed to give a quick look at the current state of the system and any recent changes. There are various other pages that can be used to drill down into the data.</P> <P>&nbsp;</P> <H1>Progress So Far</H1> <P>&nbsp;</P> <P>As indicated in the chart at the beginning, we’ve had lots of improvements in performance over the last year. One nice feature of the dashboard is the ability to click on a data point and get linked directly to the relevant Git commit used. This allows us to easily find what code change caused the impacted performance. Below is a list of just a few of the recent commits that had the biggest impact on single connection upload performance.</P> <P>&nbsp;</P> <UL> <LI><A href="#" target="_blank" rel="noopener">d985d44</A> – Improves the flow control tuning logic</LI> <LI><A href="#" target="_blank" rel="noopener">1f4bfd7</A> – Refactors the perf tool</LI> <LI><A href="#" target="_blank" rel="noopener">ec6a3c0</A> – Fix a kernel issue related to starving NIC packet buffers</LI> <LI><A href="#" target="_blank" rel="noopener">be57c4a</A> – Refactors how we use RSS processor to schedule work</LI> <LI><A href="#" target="_blank" rel="noopener">084d034</A> – Refactors OpenSSL crypto abstraction layer</LI> <LI><A href="#" target="_blank" rel="noopener">9f10e02</A> – Switches to OpenSSL 1.1.1 branch instead of 3.0</LI> <LI><A href="#" target="_blank" rel="noopener">ee9fc96</A> – Adds GSO support to Linux data path abstraction</LI> <LI><A href="#" target="_blank" rel="noopener">a5e67c3</A> – Refactors UDP send logic to run on data path thread</LI> </UL> <P>&nbsp;</P> <P>Most of these changes came about from this simple process:</P> <P>&nbsp;</P> <OL> <LI>Collect performance traces.</LI> <LI>Analyze traces for bottlenecks.</LI> <LI>Improve biggest bottleneck.</LI> <LI>Test for regressions.</LI> <LI>Repeat.</LI> </OL> <P>&nbsp;</P> <P>This is a constantly ongoing process to always improve performance. We’ve done considerable work to make parts of this process easier. For instance, we’ve created our own <A href="#" target="_blank" rel="noopener">WPA plugin</A> for analyzing MsQuic performance traces. We also continue to spend time stabilizing our existing performance so that we can better catch possible regressions going forward.</P> <P>&nbsp;</P> <H1>Future Work</H1> <P>&nbsp;</P> <P>We’ve done a lot of work so far and come a long way, but the push for improved performance is never ending. There’s always another bottleneck to improve/eliminate. There’s always a little better/faster way of doing things. There’s always more tooling that can be created to improve the overall process. We will continue to put effort into all these.</P> <P>&nbsp;</P> <P>Going forward, we want to investigate additional hardware offloads and software optimization techniques. We want to build upon the work going on in the ecosystem and help to standardize these optimizations and integrate it them into the OS platform and then into MsQuic. Our hope is that we will make MsQuic the first choice for customer workloads by bringing the network performance benefits QUIC promises without having to make a trade-off with computational efficiency.</P> <P>&nbsp;</P> <P>As always, for more info&nbsp;<SPAN>on MsQuic continue reading&nbsp;</SPAN><A href="#" target="_blank" rel="noopener noreferrer">on GitHub</A><SPAN>.</SPAN></P> <P>&nbsp;</P> <P><SPAN>-- The MsQuic Team (Anthony Rossi, Nick Banks, Praveen Balasubramanian, &amp; Thad House)</SPAN></P> Thu, 15 Apr 2021 18:54:43 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/making-msquic-blazing-fast/ba-p/2268963 nibanks 2021-04-15T18:54:43Z Troubleshooting Switch Misconfiguration https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshooting-switch-misconfiguration/ba-p/2223614 <P>It’s 5 PM on Friday evening – the weekend will soon be here. You do one last sweep of your inbox before signing off when <SPAN style="text-decoration: line-through;">your cellphone</SPAN> the bat phone rings. Someone didn’t get the memo about the unwritten operational rule of IT Administration: Never make changes on a Friday. The phone itself seems terrified with every ring. A panicked voice on the other end says, “I can’t ping my VM.” Pandemonium ensues…</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 426px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265613i3DEF06637DACABF9/image-dimensions/426x346?v=v2" width="426" height="346" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>Today we’re going to talk about a new, free, downloadable tool that can help.</P> <P>&nbsp;</P> <P>Networks are complex. There are many different vendors, with many different configurations – Even your network team might be different than your Server/HCI team. In the revelry mentioned above, everything may look the same on your hosts, but it’s hard to know if the issue is caused by the host or the physical network without being able to see the physical network configuration.</P> <P>&nbsp;</P> <P>If LLDP is enabled on your switchports, it can be an easy task to quickly validate some of the physical network settings. LLDP or Link Layer Discovery Protocol is an IEEE standard (802.1AB) that allows networked devices to advertise their configuration (among other things) to neighboring devices. To Windows and Azure Stack HCI, the neighboring device is the physical switchport that its connected (via the NIC). LLDP’s <A href="#" target="_blank" rel="noopener">Wikipedia site</A> has a nice intro where you can learn a bit more.</P> <P>&nbsp;</P> <P>With LLDP, switchports can advertise the VLAN, MTU, and DCB configuration among others information which can be critical information for Azure Stack HCI systems. However, not all switches support advertisement of the same information. Without getting into the details (which you can read more about on the Wikipedia site linked above), the switch will determine how much information you can view.</P> <P>&nbsp;</P> <H1>Azure Stack HCI Network Switches</H1> <P>To improve Azure Stack HCI reliability where we have a purpose-built OS, we have begun to require that switches support LLDP. Most importantly, we require that they support some of the “organizationally specific <A href="#" target="_blank" rel="noopener">Custom TLVs</A>.” That is a fair amount of jargon, but it boils down to supporting capabilities like VLAN, MTU, etc. In the picture below, you can see the Organizationally Specific TLVs (type 127) along with the MTU and PFC configuration of the switchport this NIC is attached.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 496px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265614iC290474A144D0305/image-dimensions/496x458?v=v2" width="496" height="458" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P class="lia-indent-padding-left-30px"><EM><STRONG>Note:</STRONG> We intend to grow the list of required TLVs over time as we work with network vendors. Check the Custom TLVs documentation link just above for updates.</EM></P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <P class="lia-indent-padding-left-30px"><EM><STRONG>Help! I need to buy a switch for Azure Stack HCI!</STRONG></EM></P> <P class="lia-indent-padding-left-30px"><EM>We document some <A href="#" target="_blank" rel="noopener">Network Switches for Azure Stack HCI</A> that the vendor has verified meet the requirements – the list will grow as we hear from the various switch vendors. Talk to your Network Vendor to see if your switch meets the requirements.</EM></P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <P>Having this information at your disposal can help you answer several critical questions particularly when you want to get started on your weekend:</P> <UL> <LI>Did you misconfigure your host or is it the physical network?</LI> <LI>Did the network engineer add the necessary configuration to the correct switchport?</LI> <LI>Is the switchport configuration the same on each team member?</LI> <LI>Is the switchport configuration the same between each cluster node?</LI> </UL> <P class="lia-indent-padding-left-30px"><EM><STRONG>Help! My Network Admin says LLDP is insecure!</STRONG></EM></P> <P class="lia-indent-padding-left-30px"><EM>LLDP does not require credentials to receive information but that doesn’t mean it’s insecure. LLDP allows the administrator of the network device to choose which information (TLVs), if any, is sent to neighbors with the intention that this information can be used for diagnostic purposes.</EM></P> <P>&nbsp;</P> <H1>Get Started</H1> <P>Back to our IT hero for a moment. How can you quickly determine whether the issue is on the switch or you missed some settings on your host?</P> <P>&nbsp;</P> <P>An LLDP enabled switchport will periodically (typically every 30 seconds) send messages to its neighbors, including the juicy information you may want as an IT Administrator to determine whether your physical host configuration matches that of the physical network.</P> <P>Retrieving this information is traditionally a bit of a challenge, however there is a tool to make this simple.</P> <P>&nbsp;</P> <P class="lia-indent-padding-left-30px"><EM><STRONG>Note</STRONG>: If you’re not in control of your network switches, make sure you ask your network team to enable LLDP and any “organizationally specific TLVs” that the switch supports.</EM></P> <P>&nbsp;</P> <H2>Install the Module</H2> <P>First install the DataCenterBridging module from the PowerShell gallery. This module contains a few goodies and has been updated to include the functions to parse the LLDP data from the switch.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 759px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265615i58487970BB458FE6/image-dimensions/759x118?v=v2" width="759" height="118" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>There are four available commands at the time of writing:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 455px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265616i9EF2ECEC90FFBCDF/image-dimensions/455x173?v=v2" width="455" height="173" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <H2>Getting the Physical Switch Information</H2> <P>Let’s start off by trying to get the LLDP information using Get-FabricInfo. With each of the commands you can specify the SET Switch or individual Interface names (using the InterfaceName parameter). In this case, we are specifying the SET Switch that starts with the name <EM>Converged.</EM> The cmdlet finds all the physical NICs attached the switch and looks for available LLDP messages on each interface.</P> <P>&nbsp;</P> <P>At first run, it probably will not find anything. The cmdlet tells you to run Test-FabricInfo to help identify the problem.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 802px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265618i0D1162E2F108C553/image-dimensions/802x162?v=v2" width="802" height="162" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>Running Test-FabricInfo identifies a few problems that we need to resolve.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 660px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265619i67030A0F3C4CEC6F/image-dimensions/660x218?v=v2" width="660" height="218" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>You can use Enable-FabricInfo to resolve all the problems in one shot. This will install the feature and ensure that the LLDP agent is enabled on the underlying interfaces, etc.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 631px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265620iC0651A37325F5B4B/image-dimensions/631x88?v=v2" width="631" height="88" role="button" title="image.png" alt="image.png" /></span></P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <P class="lia-indent-padding-left-30px"><EM><STRONG>Note:</STRONG> Want to know everything this is doing? Look at <A href="#" target="_blank" rel="noopener">the code on GitHub</A>!</EM></P> <P>&nbsp;</P> <P>Next, run Test-FabricInfo again to determine if all the requirements are met. You can see we got a little better. Only two remaining issues; we didn’t find any LLDP packets for the interfaces in the SET switch.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 672px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265622iCF7F127E8F0C3806/image-dimensions/672x222?v=v2" width="672" height="222" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>Wait about 30 seconds – the typical amount of time that a switchport will send LLDP packets – and try again. If you still fail after the messages above, <STRONG>contact your network administrator and ensure that LLDP is enabled on the switchports connected to your team members.</STRONG></P> <P>&nbsp;</P> <P>If LLDP is enabled on your physical switch, you will see the following below which indicates that Test-FabricInfo found an LLDP message from the physical switch for each member of the Converged team.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 679px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265623iDBEE0B490B1C1FA7/image-dimensions/679x221?v=v2" width="679" height="221" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>Now we are ready to run Get-FabricInfo. Make sure you put the output into a variable, so you can inspect it. In this case, we add everything to the <STRONG>$FabricInfo</STRONG> variable which has an object for every team member.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 538px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265624iFB754CF64B7AC9DD/image-size/large?v=v2&amp;px=999" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>You can walk the individual team members to see information collected on the Windows or HCI host (under InterfaceDetails) or the physical switch (Fabric) to which the NICs are connected.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 488px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265625i558C9D816CC33147/image-dimensions/488x244?v=v2" width="488" height="244" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>For example, here’s a look at the IP and Subnet information on pNIC01. We collect this so it’s easy to compare to the information collected from the switch. As you can see, we have the IP Address, Subnet, VLAN, etc.</P> <P>&nbsp;</P> <P>In this case, we have a virtual switch on the host and as part of the storage configuration on this system, we have a team mapped host vNIC. The IP, Subnet, etc. are being displayed from that team-mapped host vNIC. If the team member isn’t part of a virtual switch, we’ll display the configuration on the physical NIC.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 458px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265626i6A61EF57BE367A6F/image-dimensions/458x249?v=v2" width="458" height="249" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>Now let’s take a look at what the switch sent us and what we can learn about the physical network (as mentioned before, the information will vary based on what the switch supports):</P> <UL> <LI><STRONG>NativeVLAN: 1133</STRONG> – Untagged traffic will be sent over this vlan</LI> <LI><STRONG>VLANID: Info Not Provided…</STRONG> This includes the trunked VLANs that can be carried on this switchport. The switch below did not include this information in the packet sent to the host.</LI> <LI><STRONG>FrameSize: 9236</STRONG> – The physical NIC and virtual NICs MTU configuration should not exceed the switches value or traffic will be segmented (or in some cases dropped).</LI> <LI><STRONG>PFC is enabled on Priority3</STRONG> – Data needing lossless communication (e.g. RoCE-based RDMA) should use Priority 3.</LI> </UL> <P class="lia-indent-padding-left-30px"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 426px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265627i4DE90599B5BBD046/image-dimensions/426x265?v=v2" width="426" height="265" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>From this information, we can determine that VLAN 711 (on the storage vNIC) is not using the native vlan, and the switch is not showing the trunked VLANs in LLDP either. This leads to two conclusions:</P> <UL> <LI>We should check the switch configuration or contact our network administrator if network connectivity is not available on pNIC01 because we could not confirm that traffic is available here.</LI> <LI>We should ask our network administrator to find us a switch that does advertise this information so that we can identify this problem ourselves (and without ruining <U>their</U> weekend).</LI> </UL> <P>Here’s the same view but from another switch. This switch did not send the PFC information, but it does show the VLAN IDs available to the host (1, 11, 12, and 40).</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 368px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/265628i92C29D37D0B24327/image-size/large?v=v2&amp;px=999" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>From here, we can tell that VLAN 711 is not available on the physical network which is at least one obvious reason why there may not be network connectivity on this link.</P> <P>Some of the other problems on the physical network that you can easily identify:</P> <UL> <LI>Missing VLANs</LI> <LI>Misconfigured Jumbo Frames</LI> <LI>Misconfigured PFC settings</LI> <LI>Topology problems e.g. cabled to the wrong switch (check ChassisGroups for this information)</LI> </UL> <P>Reminder: The information displayed is dependent on the switch’s capabilities. If the switch is unable to provide us with a certain TLV, we display the text “Information Not Provided By Switch.” If you see this message, you should work with your network administrator to identify if the information can be included.</P> <P>&nbsp;</P> <H1>Summary</H1> <P>Get-FabricInfo allows you to answer several questions about the physical network configuration that may come in handy when troubleshooting diagnostic issues. Is the physical network setup as I expected it? Is the configuration the same between cluster nodes? All of this and more can be answered if your switch supports LLDP and you’re running Windows or Azure Stack HCI.</P> <P>Hopefully that Friday afternoon call isn’t quite so scary anymore!</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>Dan “weekend warrior” Cuomo</P> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-end-end-radius: initial; border-end-start-radius: initial; border-inline: initial; border-start-end-radius: initial; border-start-start-radius: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> Fri, 19 Mar 2021 20:46:43 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshooting-switch-misconfiguration/ba-p/2223614 Dan Cuomo 2021-03-19T20:46:43Z Azure Kubernetes Service on Azure Stack HCI Parity with AKS PowerShell https://gorovian.000webhostapp.com/?exam=t5/networking-blog/azure-kubernetes-service-on-azure-stack-hci-parity-with-aks/ba-p/2179328 <P><FONT size="5" color="#00CCFF"><STRONG>AksHci&nbsp;PowerShell&nbsp;February&nbsp;Release&nbsp;</STRONG></FONT></P> <P>If you were one of&nbsp;many people who gave us feedback on our December release,&nbsp;we have exciting news for you!&nbsp;In our February release,&nbsp;we’ve delivered one of the most requested features, additional networking options! While you can use PowerShell or Windows Admin Center to provision and manage your Azure Kubernetes Service clusters on Azure Stack HCI (AKS-HCI), this post will focus on&nbsp; the&nbsp;AksHci&nbsp;PowerShell module and how it is moving closer to being&nbsp;aligned with the AKS PowerShell module.&nbsp;</P> <P>&nbsp;</P> <P><FONT size="5" color="#00CCFF"><STRONG>What’s new?&nbsp;</STRONG></FONT></P> <P>In this version of the&nbsp;AksHci&nbsp;PowerShell, new capabilities are now available such as creating a private virtual network,&nbsp;static IP deployment,&nbsp;and Active Directory integration.&nbsp;</P> <P>&nbsp;</P> <P>Now, with the new command `New-AksHciNetworkSetting`,&nbsp;users are given the option to deploy with DHCP or&nbsp;static IP. We recommend deploying with Static IP&nbsp;because the IP addresses remain the same over time unless it is changed manually.&nbsp;This command will create&nbsp;a configuration object for&nbsp;a virtual network for the control plane, load balancer, agent endpoints, and a static IP range for nodes in all clusters.&nbsp;To&nbsp;deploy a&nbsp;cluster with&nbsp;a virtual network based on the configuration object created,&nbsp;you will pass the assigned name of the object to the new parameter `-vnet` in the command `Set-AksHciConfig`.&nbsp;</P> <P>&nbsp;</P> <P><STRONG>New-AksHciNetworkSetting&nbsp;example for&nbsp;static IP&nbsp;deployment&nbsp;</STRONG></P> <TABLE style="width: 800px; border-style: solid; border-color: black;" width="800"> <TBODY> <TR> <TD width="623.333px"> <P>$vnet&nbsp;= New-AksHciNetworkSetting&nbsp;-vnetName&nbsp;"External" -k8sNodeIpPoolStart "172.16.10.0"</P> <P>-k8sNodeIpPoolEnd "172.16.10.255" -vipPoolStart&nbsp;"172.16.255.0" -vipPoolEnd&nbsp;"172.16.255.254"</P> <P>-ipAddressPrefix&nbsp;"172.16.0.0/16" -gateway "172.16.0.1" -dnsServers&nbsp;"172.16.0.1"&nbsp;&nbsp;</P> <P>&nbsp;</P> </TD> </TR> <TR> <TD width="623.333px"> <P>Set-AksHciConfig -imageDir&nbsp;c:\clusterstorage\volume1\Images</P> <P>-cloudConfigLocation&nbsp;c:\clusterstorage\volume1\Config -vnet&nbsp;$vnet&nbsp;-enableDiagnosticData&nbsp;</P> <P>-cloudservicecidr&nbsp;"172.16.10.10/16"&nbsp;</P> <P>&nbsp;</P> </TD> </TR> <TR> <TD width="623.333px"> <P>Install-AksHci&nbsp;</P> </TD> </TR> </TBODY> </TABLE> <P><EM>*Note: The values for the parameters need to be configured to your environment.&nbsp;</EM></P> <P>&nbsp;</P> <P><STRONG>New-AksHciNetworkSetting&nbsp;example&nbsp;for a DHCP&nbsp;deployment&nbsp;</STRONG></P> <TABLE style="width: 800px; border-style: solid; border-color: black;" width="800"> <TBODY> <TR> <TD width="623.333px"> <P>$vnet&nbsp;= New-AksHciNetworkSetting&nbsp;-vnetName&nbsp;"External" -vipPoolStart&nbsp;"172.16.255.0"</P> <P>-vipPoolEnd&nbsp;"172.16.255.254"&nbsp;</P> <P>&nbsp;</P> </TD> </TR> <TR> <TD width="623.333px"> <P>Set-AksHciConfig -imageDir&nbsp;c:\clusterstorage\volume1\Images</P> <P>-cloudConfigLocation&nbsp;c:\clusterstorage\volume1\Config -vnet&nbsp;$vnet&nbsp;-enableDiagnosticData"&nbsp;</P> <P>&nbsp;</P> </TD> </TR> <TR> <TD width="623.333px"> <P>Install-AksHci&nbsp;</P> </TD> </TR> </TBODY> </TABLE> <P><EM>*Note: The values for the parameters need to be configured to your environment.&nbsp;</EM></P> <P>&nbsp;</P> <P>For more information about the&nbsp;`New-AksHciNetworkSetting`&nbsp;command and its parameters, go&nbsp;<A title="new-akshcinetworkingsetting" href="#" target="_self">here</A>.</P> <P>&nbsp;</P> <P>For more information on virtual&nbsp;network, Static IP, and DHCP, go&nbsp;<A title="networking concepts" href="#" target="_self">here</A>.</P> <P>&nbsp;</P> <P>In addition to the new virtual network and Static IP features, you can now integrate Active Directory (AD) with Azure Kubernetes Service on Azure Stack HCI. Without Active Directory, connection to the API server relied on a certificate-based&nbsp;kubeconfig&nbsp;file.&nbsp;Having secrets such as these certs in the&nbsp;kubeconfig&nbsp;file creates a greater opportunity for those secrets to be leaked.&nbsp;Now, users can enable AD authentication to use AD single sign-on (SSO) to securely connect to the API server.&nbsp;This new feature introduces a new parameter,&nbsp;-enableAdAuth`, to the command `New-AksHciCluster` and a new command, `Install-AksHciAdAuth`&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Install-AksHciAdAuth example&nbsp;</STRONG></P> <TABLE style="width: 800px; border-style: solid; border-color: black;" data-tablestyle="MsoTableGrid" data-tablelook="1696"> <TBODY> <TR> <TD width="723.333px" height="56px" data-celllook="0"> <P><SPAN data-contrast="auto">New-AksHciCluster -name mynewcluster1 –enableADAuth</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> </TR> <TR> <TD width="723.333px" height="111px" data-celllook="0"> <P><SPAN data-contrast="auto">Install-AksHciAdAuth&nbsp;-name mynewcluster1 -keytab&nbsp;&lt;.\current.keytab&gt;</SPAN></P> <P><SPAN data-contrast="auto"> -previousKeytab&nbsp;&lt;.\previous.keytab&gt; -SPN &lt;service/principal@CONTOSO.COM&gt; </SPAN></P> <P><SPAN data-contrast="auto">-adminUser&nbsp;CONTOSO\Bob</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;<EM>*Note: The values for the parameters need to be configured to your environment.&nbsp;</EM></SPAN></P> <P>&nbsp;</P> <P>For more information on the `Install-AksHciAdAuth` command and its parameters, go&nbsp;<A title="install-akshciadauth" href="#" target="_self">here</A>.&nbsp;</P> <P>&nbsp;</P> <P>For&nbsp;a tutorial on&nbsp;Active Directory integration with AKS-HCI, go&nbsp;<A title="ad auth tutorial" href="#" target="_self">here</A>.</P> <P>&nbsp;</P> <P><FONT size="5" color="#00CCFF"><STRONG>Parity with Azure Kubernetes Service (AKS)&nbsp;PowerShell&nbsp;</STRONG></FONT></P> <P>Not only do these new features provide a wider range of capabilities and configuration for our customers,&nbsp;but&nbsp;they also bring the AKS-HCI platform to be&nbsp;more closely&nbsp;aligned with&nbsp;AKS&nbsp;capabilities.&nbsp;The AKS-HCI team is working on bringing&nbsp;parity between the two platforms. One of the goals for this is to make sure that the user experience for AKS and AKS-HCI is as closely aligned as possible. There are two ways to provision and manage your AKS-HCI clusters: Windows Admin Center and the AKS-HCI PowerShell module, which are&nbsp;designed to have the same user experience as the Azure Portal&nbsp;and&nbsp;AKS PowerShell&nbsp;respectively.&nbsp;</P> <P>&nbsp;</P> <P>Below, these are some examples of AKS-HCI and AKS commands.&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Provision&nbsp;a&nbsp;Kubernetes&nbsp;cluster&nbsp;</STRONG></P> <TABLE style="height: 85px; width: 800px; border-style: solid; border-color: black;" width="800"> <TBODY> <TR> <TD width="348.667px" height="29px"> <P>AKS-HCI&nbsp;</P> </TD> <TD width="350.667px" height="29px"> <P>AKS&nbsp;</P> </TD> </TR> <TR> <TD width="348.667px" height="56px"> <P>New-AksHciCluster&nbsp;-name&nbsp;mycluster&nbsp;</P> </TD> <TD width="350.667px" height="56px"> <P>New-AzAksCluster&nbsp;-name&nbsp;mycluster</P> <P>-resourceGroupName&nbsp;myresourcegroup&nbsp;</P> </TD> </TR> </TBODY> </TABLE> <P><EM>*Note: The parameter ‘-resourceGroupName’ is different because AKS-HCI is run on-premises and its resource are not grouped.&nbsp;</EM></P> <P>&nbsp;</P> <P><STRONG>List deployed Kubernetes&nbsp;clusters&nbsp;</STRONG></P> <TABLE style="width: 800px; border-color: black; border-style: solid;" width="800"> <TBODY> <TR> <TD width="311"> <P>AKS-HCI&nbsp;</P> </TD> <TD width="311"> <P>AKS&nbsp;</P> </TD> </TR> <TR> <TD width="311"> <P>Get-AksHciCluster&nbsp;</P> </TD> <TD width="311"> <P>Get-AzAksCluster&nbsp;</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P><STRONG>Delete&nbsp;a Kubernetes cluster&nbsp;</STRONG></P> <TABLE style="width: 800px; border-color: black; border-style: solid;" width="800"> <TBODY> <TR> <TD width="311"> <P>AKS-HCI&nbsp;</P> </TD> <TD width="311"> <P>AKS&nbsp;</P> </TD> </TR> <TR> <TD width="311"> <P>Remove-AksHciCluster&nbsp;-name&nbsp;mycluster&nbsp;</P> </TD> <TD width="311"> <P>Remove-AzAksCluster&nbsp;-name&nbsp;mycluster&nbsp;</P> <P>-resourceGroupName&nbsp;myresourcegroup&nbsp;</P> </TD> </TR> </TBODY> </TABLE> <P><EM>*Note: The parameter ‘-resourceGroupName’ is different because AKS-HCI is run on-premises and its resource are not grouped.&nbsp;</EM></P> <P>&nbsp;</P> <P>There are&nbsp;still&nbsp;some disparities between the two modules, but we are working on closing these gaps in future releases to provide customers with a seamless experience&nbsp;for a hybrid environment in AKS and AKS-HCI.&nbsp;</P> <P>&nbsp;</P> <P>We would love feedback on AKS consistency in PowerShell, hearing about any other AKS features you would like to see in AKS-HCI, or if you are interested in&nbsp;an on-premises Kubernetes solution! Please fill out this quick survey <A title="feedback survey" href="#" target="_self">here</A>.</P> <P>&nbsp;</P> <P>Learn how to set up your Azure Kubernetes Service host on Azure Stack HCI <A href="#" target="_self">here</A>.</P> Wed, 03 Mar 2021 17:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/azure-kubernetes-service-on-azure-stack-hci-parity-with-aks/ba-p/2179328 jessicaguan 2021-03-03T17:00:00Z Quick Tip: Does my NIC support VMMQ? https://gorovian.000webhostapp.com/?exam=t5/networking-blog/quick-tip-does-my-nic-support-vmmq/ba-p/2156613 <P>Hi Folks - Most often, when a virtual machine or container is receiving network traffic, the traffic passes through the virtualization stack in the host. This requires host (parent partition) CPU cycles.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 493px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/256678iC482A2298F6A6F3F/image-dimensions/493x220?v=v2" width="493" height="220" role="button" title="image.png" alt="Synthetic Data Path" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Synthetic Data Path</span></span></P> <P>&nbsp;</P> <P>If the amount of traffic being processed exceeds that which a single core can handle, the received network traffic must be spread across multiple CPUs. This “spreading” can occur in the operating system – at the expense of more CPU cycles, or hardware (the NIC) as an offload. In hardware, we call this capability <STRONG>V</STRONG>irtual <STRONG>M</STRONG>achine <STRONG>M</STRONG>ulti-<STRONG>Q</STRONG>ueue. The benefit of VMMQ is actually two-fold:</P> <UL> <LI>It allows you to reach higher throughput into your virtual systems (VMs/Containers)</LI> <LI>It reduces the cost (in terms of host resources) of processing that network traffic</LI> </UL> <P>VMMQ is a combined feature of the NIC, driver/firmware, and operating system. All of these must support VMMQ and be configured properly for you to leverage this offload.</P> <P>&nbsp;</P> <P>To identify if your adapter supports VMMQ, use the <STRONG>Get-NetAdapterAdvancedProperty</STRONG> cmdlet&nbsp;to see the advanced registry property *RSSOnHostVPorts or “Virtual Switch RSS” – We won’t go into what the naming means but suffice to say that if you see this capability displayed using the command below, your NIC and driver/firmware combination supports VMMQ.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image.png" style="width: 668px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/256692i6E860051C2143E6F/image-dimensions/668x152?v=v2" width="668" height="152" role="button" title="image.png" alt="image.png" /></span></P> <P>&nbsp;</P> <P>Now you simply need to follow the instructions in&nbsp;<A href="#" target="_blank" rel="noopener">this article</A>&nbsp;for how to configure it.</P> <P>&nbsp;</P> <P>Hope this quick tip was helpful!</P> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-inline: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-inline: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-inline: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-inline: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-inline: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> <DIV class="ms-editor-squiggler" style="color: initial; font: initial; font-feature-settings: initial; font-kerning: initial; font-optical-sizing: initial; font-variation-settings: initial; forced-color-adjust: initial; text-orientation: initial; text-rendering: initial; -webkit-font-smoothing: initial; -webkit-locale: initial; -webkit-text-orientation: initial; -webkit-writing-mode: initial; writing-mode: initial; zoom: initial; place-content: initial; place-items: initial; place-self: initial; alignment-baseline: initial; animation: initial; appearance: initial; aspect-ratio: initial; backdrop-filter: initial; backface-visibility: initial; background: initial; background-blend-mode: initial; baseline-shift: initial; block-size: initial; border-block: initial; border: initial; border-radius: initial; border-collapse: initial; border-inline: initial; inset: initial; box-shadow: initial; box-sizing: initial; break-after: initial; break-before: initial; break-inside: initial; buffered-rendering: initial; caption-side: initial; caret-color: initial; clear: initial; clip: initial; clip-path: initial; clip-rule: initial; color-interpolation: initial; color-interpolation-filters: initial; color-rendering: initial; color-scheme: initial; columns: initial; column-fill: initial; gap: initial; column-rule: initial; column-span: initial; contain: initial; contain-intrinsic-size: initial; content: initial; content-visibility: initial; counter-increment: initial; counter-reset: initial; counter-set: initial; cursor: initial; cx: initial; cy: initial; d: initial; display: block; dominant-baseline: initial; empty-cells: initial; fill: initial; fill-opacity: initial; fill-rule: initial; filter: initial; flex: initial; flex-flow: initial; float: initial; flood-color: initial; flood-opacity: initial; grid: initial; grid-area: initial; height: 0px; hyphens: initial; image-orientation: initial; image-rendering: initial; inline-size: initial; inset-block: initial; inset-inline: initial; isolation: initial; letter-spacing: initial; lighting-color: initial; line-break: initial; list-style: initial; margin-block: initial; margin: initial; margin-inline: initial; marker: initial; mask: initial; mask-type: initial; max-block-size: initial; max-height: initial; max-inline-size: initial; max-width: initial; min-block-size: initial; min-height: initial; min-inline-size: initial; min-width: initial; mix-blend-mode: initial; object-fit: initial; object-position: initial; offset: initial; opacity: initial; order: initial; origin-trial-test-property: initial; orphans: initial; outline: initial; outline-offset: initial; overflow-anchor: initial; overflow-wrap: initial; overflow: initial; overscroll-behavior-block: initial; overscroll-behavior-inline: initial; overscroll-behavior: initial; padding-block: initial; padding: initial; padding-inline: initial; page: initial; page-orientation: initial; paint-order: initial; perspective: initial; perspective-origin: initial; pointer-events: initial; position: initial; quotes: initial; r: initial; resize: initial; ruby-position: initial; rx: initial; ry: initial; scroll-behavior: initial; scroll-margin-block: initial; scroll-margin: initial; scroll-margin-inline: initial; scroll-padding-block: initial; scroll-padding: initial; scroll-padding-inline: initial; scroll-snap-align: initial; scroll-snap-stop: initial; scroll-snap-type: initial; shape-image-threshold: initial; shape-margin: initial; shape-outside: initial; shape-rendering: initial; size: initial; speak: initial; stop-color: initial; stop-opacity: initial; stroke: initial; stroke-dasharray: initial; stroke-dashoffset: initial; stroke-linecap: initial; stroke-linejoin: initial; stroke-miterlimit: initial; stroke-opacity: initial; stroke-width: initial; tab-size: initial; table-layout: initial; text-align: initial; text-align-last: initial; text-anchor: initial; text-combine-upright: initial; text-decoration: initial; text-decoration-skip-ink: initial; text-indent: initial; text-overflow: initial; text-shadow: initial; text-size-adjust: initial; text-transform: initial; text-underline-offset: initial; text-underline-position: initial; touch-action: initial; transform: initial; transform-box: initial; transform-origin: initial; transform-style: initial; transition: initial; user-select: initial; vector-effect: initial; vertical-align: initial; visibility: initial; -webkit-app-region: initial; border-spacing: initial; -webkit-border-image: initial; -webkit-box-align: initial; -webkit-box-decoration-break: initial; -webkit-box-direction: initial; -webkit-box-flex: initial; -webkit-box-ordinal-group: initial; -webkit-box-orient: initial; -webkit-box-pack: initial; -webkit-box-reflect: initial; -webkit-highlight: initial; -webkit-hyphenate-character: initial; -webkit-line-break: initial; -webkit-line-clamp: initial; -webkit-mask-box-image: initial; -webkit-mask: initial; -webkit-mask-composite: initial; -webkit-perspective-origin-x: initial; -webkit-perspective-origin-y: initial; -webkit-print-color-adjust: initial; -webkit-rtl-ordering: initial; -webkit-ruby-position: initial; -webkit-tap-highlight-color: initial; -webkit-text-combine: initial; -webkit-text-decorations-in-effect: initial; -webkit-text-emphasis: initial; -webkit-text-emphasis-position: initial; -webkit-text-fill-color: initial; -webkit-text-security: initial; -webkit-text-stroke: initial; -webkit-transform-origin-x: initial; -webkit-transform-origin-y: initial; -webkit-transform-origin-z: initial; -webkit-user-drag: initial; -webkit-user-modify: initial; white-space: initial; widows: initial; width: initial; will-change: initial; word-break: initial; word-spacing: initial; x: initial; y: initial; z-index: initial;">&nbsp;</DIV> Mon, 22 Feb 2021 19:49:06 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/quick-tip-does-my-nic-support-vmmq/ba-p/2156613 Dan Cuomo 2021-02-22T19:49:06Z Calico for Windows goes Open Source https://gorovian.000webhostapp.com/?exam=t5/networking-blog/calico-for-windows-goes-open-source/ba-p/1620297 <P>&nbsp;</P> <P>Since Windows worker node support reached GA in Kubernetes Microsoft and Tigera have listened closely to feedback from the community. <BR />A big contention point of Windows Container users in the Kubernetes community is:<EM> “One of the most important open source network policy tools in the market is not available for Windows.”</EM> &nbsp;<BR />This is limiting adoption of Windows worker nodes for Kubernetes in environments big and small as customers cannot fulfill their policy and compliance requirements like they could address them in Linux.</P> <P>Over the last couple of years Microsoft and Tigera have been working together to close some significant gaps in the Windows container networking stack. The work includes enabling Direct Server Return, enhanced policy, packet logging, expanded firewall support in the Host Networking Service of Windows, multi-subnet support and many more large and small improvements.</P> <P>&nbsp;</P> <P><STRONG>Now for the good news:</STRONG></P> <P>&nbsp;</P> <P>We are incredibly happy to share that with <STRONG>Calico 3.16</STRONG> - Windows container support is now GA in open source Calico (<A href="#" target="_blank" rel="noopener">Release Notes</A>)</P> <P>With this release, windows containers can be deployed and secured in Azure cloud, any other cloud computing provider or on-premises using networking components in Windows Server and Calico network policy.&nbsp;</P> <P>More details about Calico for Windows version 3.16 can be found <A href="#" target="_self">in this on demand video.</A>&nbsp;</P> <P>If you have any questions or concerns contact us at the Calico User Slack <A href="#" target="_blank" rel="noopener">Windows channel</A>.</P> <P>The official announcement blog post can be found <A title="Calico for Windows Release blog" href="#" target="_self">here</A>.</P> <P>Want to get started right away? Check out the 5 minute quick start guide <A href="#" target="_blank" rel="noopener">here</A>.&nbsp;&nbsp;&nbsp;&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> Tue, 22 Sep 2020 14:46:24 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/calico-for-windows-goes-open-source/ba-p/1620297 mkostersitz 2020-09-22T14:46:24Z Windows Server Insiders getting gRPC support in Http.sys https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-server-insiders-getting-grpc-support-in-http-sys/ba-p/1534273 <P><EM>Credit and thanks for feature work to Niranjan Inamdar</EM></P> <P>&nbsp;</P> <P>We keep hearing many of you are eager to host gRPC-enabled services on IIS in Azure, such as&nbsp;<A href="#" target="_blank" rel="noopener">here on GitHub</A><SPAN>.</SPAN> However, the lack of support for several HTTP features in the platform components is blocking you. We listened, and we have good news: we built HTTP support for gRPC into Http.sys! We are targeting the feature for Windows Server vNext. You can try it out today if you have access to Windows Server Insider builds.</P> <P>&nbsp;</P> <P>Let us back up a bit. gRPC is a “modern open-source high-performance RPC framework”. You can read more about it <A href="#" target="_blank" rel="noopener">here on the gRPC website</A>. It allows clients and backends to communicate with one another with relative ease while using different frameworks and languages. In theory, such communication is achievable with JSON serialization. However, gRPC significantly outperforms it in terms of message sizes and speed of serialization. For its full functionality, it has more requirements from the implementation of the HTTP/2 protocol, which is the transport for gRPC. Unlike the browser scenarios, it requires support for bidirectional streaming and response trailers. Http.sys has always supported bidirectional streaming, but the response trailers were a feature gap up to this point. You can read more about the use of gRPC <A href="#" target="_blank" rel="noopener">here in the context of .NET and usage patterns</A>.</P> <P>&nbsp;</P> <P>If you want to try out native gRPC support in Http.sys, you will need to start by getting an Insider build of Windows Server. You can do that by following the instructions on the <A href="#" target="_blank" rel="noopener">Windows Server Insider Preview page</A>.</P> <P>&nbsp;</P> <P>Once you have your Windows Server vNext installed, you can test out the new HTTP features, whether you are building on top of Http.sys or simply want to see it working under the hood. The following code sample will show you how to verify the necessary HTTP features are working properly.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="applescript">HTTP_DATA_CHUNK Chunks[2]; HTTP_UNKNOWN_HEADER Trailers[2]; PSTR ResponseBody = "Response Body"; HTTP_RESPONSE Response; HTTP_CACHE_POLICY CachePolicy = {0}; Trailers[0].pName = "Trailer1"; Trailers[0].NameLength = (USHORT)strlen(Trailer[0].pName); Trailers[0].pRawValue = "Value1"; Trailers[0].RawValueLength = (USHORT)strlen(Trailer[0].pRawValue); Trailers[1].pName = "Trailer2"; Trailers[1].NameLength = (USHORT)strlen(Trailer[0].pName); Trailers[1].pRawValue = "Value2"; Trailers[1].RawValueLength = (USHORT)strlen(Trailer[0].pRawValue); Chunks[0].DataChunkType = HttpDataChunkFromMemory; Chunks[0].FromMemory.pBuffer = (PVOID)ResponseBody; Chunks[0].FromMemory.BufferLength = (USHORT)strlen(ResponseBody); Chunks[1].DataChunkType = HttpDataChunkTrailers; Chunks[1].Trailers.TrailerCount = RTL_NUMBER_OF(Trailers); Chunks[1].Trailers.pTrailers = Trailers; RtlZeroMemory(&amp;Response, sizeof(Response)); Response.StatusCode = 200; Response.pReason = "OK"; Response.ReasonLength = (USHORT) strlen(Response.pReason); Response.pEntityChunks = Chunks; Response.EntityChunkCount = RTL_NUMBER_OF(Chunks); Error = HttpSendHttpResponse(RequestQueue, Request-&gt;RequestId, 0, &amp;Response, &amp;CachePolicy, &amp;BytesSent, NULL, NULL, NULL, NULL);</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>We are excited about this feature and eager to hear you feedback!</P> Wed, 05 Aug 2020 17:37:35 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-server-insiders-getting-grpc-support-in-http-sys/ba-p/1534273 tojens 2020-08-05T17:37:35Z Pointer: Domain Time Synchronization in the Age of Working from Home https://gorovian.000webhostapp.com/?exam=t5/networking-blog/pointer-domain-time-synchronization-in-the-age-of-working-from/ba-p/1443289 <P>If you're looking for tips on how to keep your remote workers connected and synchronized with time (which is critical for domain joined systems), look no further than this blog on Sarath Madakasira's blog on&nbsp;<A title="Domain Time Synchronization in the Age of Working from Home" href="https://gorovian.000webhostapp.com/?exam=t5/core-infrastructure-and-security/domain-time-synchronization-in-the-age-of-working-from-home/ba-p/1440820" target="_blank" rel="noopener">Domain Time Synchronization in the Age of Working from Home</A>&nbsp;</P> <P>&nbsp;</P> <P>Special thanks to the Core Infrastructure and Security blog for posting this for us!</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>Dan Cuomo</P> Fri, 05 Jun 2020 16:04:41 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/pointer-domain-time-synchronization-in-the-age-of-working-from/ba-p/1443289 Dan Cuomo 2020-06-05T16:04:41Z Introducing Packet Monitor https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-packet-monitor/ba-p/1410594 <P><SPAN>Network connectivity issues are often hard to diagnose.&nbsp;There are multiple machines involved in a single data transfer; at least two endpoints and a complex network infrastructure in the middle. Lately, with the introduction of network virtualization, more of the infrastructure capabilities like routing and switching are being integrated into the endpoints. The additional complexity in the endpoints often leads to connectivity issues that are hard to diagnose. This new infrastructure requires a more comprehensive network diagnostics approach.</SPAN></P> <P><SPAN>&nbsp;</SPAN></P> <H2><SPAN>Packet Monitor</SPAN></H2> <P>&nbsp;</P> <P><SPAN><A href="#" target="_blank" rel="noopener">Packet Monitor&nbsp;(PacketMon)</A>&nbsp;is&nbsp;an in-box&nbsp;cross-component&nbsp;network diagnostics&nbsp;tool for Windows.&nbsp;It&nbsp;can be used for&nbsp;packet capture, packet drop detection,&nbsp;packet filtering and counting.&nbsp;The tool is especially helpful in virtualization scenarios like container networking, SDN, etc. It is available&nbsp;in-box&nbsp;via&nbsp;pktmon.exe&nbsp;command, and&nbsp;via&nbsp;Windows Admin Center extensions.</SPAN></P> <P><SPAN>&nbsp;</SPAN></P> <H3>Overview</H3> <P>&nbsp;</P> <P>Any machine that communicates over the network has at least one network adapter. All the components between this adapter and an application form a networking stack. The networking stack is a set of networking components that process and move networking traffic. In traditional scenarios, the networking stack is small, and all the packet routing and switching happens in external devices.</P> <P class="lia-align-center">&nbsp;</P> <P class="lia-align-center"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="small networking stack5.png" style="width: 200px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/193860i0C41843F280F946E/image-size/small?v=v2&amp;px=200" role="button" title="small networking stack5.png" alt="Networking stack in traditional scenarios" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Networking stack in traditional scenarios</span></span></P> <P>However, with the advent of network virtualization, the size of the networking stack has multiplied. This extended networking stack now includes components, like the Virtual Switch, that handle packet processing and switching. Such flexible environment allows for much better resource utilization and security isolation, but it also leaves more room for configuration mistakes that are hard to diagnose. Accordingly, a visibility within the networking stack is needed to pinpoint these mistakes, and PacketMon provides that visibility.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="george-guirguis_1-1590133421148.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/193837iC08C91D26687A17A/image-size/medium?v=v2&amp;px=400" role="button" title="george-guirguis_1-1590133421148.png" alt="PacketMon's cross-component packet capture" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">PacketMon's cross-component packet capture</span></span></P> <P>PacketMon intercepts packets at multiple locations throughout the networking stack, exposing the packet route. If a packet was dropped by a supported component in the networking stack, PacketMon will report that packet drop. This allows users to differentiate between a component that is the intended destination for a packet and a component that is interfering with a packet. Additionally, PacketMon will report drop reasons; for example, MTU Mistmatch, or Filtered VLAN, etc. These drop reasons provide the root cause of the issue without the need to exhaust all the possibilities. PacketMon also provides packet counters for each intercept point to allow a high-level packet flow examination without the need for time-consuming log analysis.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Super Enhanced Drop Reason cropped.gif" style="width: 346px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/193842iADF7229A20E3D07B/image-size/medium?v=v2&amp;px=400" role="button" title="Super Enhanced Drop Reason cropped.gif" alt="PacketMon's packet drop and drop reason reporting" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">PacketMon's packet drop and drop reason reporting</span></span></P> <P>&nbsp;</P> <H3>Functionality:</H3> <P>&nbsp;</P> <P>Packetmon was first released in Windows 10 and Windows Server 2019 version 1809 (October 2018 update). Since then, its functionality has been evolving through Windows releases. Below are some of the main capabilities and limitations of PacketMon in Windows 10 and Windows Server 2019 version 2004 (May 2020 Update).</P> <P>&nbsp;</P> <H5><FONT size="3">Capabilities:</FONT></H5> <UL> <LI><SPAN>Packet capture at&nbsp;multiple locations&nbsp;of&nbsp;the networking stack&nbsp;</SPAN></LI> <LI><SPAN>Packet&nbsp;drop detection, including drop reason reporting</SPAN></LI> <LI><SPAN>Runtime packet filtering with encapsulation support&nbsp;</SPAN></LI> <LI><SPAN>Flexible packet&nbsp;counters</SPAN></LI> <LI><SPAN>Real-time on-screen packet monitoring&nbsp;</SPAN></LI> <LI><SPAN>High volume in-memory logging</SPAN></LI> <LI><SPAN>Microsoft Network Monitor (NetMon) and Wireshark (pcapng)&nbsp;</SPAN><SPAN>compatibility</SPAN></LI> </UL> <H5><FONT size="3"><SPAN>Limitations:</SPAN></FONT></H5> <UL> <LI> <P><SPAN>Supports Ethernet media type only</SPAN></P> </LI> <LI> <P><SPAN>No Firewall integration</SPAN></P> </LI> <LI> <P><SPAN>Drop reporting is only available for supported components</SPAN></P> </LI> </UL> <P><SPAN>&nbsp;</SPAN></P> <H2>Summary</H2> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Packet Monitor</A> is an in-box network diagnostics tool. It fills a gap in diagnosing virtual environments by providing visibility within the networking stack as it captures packets throughout the networking stack and reports packet drops. In subsequent posts, we will explore how to get started with PacketMon, and how to use it to diagnose specific scenarios. For documentation about PacketMon, please go <A href="#" target="_blank" rel="noopener">here</A>.</P> Thu, 24 Dec 2020 19:44:52 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-packet-monitor/ba-p/1410594 george-guirguis 2020-12-24T19:44:52Z Windows Insiders can now test DNS over HTTPS https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-insiders-can-now-test-dns-over-https/ba-p/1381282 <P><EM>Credit and thanks for feature work to Alexandru Jercaianu and Vladimir Cernov</EM></P> <P>&nbsp;</P> <P>If you have been waiting to try DNS over HTTPS (DoH) on Windows 10, you're in luck: the first testable version is now available to Windows Insiders!&nbsp;If you haven’t been waiting for it, and are wondering what DoH is all about, then be aware this feature will change how your device connects to the Internet and is in an early testing stage so only proceed if you’re sure you’re ready. Having said that, if you want to see the Windows DoH client in action and help us create a more private Internet experience for our customers, here is what you need to do:</P> <P>&nbsp;</P> <H1>Step 1: How do I get a Windows build with DoH support?</H1> <P>&nbsp;</P> <P>First, make sure your Microsoft account is part of the Windows Insider Program. If you know you are already a Windows Insider, make sure you are in the Fast ring and go to Step 2. If not, go <A href="#" target="_blank" rel="noopener">here</A> and follow the instructions for the Fast ring so you can get the latest Insider Preview build.</P> <P>Once this is done, run Windows Update, reboot, and verify you’re running Build 19628 or higher. You can do this by <SPAN>clicking here</SPAN> or by going to the Settings app -&gt; System -&gt; About.</P> <P>&nbsp;</P> <H1>Step 2: How do I turn on the DoH feature?</H1> <P>&nbsp;</P> <P>Once you know your Windows install has our DoH client, we need to activate it. You can do that by:</P> <UL> <LI>Opening the Registry Editor</LI> <LI>Navigate to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters registry key</LI> <LI>Create a new DWORD value named “EnableAutoDoh”</LI> <LI>Set its value to 2</LI> </UL> <P>&nbsp;</P> <P><STRONG>Please note: the registry keys and values described here are only for enabling DoH client testing on Insider builds. When the DoH client is made available in general release builds, registry configuration of DoH will not be supported.</STRONG></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tojens_0-1589221350608.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/190644iA5F817EC22538D60/image-size/medium?v=v2&amp;px=400" role="button" title="tojens_0-1589221350608.png" alt="tojens_0-1589221350608.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <H1>Step 3: How do I add DoH servers to Windows?</H1> <P>&nbsp;</P> <P>Now that the DoH client is active, Windows will start using DoH if you already have one of these servers configured:</P> <P>&nbsp;</P> <TABLE> <TBODY> <TR> <TD width="312px"> <P><STRONG>Server Owner</STRONG></P> </TD> <TD width="312px"> <P><STRONG>Server IP addresses</STRONG></P> </TD> </TR> <TR> <TD width="312px"> <P><STRONG>Cloudflare</STRONG></P> </TD> <TD width="312px"> <P>1.1.1.1</P> <P>1.0.0.1</P> <P>2606:4700:4700::1111</P> <P>2606:4700:4700::1001</P> </TD> </TR> <TR> <TD width="312px"> <P><STRONG>Google</STRONG></P> </TD> <TD width="312px"> <P>8.8.8.8</P> <P>8.8.4.4</P> <P>2001:4860:4860::8888</P> <P>2001:4860:4860::8844</P> </TD> </TR> <TR> <TD width="312px"> <P><STRONG>Quad9</STRONG></P> </TD> <TD width="312px"> <P>9.9.9.9</P> <P>149.112.112.112</P> <P>2620:fe::fe</P> <P>2620:fe::fe:9</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>&nbsp;</P> <P>You can configure Windows to use any of these IP addresses as a DNS server through the Control Panel or the Settings app. The next time the DNS service restarts, we’ll start using DoH to talk to these servers instead of classic DNS over port 53. The easiest way to trigger a DNS service restart is by rebooting the computer.</P> <P>&nbsp;</P> <P>To add a DNS server in the Control Panel:</P> <UL> <LI>Go to Network and Internet -&gt; Network and Sharing Center -&gt; Change adapter settings.</LI> <LI>Right click on the connection you want to add a DNS server to and select Properties.</LI> <LI>Select either “Internet Protocol Version 4 (TCP/IPv4)” or “Internet Protocol Version 6 (TCP/IPv6)” and click Properties.</LI> <LI>Ensure the “Use the following DNS server addresses” radio button is selected and add the DNS server address into the fields below.</LI> </UL> <P>&nbsp;</P> <H1>Step 4: How do I know DoH is working?</H1> <P>&nbsp;</P> <P>Now that you have Windows configured to use DoH, you should be able to verify it’s working by seeing no more plain text DNS traffic from your device. You can do this by using Packetmon, a network traffic analyzer included with Windows.</P> <P>Start by opening a new Command Prompt or PowerShell window. Run the following command to reset any network traffic filters PacketMon may already have in place.</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="bash">pktmon filter remove</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Run the following command to add a traffic filter for port 53, the port classic DNS uses (and which should now be silent since we’re only using DoH).</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="bash">pktmon filter add -p 53</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Run the following command to start a real-time logging of traffic. All port 53 packets will be printed to the command line. If your device is only configured with DoH servers, this should show little to no traffic.</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="bash">pktmon start --etw -m real-time</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <H1>Step 5: How do I use a DoH server that isn’t on the auto-promotion list?</H1> <P>&nbsp;</P> <P>If you’re trying to test a DoH server that isn’t already on our auto-promotion list, such as your ISP’s DoH servers, you can add it to our list manually using the command line. First, identify the IP address and the DoH URI template for the server you want to add. Then, run the following command as an administrator:</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="bash">netsh dns add encryption server=&lt;your-server’s-IP-address&gt; dohtemplate=&lt;your-server’s-DoH-URI-template&gt;</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>You can verify the template was applied to the well-known DoH server list by running this command, which should show you the template being used for a given IP address:</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="bash">netsh dns show encryption server=&lt;your-server’s-IP-address&gt;</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Now when Windows is configured to use that IP address as a DNS server, it will use DoH instead of classic DNS.</P> Wed, 05 Aug 2020 17:39:13 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-insiders-can-now-test-dns-over-https/ba-p/1381282 tojens 2020-08-05T17:39:13Z MsQuic is Open Source https://gorovian.000webhostapp.com/?exam=t5/networking-blog/msquic-is-open-source/ba-p/1345441 <P>Microsoft is open sourcing our QUIC library, <A href="#" target="_blank" rel="noopener">MsQuic</A>, on GitHub under an <A href="#" target="_blank" rel="noopener">MIT license</A>. MsQuic is a cross-platform, general-purpose library that implements the <A href="#" target="_blank" rel="noopener">QUIC</A> transport protocol. QUIC is being standardized by the <A href="#" target="_blank" rel="noopener">Internet Engineering Task Force</A> (IETF). MsQuic is a client and server solution optimized for multiple usage patterns and is used by multiple Microsoft products and services. MsQuic is currently in preview and supports Windows and Linux.</P> <P>&nbsp;</P> <H2>Development Status</H2> <P>We are open sourcing MsQuic on GitHub both to support internal partners and to share our solution with the community. GitHub is the primary workspace for MsQuic, containing all the product and test code. We’re using <A href="#" target="_blank" rel="noopener">Azure Pipelines</A> to provide cross-platform builds and to run over 4,000 test cases for every commit and pull request. We are still stabilizing our GitHub environment and not yet taking external contributions, but we will in the future.</P> <P>&nbsp;</P> <P>There are several substantial benefits from moving to GitHub.</P> <UL> <LI>Code reviews will be public via the <A href="#" target="_blank" rel="noopener">GitHub’s pull request model</A>, allowing for more external input during the development process.</LI> <LI>Issues will also be tracked on GitHub to allow for better public discussion and visibility.</LI> <LI>Documentation will be in the same place as the code.</LI> </UL> <P>&nbsp;</P> <P>We hope this increased public visibility into the overall development process of the product will help build a vibrant community around MsQuic.</P> <P>MsQuic has consistently been one of the <A href="#" target="_blank" rel="noopener">most interoperable</A> implementations at the cross-industry IETF interop events. We have successfully validated interop of MsQuic with canary versions of browsers like Chrome and Edge.</P> <P>&nbsp;</P> <H2>Adoption and Deployment</H2> <P>Windows will ship with MsQuic in the kernel to support various inbox features. The Windows HTTP/3 stack is being built on top of MsQuic.</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Microsoft 365</A> is testing a preview version of IIS using HTTP/3 to reduce tail loss latencies in the last mile. This is currently active in internal dogfood environments.</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">.NET Core</A> has built HTTP/3 support into <A href="#" target="_blank" rel="noopener">Kestrel</A> and <A href="#" target="_blank" rel="noopener">HttpClient</A> on top of MsQuic. HTTP/3 support is in experimental preview for the 5.0 release of .NET Core.</P> <P>&nbsp;</P> <P><A href="https://gorovian.000webhostapp.com/?exam=t5/itops-talk-blog/smb-over-quic-files-without-the-vpn/ba-p/1183449" target="_blank" rel="noopener">SMB in Windows is also prototyping</A> MsQuic usage. QUIC brings several benefits for SMB, such as better internet reachability, a secured connection based on industry standard TLS and server authentication with certificate validation. Best of all, this brings a completely different workload on top of MsQuic, strengthening the general-purpose nature of the transport.</P> <P>&nbsp;</P> <P>There are several other Microsoft teams prototyping with user mode MsQuic, targeting a diverse set of use cases.</P> <P>&nbsp;</P> <H2>Looking Forward</H2> <P>Microsoft is an active participant and driver of QUIC in the industry and is consequently open sourcing our implementation as a reference for others. MsQuic brings performance and security improvements to many important networking scenarios. Our online services benefit the most from performance improvements like reduced tail latency and faster connection setup. Our connections will be able to seamlessly switch networks because they can survive IP address/port changes. This equates to better user experience on our edge devices.</P> <P>&nbsp;</P> <P>MsQuic, like the IETF QUIC standard, is a work in progress and does not have an official release yet, but it is ready for prototyping and testing. You can look forward to another blog with a technical deep dive into MsQuic.</P> <P>&nbsp;</P> <P>For more info on MsQuic continue reading <A href="#" target="_blank" rel="noopener">on GitHub</A>.</P> Tue, 28 Apr 2020 22:49:20 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/msquic-is-open-source/ba-p/1345441 Daniel Havey 2020-04-28T22:49:20Z Introducing SDNSecurityToolkit https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-sdnsecuritytoolkit/ba-p/1269482 <P><EM>Thanks to William Conlon for authoring today's post!</EM></P> <P>&nbsp;</P> <P>Every software producer has faced this challenge at some point: balancing flexibility with security. The most basic requirement of quality in any piece of software is that it will not expose the user to attackers, but in some cases the features of the software require us to allow the user to make mistakes that might expose vulnerabilities. In the world of operating systems, for example, we must balance letting our users configure their system how they want with protecting them to our best ability. To accomplish this we provide our users with tools to scan their system and alert them when we find something we know to be malicious or where a best practice is not being followed.</P> <P>&nbsp;</P> <P>In the world of Software Defined Networking, even with a completely secure platform there is always the possibility that a customer will misconfigure a rule and expose their system. Our SDN team for Windows Server 2019 recognizes this. While there is no way that we can ensure that an SLB rule will never be misconfigured, we know how complicated SDN deployments can be and want to empower our users with tools to protect themselves. After all, deployments may reach thousands of VIPs, load balancing rules, and inbound and outbound NATs, all of which may be changing dynamically from day to day. With more of the world migrating to cloud, we can expect users to run into complex problems if we do not provide them with effective tools for visualizing the ways that they are exposing their system to the outside world.</P> <P>&nbsp;</P> <P>To this end, we have created a new tool for analyzing your public VIPs. It is a simple PowerShell module called <A href="#" target="_blank">SDNSecurityToolkit</A>, now available on the Microsoft SDN GitHub. After loading the module you will be able to use the Invoke-SDNVipScan cmdlet. To use this cmdlet, you only need to supply the connection URI for your existing SDN deployment and the path to an Nmap executable. The tool returns an object containing all its findings and you may also specify a path for it to output a basic HTML report or to save this information as a JSON object.</P> <P>&nbsp;</P> <P>The tool runs in three stages, by running with the -verbose flag you can see these occurring in real time. The samples below have slightly different coloring for demonstration purposes. First, the tool queries across all existing load balancers for public VIPs and gathers information about what rules are making use of those VIPs:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tojens_6-1585692453108.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/181068i2D33F14C747E825B/image-size/medium?v=v2&amp;px=400" role="button" title="tojens_6-1585692453108.png" alt="tojens_6-1585692453108.png" /></span></P> <P>&nbsp;</P> <P>To provide deeper insight, the tool then scans the exposed ports for these VIPs with Nmap to look at what might be listening on the other side. This is to acknowledge the multiple layers of security that may be in play, for instance a deployment may intentionally leave certain routes open on the load balancer side while securing VMs on the backend with firewall rules or other means:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tojens_7-1585692482855.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/181069i723B4F81FF3F7764/image-size/medium?v=v2&amp;px=400" role="button" title="tojens_7-1585692482855.png" alt="tojens_7-1585692482855.png" /></span></P> <P>&nbsp;</P> <P>Finally, the tool does a quick scan of your entire public address space as it is configured in your logical network lists with the Network Controller. If it finds an open address there, it cross references against your load balancer rules and alerts you if this address is not registered and should not be exposed. In the end, it provides a full report of exposed public IP addresses matched with rules and ports and whether said ports appear as open or filtered during the Nmap scan.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="tojens_8-1585692504516.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/181070i0366EDB4DDA79B9D/image-size/medium?v=v2&amp;px=400" role="button" title="tojens_8-1585692504516.png" alt="tojens_8-1585692504516.png" /></span></P> <P>&nbsp;</P> <P>You can find a more technical description of how to set the flags and run this tool in its README, along with instructions on installing the Nmap dependency. We hope that by providing this we will give our users an edge when it comes to maintaining secure SDN deployments. We look forward to creating more tools for securing SDN deployments within this module, including an ACL scanner, which will help WS19 admins validate isolation between tenant SDN networks.</P> Wed, 01 Apr 2020 13:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-sdnsecuritytoolkit/ba-p/1269482 tojens 2020-04-01T13:00:00Z L2bridge Container Networking https://gorovian.000webhostapp.com/?exam=t5/networking-blog/l2bridge-container-networking/ba-p/1180923 <P><STRONG>Overview</STRONG></P> <P>Containers attached to a l2bridge network will be directly connected to the physical network through an&nbsp;external&nbsp;Hyper-V switch. L2bridge networks can be configured with the same IP subnet as the container host, with IPs from the physical network assigned statically. L2bridge networks can also be configured using a custom IP subnet through a HNS host endpoint that is configured as a gateway.</P> <P>&nbsp;</P> <P>In l2bridge, all container frames will have the same MAC address as the host due to Layer-2 address translation (MAC re-write) operation on ingress and egress. For larger, cross-host container deployments, this helps reduce the stress on switches having to learn MAC addresses of sometimes short-lived containers. Whenever container hosts are virtualized, this comes with the additional advantage that we do not need to enable <A href="#" target="_blank" rel="noopener">MAC address spoofing</A> on the VM NICs of the container hosts for container traffic to reach destinations outside of their host.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="overview.png" style="width: 523px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/172381iAE6E20D3FB9C92D7/image-size/large?v=v2&amp;px=999" role="button" title="overview.png" alt="Reference l2bridge network" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Reference l2bridge network</span></span></P> <P>There are several networking scenarios that are essential to successfully containerize and connect a distributed set of services, such as:</P> <OL> <LI>Outbound connectivity (Internet access)</LI> <LI>DNS resolution</LI> <LI>Container name resolution</LI> <LI>Host to container connectivity (and vice versa)</LI> <LI>Container to container connectivity (local)</LI> <LI>Container to container connectivity (remote)</LI> <LI>Binding container ports to host ports</LI> </OL> <P>We will be showing all the above on l2bridge and briefly touch on some more advanced use-cases:</P> <OL start="8"> <LI>Creating an HNS container load balancer</LI> <LI>Defining and applying network access control lists (ACLs) to container endpoints</LI> <LI>Attaching multiple NICs to a single container</LI> </OL> <DIV id="tinyMceEditorDavid Schott_0" class="mceNonEditable lia-copypaste-placeholder">&nbsp;</DIV> <P><STRONG>Pre-requisites</STRONG></P> <P>In order to follow along, 2x Windows Server machines (Windows Server, version 1809 or above) are required with:</P> <UL> <LI>Containers feature and container runtime (e.g. Docker) installed</LI> <LI>HNS Powershell Helper Module</LI> </UL> <P>To achieve this, run the following commands on the machines:</P> <P>&nbsp;</P> <LI-CODE lang="c">Install-WindowsFeature -Name Containers -Restart Install-PackageProvider -Name NuGet -RequiredVersion 2.8.5.201 -Force Install-Module -Name DockerMsftProvider -Repository PSGallery -Force Install-Package -Name Docker -ProviderName DockerMsftProvider -Force Start-Service Docker Start-BitsTransfer https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/hns.psm1</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Creating an L2bridge network</STRONG></P> <P>Many of the needed policies to setup l2bridge are conveniently <A href="#" target="_blank" rel="noopener">exposed</A> through Docker’s libnetwork driver on Windows.</P> <P>For example, an l2bridge network of name “winl2bridge” with subnet 10.244.3.0/24 can be created as follows:</P> <P>&nbsp;</P> <LI-CODE lang="c">docker network create -d l2bridge --subnet=10.244.3.0/24 -o com.docker.network.windowsshim.dnsservers=10.127.130.7,10.127.130.8 --gateway=10.244.3.1 -o com.docker.network.windowsshim.enable_outboundnat=true -o com.docker.network.windowsshim.outboundnat_exceptions=10.244.0.0/16,10.10.0.0/24,10.127.130.36/30 winl2bridge</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>The available options for network creation are documented in 2 locations (see <A href="#" target="_blank" rel="noopener">#1 here</A> and <A href="#" target="_blank" rel="noopener">#2 here</A>) but here is a table breaking down all the arguments used:</P> <TABLE border="1" width="100%"> <TBODY> <TR> <TD width="50%" height="27px"><STRONG>Name</STRONG></TD> <TD width="50%" height="27px"><STRONG>Description</STRONG></TD> </TR> <TR> <TD width="50%" height="54px"> <P><FONT color="#3366FF">-d</FONT></P> </TD> <TD width="50%" height="54px"> <P>Type of driver to use for network creation</P> </TD> </TR> <TR> <TD width="50%" height="54px"> <P><FONT color="#3366FF">--subnet</FONT></P> </TD> <TD width="50%" height="54px"> <P>Subnet range to use for network in CIDR notation</P> </TD> </TR> <TR> <TD width="50%" height="54px"> <P><FONT color="#3366FF">-o com.docker.network.windowsshim.dnsservers</FONT></P> </TD> <TD width="50%" height="54px"> <P>List of DNS servers to assign to containers.</P> </TD> </TR> <TR> <TD width="50%" height="27px"> <P><FONT color="#3366FF">--gateway</FONT></P> </TD> <TD width="50%" height="27px"> <P>IPv4 Gateway of the assigned subnet.</P> </TD> </TR> <TR> <TD width="50%" height="107px"> <P><FONT color="#3366FF">-o com.docker.network.windowsshim.enable_outboundnat</FONT></P> </TD> <TD width="50%" height="107px"> <P>Apply outbound NAT HNS policy to container vNICs/endpoints. All traffic from the container will be SNAT’ed to the host IP. If the container subnet is not routable, this policy is needed for containers to reach destinations outside of their own respective subnet.</P> </TD> </TR> <TR> <TD width="50%" height="187px"> <P><FONT color="#3366FF">-o com.docker.network.windowsshim.outboundnat_exceptions</FONT></P> </TD> <TD width="50%" height="187px"> <P>List of destination IP ranges in CIDR notation where NAT operations will be skipped. This will typically include the container subnet (e.g. 10.244.0.0/16), load balancer subnet (e.g. 10.10.0.0/24), and a range for the container hosts (e.g. 10.127.130.36/30).</P> </TD> </TR> </TBODY> </TABLE> <P><STRONG><FONT color="#FF0000">IMPORTANT:</FONT></STRONG> Usually, l2bridge requires that the specified gateway (“10.244.3.1”) exists somewhere in the network infrastructure and that the gateway provides proper routing for our designated prefix. We will be showing an alternative approach where we will create an HNS endpoint on the host from scratch and configure it so that it acts as a gateway.</P> <P><STRONG><FONT color="#FF6600">NOTE:</FONT></STRONG> You may see a network blip for a few seconds while the vSwitch is being created for the first l2bridge network.</P> <P><STRONG><FONT color="#339966">TIP:</FONT></STRONG> You can create multiple l2bridge networks on top of a single vSwitch, “consuming” only one NIC. It is even possible to isolate the networks by VLAN using -o com.docker.network.windowsshim.vlanid flag.</P> <P>&nbsp;</P> <P>Next, we will enable forwarding on the host vNIC and setup a host endpoint as a quasi gateway for the containers to use.</P> <P>&nbsp;</P> <LI-CODE lang="c"># Import HNS Powershell module ipmo .\hns.psm1 # Enable forwarding netsh int ipv4 set int "vEthernet (Ethernet)" for=en $network = get-hnsnetwork | ? Name -Like $(docker network inspect --format='{{.ID}}' winl2bridge) # Create default gateway (need to use x.x.x.2 as x.x.x.1 is already reserved) $hnsEndpoint = New-HnsEndpoint -NetworkId $network.ID -Name cbr0_ep -IPAddress 10.244.3.2 -Verbose # Attach gateway endpoint to host network compartment Attach-HnsHostEndpoint -EndpointID $hnsEndpoint.Id -CompartmentID 1 # Enable forwarding for default gateway netsh int ipv4 set int "vEthernet (cbr0_ep)" for=en netsh int ipv4 add neighbors "vEthernet (cbr0_ep)" "10.244.3.1" "00-01-e8-8b-2e-4b" </LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><FONT color="#FF6600"><STRONG>NOTE: </STRONG></FONT>The last netsh command above would not be needed if we supplied a proper gateway that exists in the network infrastructure at network creation. Since we created a host endpoint to use in place of a gateway, we need to add a static ARP entry with a dummy MAC so that traffic is able to leave our host without being stuck waiting for an ARP probe to resolve this gateway IP.</P> <P>&nbsp;</P> <P>This is all that is needed to setup a local l2bridge container network with working outbound connectivity, DNS resolution, and of course container to container and container to host connectivity.</P> <P>&nbsp;</P> <P><STRONG>Multi-host Deployment</STRONG></P> <P>One of the most compelling reasons for using l2bridge is the ability to connect containers not only on the local machine, but also with remote machines to form a network. For communication across container hosts, one needs to plumb static routes so that each host knows where a given container lives.</P> <P>&nbsp;</P> <P>For demonstration, assume there are 2 container host machines (Host “A”, Host “B”) with IP 10.127.132.38 and 10.127.132.36 and container subnets 10.244.2.0/24 and 10.244.3.0/24 respectively.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="l2bridge_internode.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/172382i52E7C44E0E9811C3/image-size/large?v=v2&amp;px=999" role="button" title="l2bridge_internode.gif" alt="Static routes for cross-node l2bridge container connectivity" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Static routes for cross-node l2bridge container connectivity</span></span></P> <P>&nbsp;</P> <P>To realize connecting containers across the 2 hosts, the following commands would need to be executed on host A:</P> <P>&nbsp;</P> <LI-CODE lang="c">New-NetRoute -InterfaceAlias "vEthernet (Ethernet)" -DestinationPrefix 10.244.3.0/24 -NextHop 10.127.132.36</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Similarly, on host B the following also needs to be executed:</P> <P>&nbsp;</P> <LI-CODE lang="c">New-NetRoute -InterfaceAlias "vEthernet (Ethernet)" -DestinationPrefix 10.244.2.0/24 -NextHop 10.127.132.38</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Now l2bridge containers running both locally and on remote hosts can communicate with each other.</P> <P>&nbsp;</P> <P><FONT color="#339966"><STRONG>TIP:</STRONG></FONT> On public cloud platforms, one also needs to add these routes to the default system’s route table, so the underlying host cloud network knows how to forward packets with container IPs to the correct destination. For instance on Azure, <A href="#" target="_blank" rel="noopener">user-defined routes</A> of type “virtual appliance” would need to be added to the Azure virtual network. If host A and host B were VMs provisioned in an Azure resource group “$Rg”, this could be done by issuing the following <A href="#" target="_blank" rel="noopener">az</A> commands:</P> <P>&nbsp;</P> <LI-CODE lang="c">az network route-table create --resource-group $Rg --name BridgeRoute az network route-table route create --resource-group $Rg --address-prefix 10.244.3.0/24 --route-table-name BridgeRoute --name HostARoute --next-hop-type VirtualAppliance --next-hop-ip-address 10.127.130.36 az network route-table route create --resource-group $Rg --address-prefix 10.244.2.0/24 --route-table-name BridgeRoute --name HostBRoute --next-hop-type VirtualAppliance --next-hop-ip-address 10.127.130.38</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><STRONG>Starting l2bridge containers</STRONG></P> <P>Once all static routes have been updated and l2bridge network created on each host, it is simple to spin up containers and attach them to the l2bridge network.</P> <P>&nbsp;</P> <P>For example, to spin up two IIS containers with ID “c1”, “c2” on container subnet with gateway “10.244.3.1”:</P> <P>&nbsp;</P> <LI-CODE lang="markup">$array = @("c1", "c2") $array |foreach { docker run -d --rm --name $_ --hostname $_ --network winl2bridge mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 docker exec $_ cmd /c netsh int ipv4 add neighbors "Ethernet" "10.244.3.1" "00-01-e8-8b-2e-4b" }</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P><FONT color="#FF6600"><STRONG>NOTE: </STRONG></FONT>The last netsh command above would not be needed if we supplied a proper gateway that exists in the network infrastructure at network creation. Since we created a host endpoint to use in place of a gateway, we need to add a static ARP entry with a dummy MAC so that traffic is able to leave our host without being stuck waiting for an ARP probe to resolve this gateway IP.</P> <P>&nbsp;</P> <P>Here is a video demonstrating all the connectivity paths available after launching the containers:</P> <P><LI-VIDEO vid="https://youtu.be/pRWCmqrYSBU" align="center" size="medium" width="400" height="300" uploading="false" thumbnail="https://i.ytimg.com/vi/pRWCmqrYSBU/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <P><STRONG>Publishing container ports to host ports</STRONG></P> <P>One feature to expose containerized applications and make them more available is to map container ports to an external port on the host.</P> <P>For example, to map TCP container port 80 to the host port 8080, and assuming the container has respective endpoint with ID “<FONT color="#3366FF">448c0e22-a413-4882-95b5-2d59091c11b8</FONT>” this can be achieved using an ELB policy as follows:</P> <P>&nbsp;</P> <LI-CODE lang="c">ipmo .\hns.psm1 $publish_json = '{ "References": [ "/endpoints/448c0e22-a413-4882-95b5-2d59091c11b8" ], "Policies": [ { "Type": "ELB", "InternalPort": 80, "ExternalPort": 8080, "Protocol": 6 } ] }' Invoke-HNSRequest -Method POST -Type policylists -Data $publish_json </LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Here is a video demonstrating how to apply the policy to bind a TCP container port to a host port and access it:</P> <P><LI-VIDEO vid="https://youtu.be/XttEb0s3H9c" align="center" size="medium" width="400" height="225" uploading="false" thumbnail="https://i.ytimg.com/vi/XttEb0s3H9c/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <P><STRONG>Advanced: Setting up Load Balancers</STRONG></P> <P>The ability to distribute traffic across multiple containerized backends using a load balancer leads to higher scalability and reliability of applications.</P> <P>&nbsp;</P> <P>For example, creating a load balancer with frontend virtual IP (VIP) 10.10.0.10:8090 on host A (IP 10.127.130.36) and backend DIPs of all local containers can be achieved as follows:</P> <P>&nbsp;</P> <LI-CODE lang="c">ipmo .\hns.psm1 [GUID[]] $endpoints = (Get-HNSEndpoint |? Name -Like "Ethernet" | Select ID).ID New-HNSLoadBalancer -Endpoints $endpoints -InternalPort 80 -ExternalPort 8090 -Vip "10.10.0.10"</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Finally, for the load balancer to be accessible from inside the containers, we also need to add two encapsulation rules for every endpoint that needs to access the load balancer:</P> <P>&nbsp;</P> <LI-CODE lang="c">$endpoints | foreach { $encap_lb = '{ "References": [ "/endpoints/' + $_ +'" ], "Policies": [ { "Type": "ROUTE", "DestinationPrefix": "10.10.0.0/24", "NeedEncap": true } ] }' $encap_mgmt = '{ "References": [ "/endpoints/' + $_ +'" ], "Policies": [ { "Type": "ROUTE", "DestinationPrefix": "10.127.130.36/32", "NeedEncap": true } ] }' Invoke-HNSRequest -Method POST -Type policylists -Data $encap_lb Invoke-HNSRequest -Method POST -Type policylists -Data $encap_mgmt }</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Here is a video showing how to create the load balancer and access it using its frontend VIP "10.10.0.10" from host and container:</P> <P><LI-VIDEO vid="https://youtu.be/JLde-dLMRcg" align="center" size="medium" width="400" height="225" uploading="false" thumbnail="https://i.ytimg.com/vi/JLde-dLMRcg/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <P><STRONG>Advanced: Setting up ACLs</STRONG></P> <P>What if instead of making applications more available, one needs to restrict traffic between containers? l2bridge networks are ideally suited for network access control lists (ACLs) that define policies which limit network access to only those workloads that are explicitly permitted.</P> <P>For example, to allow inbound network access to TCP port 80 from IP 10.244.3.75 and block all other inbound traffic to container with endpoint “<FONT color="#3366FF">448c0e22-a413-4882-95b5-2d59091c11b8</FONT>”:</P> <P>&nbsp;</P> <LI-CODE lang="c">ipmo .\hns.psm1 $json = '{ "Policies": [ { "Type": "ACL", "Action": "Allow", "Direction": "In", "LocalAddresses": "", "RemoteAddresses": "10.244.3.75", "LocalPorts": "80", "Protocol": 6, "Priority": 200 }, { "Type": "ACL", "Action": "Block", "Direction": "In", "Priority": 300 }, { "Type": "ACL", "Action": "Allow", "Direction": "Out", "Priority": 300 } ] }' Invoke-HNSRequest -Method POST -Type endpoints -Id "448c0e22-a413-4882-95b5-2d59091c11b8" -Data $acl_json</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Here is a video showing the ACL policy in action and how to apply it:</P> <P><LI-VIDEO vid="https://youtu.be/pMgx7mMb7no" align="center" size="medium" width="400" height="225" uploading="false" thumbnail="https://i.ytimg.com/vi/pMgx7mMb7no/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <P>Access control lists and Windows fire-walling is a very deep and complex topic. HNS supports more granular capabilities to implement network micro-segmentation and govern traffic flows than shown above. Most of these enhancements are available via <A href="#" target="_blank" rel="noopener">Tigera’s Calico for Windows</A> product and will be incrementally documented <A href="#" target="_self">here</A> and <A href="#" target="_self">here</A>.</P> <P>&nbsp;</P> <P><STRONG>Advanced: Multi-NIC containers</STRONG></P> <P>Attaching multiple vNICs to a single container addresses various traffic segregation and operational concerns. For example, assume there are two VLAN-isolated L2bridge networks called “winl2bridge_4096” and “winl2bridge_4097”:</P> <P>&nbsp;</P> <LI-CODE lang="c"># Create “winl2bridge_4096” with VLAN tag 4096 docker network create -d l2bridge --subnet=10.244.4.0/24 -o com.docker.network.windowsshim.dnsservers=10.127.130.7 --gateway=10.244.4.1 -o com.docker.network.windowsshim.enable_outboundnat=true -o com.docker.network.windowsshim.outboundnat_exceptions=10.244.0.0/16,11.96.0.0/24,10.127.130.36/30 -o com.docker.network.windowsshim.vlanid=4096 winl2bridge_4096 # Create “winl2bridge_4097” with VLAN tag 4097 docker network create -d l2bridge --subnet=10.244.5.0/24 -o com.docker.network.windowsshim.dnsservers=10.127.130.7 --gateway=10.244.5.1 -o com.docker.network.windowsshim.enable_outboundnat=true -o com.docker.network.windowsshim.outboundnat_exceptions=10.244.0.0/16,11.96.0.0/24,10.127.130.36/30 -o com.docker.network.windowsshim.vlanid=4097 winl2bridge_4097</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>Attaching a container to both networks can be done as follows:</P> <P>&nbsp;</P> <LI-CODE lang="c"># Create container and attach to “winl2bridge_4096” docker run -d --rm --name "multi_nic_container" --network "winl2bridge_4096" mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 # Attach to “winl2bridge_4097” docker network connect "winl2bridge_4097" "multi_nic_container"</LI-CODE> <P>&nbsp;</P> <P>To add more vNICs, we can create HNS endpoints under a given network and attach them to the container’s network compartment. For example, to add another NIC in network “winl2bridge_4096”:</P> <P>&nbsp;</P> <LI-CODE lang="c"># Get compartment ID $compartmentId = docker exec "multi_nic_container" powershell.exe "Get-NetCompartment | Select -ExpandProperty CompartmentId" # Get HNS network ID $network = get-hnsnetwork | ? Name -Like $(docker network inspect --format='{{.ID}}' winl2bridge_4096) # Create HNS endpoint under network “winl2bridge_4096” $hnsEndpoint = New-HnsEndpoint -NetworkId $network.ID -Name my_ep -IPAddress 10.244.4.10 -Verbose # Attach endpoint to target container’s network compartment Attach-HnsHostEndpoint -EndpointID $hnsEndpoint.Id -CompartmentID $compartmentId</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>By executing all the above, a single container has three vNICs ready to use now (two in “winl2bridge_4096”, one from “winl2bridge_4097”). Every endpoint may have different policies and configurations specifically tailored to meet the needs of the application and business.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="multi_nic_container.png" style="width: 684px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/172616i8D31866D2F33C235/image-size/large?v=v2&amp;px=999" role="button" title="multi_nic_container.png" alt="Container with multiple endpoints belonging to two different networks" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Container with multiple endpoints belonging to two different networks</span></span></P> <P>&nbsp;</P> <P><STRONG>Summary</STRONG></P> <P>We have covered several supported capabilities of l2bridge container networking, including:</P> <UL> <LI>Cross-host container communication (not possible via WinNAT)</LI> <LI>Logical separation of networks by VLANs</LI> <LI>Micro-segmentation using ACLs</LI> <LI>Load balancers</LI> <LI>Binding container ports to host ports</LI> <LI>Attaching multiple network adapters to containers</LI> </UL> <P>L2bridge networks require upfront configuration to install correctly but offers many useful features as well as enhanced performance and control of the container network. It is always recommended to leverage orchestrators such as Kubernetes which utilize CNI plugins to streamline and automate many of these configuration tasks, while still rewarding advanced users with a similar level of configurability. All of the HNS APIs used above and much more are also open-source in a Golang shim (see <A href="#" target="_blank" rel="noopener">hcsshim</A>).</P> <P>&nbsp;</P> <P>As always, thanks for reading and please let us know about your scenarios or questions in the comments section below!</P> Wed, 17 Jun 2020 17:54:34 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/l2bridge-container-networking/ba-p/1180923 David Schott 2020-06-17T17:54:34Z Teaming in Azure Stack HCI https://gorovian.000webhostapp.com/?exam=t5/networking-blog/teaming-in-azure-stack-hci/ba-p/1070642 <P><EM>Written by Dan Cuomo, Senior PM at Microsoft. Follow him on Twitter<SPAN>&nbsp;</SPAN><STRONG><A href="#" target="_self" rel="nofollow noreferrer">@dan2_2023</A></STRONG></EM></P> <P>&nbsp;</P> <LI-SPOILER>As we're deprecating the vSwitch attached to an LBFO team, this article introduces a new tool for converting your LBFO team to a SET team. To download this tool, run the following command or see the end of this article. <PRE class="lia-code-sample language-csharp"><CODE>Install-Module Convert-LBFO2SET​</CODE></PRE> </LI-SPOILER> <P>Windows Server currently has two inbox teaming mechanisms with two very different purposes. In this article, we’ll describe several reasons why you should use Switch Embedded Teaming (SET) for Azure Stack HCI scenarios and we’ll discuss several long-held teaming myths – We’d love to hear your feedback in the comments below. Let's get started!</P> <P>&nbsp;</P> <P>In Windows Server 2012 we released LBFO as an inbox teaming mechanism, with many customers leveraging this technology to provide load-balancing and fail-over between network adapters. Since then, the rise of software-defined storage and software defined networking has brought performance and compatibility challenges to the forefront (outlined in this article) with the LBFO architecture that required a change in direction.</P> <P>&nbsp;</P> <P>This new direction is called Switch Embedded Teaming (SET) and was introduced in Windows Server 2016. SET is available when Hyper-V is installed on any Server OS (Windows Server 2016 and higher) and Windows 10 version 1809 (and higher). You’re not required to run virtual machines to use SET, but the Hyper-V role must be installed.</P> <P>&nbsp;</P> <P>In summary, LBFO is our older teaming technology that will not see future investment, is not compatible with numerous advanced capabilities, and has been exceeded in both performance and stability by our new technology (SET). We'd like to discuss why you should move off LBFO for virtualized and cloud scenarios. Let's dig into this paragraph a bit.</P> <P>&nbsp;</P> <P>&nbsp;</P> <H1>LBFO is our older teaming technology that will not see future investment</H1> <P><span class="lia-inline-image-display-wrapper lia-image-align-left" image-alt="clipboard_image_0.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/162241iE761DB1911ECCAF2/image-size/medium?v=v2&amp;px=400" role="button" title="clipboard_image_0.png" alt="clipboard_image_0.png" /></span></P> <P>With the intent to bring software-defined technologies like SDNv2 and containers to Windows Server, it became clear that we needed an alternative teaming solution and so we set off creating SET, circa 2014. Simultaneously reaching feature parity and stability with LBFO took time; several early adopters of SET will remember some of these pains. However, SET’s stability, performance, and features have now far surpassed LBFO.</P> <P>&nbsp;</P> <P>All new features released since Windows Server 2016 (see below) <STRONG>were developed and tested with SET in mind</STRONG> – This includes all Azure Stack HCI solutions you may have purchased; Azure Stack HCI <STRONG>is not tested or certified with LBFO</STRONG>. This is largely due to development simplicity and testing; without driving too far into unimportant details, LBFO teams adapters inside NDIS which is a large and complex component – its roots <A href="#" target="_blank" rel="noopener">date back to Windows 95</A> (of course updated considerably since then). If your system has a NIC, you’re using NDIS. In the picture shown above, each component below the vSwitch was part of NDIS.</P> <P>&nbsp;</P> <P>The size and complexity of scenarios included in NDIS made for very complex testing requirements that were only compounded by virtualized and software-defined technologies that considerably hampered feature innovation. You might think that this is just a Microsoft problem, but really this affects NIC vendor driver development time and stability as well.</P> <P>&nbsp;</P> <P>All-in-all, we’re not focusing on LBFO much these days, particularly as software-defined Windows Server networking scenarios become more exotic with the rise of containers, software-defined networking, and much more. There’s a faster, more stable, and performant teaming solution, called Switch Embedded Teaming.</P> <H1>LBFO is not compatible with several advanced capabilities</H1> <P>Here’s a smattering of scenarios and features that are supported with SET but <STRONG>NOT</STRONG> LBFO:</P> <P>&nbsp;</P> <P><STRONG>Windows Admin Center</STRONG>&nbsp;- WAC is the de facto management tool for Windows Server and Azure Stack HCI, with millions of nodes under management. You can create and manage a SET team for a single host, or deploy a SET team to multiple hosts with the new Cluster Creation UI we released to help you deploy Azure Stack HCI solutions at Microsoft Ignite this year (<A href="#" target="_blank" rel="noopener">watch the session</A>, <A href="#" target="_blank" rel="noopener">try it out</A>, and give us feedback).</P> <P>&nbsp;</P> <P>LBFO is not available for configuration in Windows Admin Center.</P> <P>&nbsp;</P> <P><STRONG>RDMA Teaming</STRONG> -&nbsp;Only SET can team RDMA adapters. RDMA is used for example with Storage Spaces Direct (S2D) which requires a reliable&nbsp;<A href="#" target="_blank" rel="noopener">high bandwidth, low latency</A> network connection between each node. High-bandwidth? Low Latency? That’s RDMA’s bag so it is the recommended pattern with S2D. Reliability? That’s SET's claim-to-fame so these two are a logical pairing.</P> <P>&nbsp;</P> <P><STRONG>Guest RDMA: </STRONG>SET supports RDMA into a virtual machine. This doesn’t work with LBFO for two reasons:</P> <UL> <LI>RDMA adapters cannot be teamed with LBFO both host adapters and virtual adapters.</LI> <LI>RDMA uses SMB multichannel which requires multiple adapters to balance traffic across. Since you can’t assign a vNIC to pNIC affinity with LBFO, neither the SMB nor non-SMB traffic can be made highly available.</LI> </UL> <P><STRONG>Guest Teaming </STRONG>is a strange one; you could add multiple virtual NICs to a Hyper-V VM; inside the VM, you could use LBFO to team the virtual NICs. However, you cannot affinitize a virtual NIC (vNIC) to a physical NIC (pNIC), so it’s possible that both vNICs added in the VM are sending and receiving traffic out of the same pNIC. If that pNIC fails, you lose both of your virtual NICs.&nbsp;</P> <P>&nbsp;</P> <P>SET allows you to map each vNIC to a pNIC to ensure that they don’t overlap, ensuring that a Guest Team is able to survive an adapter outage.</P> <P>&nbsp;</P> <P><STRONG>Microsoft Software Defined Networking </STRONG>- (SDN) was first released in its modern form in Windows Server 2016 and requires a virtual switch extension called the Virtual Filtering Platform (VFP). VFP is the brains behind SDN, the same extension that runs our public cloud, Azure. VFP can only be added to a SET team.</P> <P>&nbsp;</P> <P>This means that any of the&nbsp;<A href="#" target="_blank" rel="noopener">SDN features&nbsp;</A><SPAN style="font-family: inherit;">(which are included with a Datacenter Edition license) like the Software Load Balancer, Gateways, Distributed Firewall (ACLs), and our modern network QoS capability are also unavailable if you’re using LBFO.</SPAN></P> <P>&nbsp;</P> <P><STRONG>Container Networking </STRONG>- Containers&nbsp;relies on a service called the Host Network Service (HNS). HNS also leverages VFP and as mentioned in the SDN section, VFP can only be added to a Switch Embedded Team (SET). For more information on Container Networking, please see <A href="#" target="_blank" rel="noopener">this link</A>.</P> <P>&nbsp;</P> <P><STRONG>Virtual Machine Multi-Queues -</STRONG> VMMQ is a critical performance feature for Azure Stack HCI. VMMQ allows you to assign multiple VMQs to the same virtual NIC without which, you rely on expensive software spreading operations (the OS spreads packets across multiple CPUs without hardware (NIC) assistance) that greatly increases CPU utilization on the host, reducing the number of virtual machines you can run.</P> <P>&nbsp;</P> <P>Moreover, if your vNIC doesn’t get a VMQ, all traffic is processed by the default queue. With SET you can assign multiple VMQs to the default queue which can be shared as needed by any vNIC allowing more VMs to get the bandwidth they need.</P> <P>&nbsp;</P> <P>In this video, you can see the performance (throughput) benefits of Switch Embedded Teaming over that of LBFO.&nbsp; The video demonstrates a 2x throughput improvement with SET over LBFO, while consuming ~10% additional CPU (a result of double the throughput).</P> <P>&nbsp;</P> <P><IFRAME src="https://www.youtube.com/embed/jxDPobc_TTo" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <P><STRONG>Dynamic VMMQ </STRONG>-&nbsp; d.VMMQ&nbsp;won’t work either. Dynamic VMMQ is an intelligent queue scheduling algorithm for VMMQ that recognizes when CPU cores are overworked by network traffic and reassigns that network traffic processing to other cores automatically so your workloads (e.g. VMs, applications, etc.) can run without competing for processor time.</P> <P>&nbsp;</P> <P>Here's an example of some of the benefits of Dynamic VMMQ. In the video, you can see the host, spending CPU resources processing packets for a specific virtual NIC. When a competing workload begins on the system (which would prevent the virtual NIC from reaching maximum performance), we automatically tune the system by moving one of the workloads to an available processor.</P> <P>&nbsp;</P> <P><IFRAME src="https://www.youtube.com/embed/WsrVS3LCCNM" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <P><STRONG>RSC in the vSwitch </STRONG>is an acceleration that coalesces segments destined for the same virtual NIC into a larger segment.</P> <P>&nbsp;</P> <P>Outbound network traffic is slimmed to fit into the mtu size of the physical network (default of ~1500 bytes). However, inbound traffic can be coalesced into one big segment. That one big segment takes far less processing than multiple small segments, so once traffic is received by the host, we can combine them and deliver several segments to the vNIC all at once. SET was made aware of RSC coalescing and supports this acceleration as of Windows Server 2019.</P> <P>&nbsp;</P> <P>We’re continuing to improve this feature for even better performance in the next version of Windows Server and Azure Stack HCI by enabling RSC in the vSwitch to extend over the VMBus. In the video below, we show one VM sending traffic to another VM with the improved acceleration disabled - This is using only the original Windows Server 2019, RSC in the vSwitch capabilities.&nbsp;</P> <P>&nbsp;</P> <P>Next we enable the Windows Server vNext improvements; throughput is improved by ~17 Gbps while CPU resourcing <STRONG>is reduced</STRONG> by approximately 12% (20 cores on the system). This type of traffic pattern is specifically valuable for container scenarios that reside on the same host.</P> <P>&nbsp;</P> <P><IFRAME src="https://www.youtube.com/embed/QKvlpsq5j3I" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <H1>LBFO has been exceeded in both performance and stability by SET</H1> <P><EM>Note: Guest RDMA, RSC in the vSwitch, VMMQ, and Dynamic VMMQ belong in this category as well.</EM></P> <P>&nbsp;</P> <P><STRONG>Certified Azure Stack and Azure Stack HCI solutions test only SET</STRONG></P> <P>If all that wasn’t enough, both Microsoft and our partners validate and certify their solutions on SET, not LBFO. If you bought a certified Azure Stack HCI solution from one of our partners <STRONG>OR</STRONG> a standard or premium logo’d NIC, it was tested and validated with Switch Embedded Teaming. That means all certification tests where run with SET.</P> <P>&nbsp;</P> <P><STRONG>Link Aggregation Control Protocol (LACP)</STRONG></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-left" image-alt="clipboard_image_0.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/168469i809A10F89B3E218D/image-size/medium?v=v2&amp;px=400" role="button" title="clipboard_image_0.png" alt="clipboard_image_0.png" /></span></P> <P>Ok, so this one is a little counter-intuitive. LACP, allows for port-channels or switch-dependent teams to send traffic to the host over more than one physical port simultaneously.</P> <P>&nbsp;</P> <P>For<EM> <STRONG>native hosts</STRONG></EM> this means that every port in the port-channel can send traffic simultaneously – for the system on the right with 2 x 50 Gbps NICs, it looks like one big pipe with a <EM>native host</EM> potentially receiving 100 Gbps. Naturally, you'd expect that this capability could extend to virtual NICs as well.</P> <P>&nbsp;</P> <P>But things change with virtualization. When the traffic gets to the host, the NICs need to interrupt multiple, independent processors to exceed what a single CPU core can process – This is what VMMQ does, and as mentioned, VMMQ does not work with LBFO.</P> <P>&nbsp;</P> <P>LBFO limits you to a single VMQ and despite having (in the picture) 100 Gbps of inbound bandwidth, you would only receive about 5 Gbps per virtual NIC (or up to ~20 Gbps per vNIC at the painful expense of OS-based software spreading that could be used for running virtual machine workloads).</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-left" image-alt="clipboard_image_1.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/168470i46F2364CE22C92ED/image-size/medium?v=v2&amp;px=400" role="button" title="clipboard_image_1.png" alt="clipboard_image_1.png" /></span></P> <P>&nbsp;</P> <P>With SET, switch-independent teaming, and the hardware assistance of VMMQ and enough CPUs in the system, you could receive all 100 Gbps of data into the host.</P> <P>&nbsp;</P> <P>In summary, LACP provides no throughput benefits for Azure Stack HCI scenarios, incurs higher CPU consumption, and cannot auto-tune the system to avoid competition between workloads for virtualized scenarios (Dynamic VMMQ).</P> <P>&nbsp;</P> <P><STRONG>Asymmetric Adapters</STRONG></P> <P>While we're myth-busting, let’s talk about adapter symmetry which describes the length to which adapters have the same make, model, speed, and configuration – SET requires adapter symmetry for Microsoft support. Usually the easiest way to identify this symmetry is by the device Interface Description (with PowerShell, use Get-NetAdapter). If the interface description matches (with exception of the unique number given to each adapter e.g. Intel NIC #1, Intel NIC #2, etc.) then the adapters are symmetric.</P> <P>&nbsp;</P> <P>Prior to Windows Server 2016, conventional wisdom stated that you should use different NICs with different drivers in a team. The thinking was that if one driver had an issue, another team member would survive, and the team would remain up. This is a common benefit customers cite in favor of LBFO: it supports asymmetric adapters.</P> <P>&nbsp;</P> <P>However, two drivers mean twice as many things can go wrong in fact increasing the likelihood of a problem. Instead, a properly designed infrastructure with symmetric adapters are far more stable in our review of customer support cases. As a result, support for asymmetric teams are no longer a differentiator for LBFO nor do we recommend it for Azure Stack HCI scenarios where reliability is the #1 requirement.</P> <P>&nbsp;</P> <P><STRONG>LBFO for management adapters</STRONG></P> <P>Some customers I’ve worked with have asked if they should use LBFO for management adapters when the vSwitch is not attached – Our recommendation is to always use SET whenever available. A management adapter’s goal in life is to be stable and we see less support cases with SET.</P> <P>&nbsp;</P> <P>To be clear, if the adapter is not attached to a virtual switch, LBFO is acceptable however, you should endeavor to use SET whenever possible due to the support reasons outlined in this article.</P> <P>&nbsp;</P> <P><FONT size="6"><STRONG>vSwitch on LBFO Deprecation Status</STRONG></FONT></P> <P><FONT size="3">Recently, <A href="#" target="_self">we publicly announced</A> our plans to deprecate the use of LBFO with the Hyper-V virtual switch. Moving forward, and due to the various reasons outlined in this article, we have decided to block the binding of the vSwitch on LBFO.</FONT></P> <P>&nbsp;</P> <P><FONT size="3">Prior to upgrading from Windows Server 2019 to vNext or if you have a fresh install of vNext, you will need to convert any LBFO teams to a SET team if it's attached to a Hyper-V virtual switch. To make this simpler, we're releasing a tool (available on the PowerShell gallery) called&nbsp;<A href="#" target="_self">Convert-LBFO2SET</A>.</FONT></P> <P>&nbsp;</P> <P><FONT size="3">You can install this tool using the command:</FONT></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="csharp">Install-Module Convert-LBFO2SET</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Or for disconnected systems:</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <LI-CODE lang="csharp">Save-Module Convert-LBFO2SET -Path C:\SomeFolderPath</LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><FONT size="3">Please see the wiki for instructions on how to use the tool however here's an example where we convert a system with 10 host vNICs, 10 generation 1 VMs, and 10 generation 2 VMs.&nbsp;</FONT></P> <P><FONT size="3">&nbsp;</FONT></P> <P><IFRAME src="https://www.youtube.com/embed/Aedjp2_y4bU" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"></IFRAME></P> <P>&nbsp;</P> <P><STRONG>Summary</STRONG></P> <P>LBFO remains our teaming solution if you are running your workloads on bare metal servers. If however, you are running virtualized or cloud scenarios like Azure Stack HCI, you should give Switch Embedded Teaming serious consideration. As we’ve described in this article, SET has been the Microsoft recommended teaming solution and focus since Windows Server 2016 as it brings better performance, stability, and feature support compared to LBFO.</P> <P>&nbsp;</P> <P>Are there other questions you have about SET and LBFO? Please submit your questions in the comments below!</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>Dan “All SET for Azure Stack HCI” Cuomo</P> <P>&nbsp;</P> Sat, 21 Mar 2020 17:31:32 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/teaming-in-azure-stack-hci/ba-p/1070642 Dan Cuomo 2020-03-21T17:31:32Z Windows will improve user privacy with DNS over HTTPS https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-will-improve-user-privacy-with-dns-over-https/ba-p/1014229 <P><EM>Brought to you by Tommy Jensen, Ivan Pashov, and Gabriel Montenegro</EM></P> <P>&nbsp;</P> <P>Here in Windows Core Networking, we’re interested in keeping your traffic as private as possible, as well as fast and reliable. While there are many ways we can and do approach user privacy on the wire, today we’d like to talk about encrypted DNS. Why? Basically, because supporting encrypted DNS queries in Windows will close one of the last remaining plain-text domain name transmissions in common web traffic.<BR /><BR /></P> <P>Providing encrypted DNS support without breaking existing Windows device admin configuration won't be easy. However, <A href="#" target="_blank">at Microsoft we believe that</A> "we have to treat privacy as a human right. We have to have end-to-end cybersecurity built into technology."</P> <P>&nbsp;</P> <P>We also believe Windows adoption of encrypted DNS will help make the overall Internet ecosystem healthier. There is an assumption by many that DNS encryption requires DNS centralization. This is only true if encrypted DNS adoption isn’t universal. To keep the DNS decentralized, it will be important for client operating systems (such as Windows) and Internet service providers alike to <A href="#" target="_blank">widely adopt encrypted DNS</A>.</P> <P>&nbsp;</P> <P>With the decision made to build support for encrypted DNS, the next step is to figure out what kind of DNS encryption Windows will support and how it will be configured. Here are our team's guiding principles on making those decisions:</P> <P>&nbsp;</P> <UL> <LI><STRONG>Windows DNS needs to be as private and functional as possible by default without the need for user or admin configuration because Windows DNS traffic represents a snapshot of the user’s browsing history.</STRONG> To Windows users, this means their experience will be made as private as possible by Windows out of the box. For Microsoft, this means we will look for opportunities to encrypt Windows DNS traffic without changing the configured DNS resolvers set by users and system administrators.</LI> <LI><STRONG>Privacy-minded Windows users and administrators need to be guided to DNS settings even if they don't know what DNS is yet.</STRONG> Many users are interested in controlling their privacy and go looking for privacy-centric settings such as app permissions to camera and location but may not be aware of or know about DNS settings or understand why they matter and may not look for them in the device settings.</LI> <LI><STRONG>Windows users and administrators need to be able to improve their DNS configuration with as few simple actions as possible.</STRONG> We must ensure we don't require specialized knowledge or effort on the part of Windows users to benefit from encrypted DNS. Enterprise policies and UI actions alike should be something you only have to do once rather than need to maintain.</LI> <LI><STRONG>Windows users and administrators need to explicitly allow fallback from encrypted DNS once configured.</STRONG> Once Windows has been configured to use encrypted DNS, if it gets no other instructions from Windows users or administrators, it should assume falling back to unencrypted DNS is forbidden.</LI> </UL> <P>&nbsp;</P> <P>Based on these principles, we are making plans to adopt <A href="#" target="_blank">DNS over HTTPS</A> (or DoH) in the Windows DNS client. As a platform, Windows Core Networking seeks to enable users to use whatever protocols they need, so we’re open to having other options such as DNS over TLS (DoT) in the future. For now, we're prioritizing DoH support as the most likely to provide immediate value to everyone. For example, DoH allows us to reuse our existing HTTPS infrastructure.</P> <P>&nbsp;</P> <P>For our first milestone, we'll start with a simple change: use DoH for DNS servers Windows is already configured to use. There are now several public DNS servers that support DoH, and if a Windows user or device admin configures one of them today, Windows will just use classic DNS (without encryption) to that server. However, since these servers and their DoH configurations are well known, Windows can automatically upgrade to DoH while using the same server. We feel this milestone has the following benefits:</P> <P>&nbsp;</P> <UL> <LI><STRONG>We will not be making any changes to which DNS server Windows was configured to use by the user or network.</STRONG> Today, users and admins decide what DNS server to use by picking the network they join or specifying the server directly; this milestone won’t change anything about that. Many people use ISP or public DNS content filtering to do things like block offensive websites. Silently changing the DNS servers trusted to do Windows resolutions could inadvertently bypass these controls and frustrate our users. We believe device administrators have the right to control where their DNS traffic goes.</LI> <LI><STRONG>Many users and applications that want privacy will start getting the benefits without having to know about DNS. </STRONG>In line with principle 1, the DNS queries become more private with no action from either apps or users. When both endpoints support encryption, there’s no reason to wait around for permission to use encryption!</LI> <LI><STRONG>We can start seeing the challenges in enforcing the line on preferring resolution failure to unencrypted fallback. </STRONG>In line with principle 4, this DoH use will be enforced so that a server confirmed by Windows to support DoH will not be consulted via classic DNS. If this preference for privacy over functionality causes any disruption in common web scenarios, we’ll find out early.</LI> </UL> <P>&nbsp;</P> <P>In future milestones, we'll need to create more privacy-friendly ways for our users to discover their DNS settings in Windows as well as make those settings DoH-aware. This will give users, device admins, and enterprise admins the ability to configure DoH servers explicitly.&nbsp;</P> <P>&nbsp;</P> <P>Why announce our intentions in advance of DoH being available to Windows Insiders? With encrypted DNS gaining more attention, we felt it was important to make our intentions clear as early as possible. We don’t want our customers wondering if their trusted platform will adopt modern privacy standards or not.</P> <P>&nbsp;</P> <P>If you are interested in joining the larger industry conversation about encrypting the DNS, check out one of the IETF working groups working with DNS (<A href="#" target="_blank">ABCD</A>, <A href="#" target="_blank">Apps Doing DNS</A>, <A href="#" target="_blank">DNSOP</A>, <A href="#" target="_blank">DPRIVE</A>) or the new <A href="#" target="_blank">Encrypted DNS Deployment Initiative</A>.</P> <P>&nbsp;</P> <P>Do you have questions or feedback for us regarding the Windows plan to adopt encrypted DNS? We’d love to hear from you! Feel free to comment below.</P> Mon, 18 Nov 2019 05:00:55 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-will-improve-user-privacy-with-dns-over-https/ba-p/1014229 tojens 2019-11-18T05:00:55Z Core Networking and Internet Standards https://gorovian.000webhostapp.com/?exam=t5/networking-blog/core-networking-and-internet-standards/ba-p/924431 <H1>Internet Standards</H1> <P>&nbsp;</P> <P>Constantly throughout the day, we (and devices on our behalf) all use the Internet:&nbsp; TCP, UDP, HTTP, TLS, DHCP, DNS, and countless other protocols that define the Internet or improve how it runs.&nbsp; How do these foundational protocols get created? Let's talk about standards. In particular, let's talk about one of the organizations Windows Core Networking participates in: the Internet Engineering Task Force (<A href="#" target="_blank">IETF</A>).&nbsp; The IETF is the primary standards developing organization (SDO) for protocols and practices that run and improve the Internet. And it does so in a very particular manner, as SDOs go:</P> <UL> <LI>It espouses open development principles. All specifications ("Requests for Comments" or "RFCs") and drafts in progress ("Internet-Drafts") are openly available.</LI> <LI>It has no formal membership. All one needs to participate is an email address (participation is supposed to be by individuals, rather than companies, groups or governments).</LI> <LI>Decisions are based on "rough consensus and running code". No formal procedures like Robert's Rules of Order.&nbsp; No voting.&nbsp; Working group chairs gauge consensus via methods such as informal polls (e.g., <A href="#" target="_blank">hums</A>, show of hands), as well as mailing list discussions.</LI> </UL> <P>&nbsp;The IETF has been around since 1986 and has published over eight thousand RFCs.&nbsp;</P> <P>&nbsp;The IETF is evolving and adapting. It is increasingly adopting practices of the software development world. GitHub is often used to develop specifications. Furthermore, implementations of protocols and their interoperability is the goal of the largest growth area: the "Hackathon".</P> <H1>Microsoft Core Networking Contributions</H1> <P>Microsoft as a whole has been a technical leader in IETF standards activities for decades. Past contributions have benefitted HTTP, IPv6, WebRTC, PPP, PPTP/L2TP, RADIUS &amp; EAP, DNS, iCalendar, WEBDAV, IoT, security, IP mobility, routing and myriad other topics.</P> <P>As for specific contributions by Microsoft Core Networking, let's examine QUIC as an example. QUIC is a new transport protocol for the Internet. QUIC is designed to be more secure, performant and evolvable compared to TCP. HTTP/3 is the next version of the HTTP protocol which will exclusively use QUIC as the transport. Microsoft has been involved right from the beginning of the standardization effort. Like many other recent Internet-Drafts, the working group has adopted GitHub for all work including issue tracking.</P> <P>Over the past year, Microsoft Core Networking has contributed several dozen pull requests and issues which have resulted in major changes to the set of QUIC and HTTP/3 Internet-Drafts, and lots of email and in-person discussions. Microsoft has also participated in every interop event held since the beginning and our implementation has been one of the most interoperable each time. As an example of a lasting contribution, Microsoft was an early and vocal advocate of using standard TLS 1.3 as the security mechanism in QUIC, which was the approach that then gained consensus. Microsoft also participated in many design teams which helped overcome issues that could have delayed the standardization efforts.&nbsp; The results of all these efforts have direct benefits to our products.&nbsp; Early testing shows QUIC outperforms TCP and we expect it to reduce last mile latency for online services.</P> <P>&nbsp;</P> <P>Other contributions from Windows Core Networking (either directly as editors or working with the main authors and relevant working group) are on topics such as TCP congestion control (DCTCP, CUBIC, LEDBAT++, rLEDBAT), TCP optimizations (TCP Fast Open, RACK, HyStart++), &nbsp;IPv6 (including IPv6 for IoT), WebSockets, and HTTP/2.</P> <P>Additionally, we have been active otherwise:</P> <UL> <LI>As members of the Internet Architecture Board</LI> <LI>And by chairing or co-chairing the following IETF Working Groups: 6lo (IPv6 over Networks of Resource-constrained Nodes), HyBi (websockets), 16ng (WiMax), CSI (IPv6 secure neighbor discovery), behave (Behavior Engineering for Hindrance Avoidance), pcp (Port Control Protocol) and mipshop (Mobility for IP: Performance, Signaling and Handoff Optimization).</LI> </UL> <P>In summary, Microsoft recognizes that in an increasingly interconnected computing landscape, enabling interoperability between products from different vendors has become more important than ever.&nbsp; Support for standards is a key part of Microsoft's <A href="#" target="_blank">Interoperability Principles</A> for the company, and support for open standards in particular helps reinforce to the industry that "Microsoft Runs on Trust".</P> <P>&nbsp;</P> <P>Dave Thaler</P> <P>Gabriel Montenegro</P> <P>Praveen Balasubramanian</P> Tue, 22 Oct 2019 13:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/core-networking-and-internet-standards/ba-p/924431 Gabriel Montenegro 2019-10-22T13:00:00Z Windows Containers and Rancher 2.3 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-containers-and-rancher-2-3/ba-p/874943 <P><SPAN data-contrast="auto">By Mike&nbsp;</SPAN><SPAN data-contrast="auto">Kostersitz</SPAN><SPAN data-contrast="auto">, Principal Program Manager, Microsoft</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Container technology is transforming the face of business and application development. 70% of on-premises workloads today&nbsp;</SPAN><SPAN data-contrast="auto">are running</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">on the Windows Server operating system and enterprise customers are looking to modernize these workloads and make use of containers.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">We have introduced support for Windows Containers in Windows Server 2016 and graduated support for Windows Server worker nodes in Kubernetes&nbsp;</SPAN><SPAN data-contrast="auto">1.14&nbsp;</SPAN><SPAN data-contrast="auto">clusters</SPAN><SPAN data-contrast="auto">. W</SPAN><SPAN data-contrast="auto">ith Windows Server 2019&nbsp;</SPAN><SPAN data-contrast="auto">we have</SPAN><SPAN data-contrast="auto">&nbsp;expanded support in Kubernetes 1.16.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">For our customers one of the preferred ways to increase the adoption of containers and Kubernetes is to work to make it easier for operators to deploy it and for developers to use it.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-contrast="auto">Towards that end Microsoft has invested in AKS and Windows Container support with this goal in mind while workin</SPAN><SPAN data-contrast="auto">g with partners such as Rancher Labs who has built their organization on the principle of "Run Kubernetes Everywhere".</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">With the release of Rancher 2.3, Rancher is the first to have graduated Windows support to GA and can now deploy Kubernetes clusters with Windows support from within the user experience.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Using Rancher 2.3 users can deploy Windows Kubernetes clusters in AKS, Azure Cloud, any other cloud computing provider or on-premises using the supported and proven&nbsp;</SPAN><SPAN data-contrast="auto">n</SPAN><SPAN data-contrast="auto">etwork&nbsp;</SPAN><SPAN data-contrast="auto">components in Windows Server as well as Kubernetes.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Rancher 2.3 will support Flannel as the CNI plugin and Overlay Networking with&nbsp;</SPAN><SPAN data-contrast="auto">VxLAN</SPAN><SPAN data-contrast="auto">&nbsp;to enable communication between Windows and Linux&nbsp;</SPAN><SPAN data-contrast="auto">containers, services</SPAN><SPAN data-contrast="auto">,</SPAN><SPAN data-contrast="auto">&nbsp;and application</SPAN><SPAN data-contrast="auto">s</SPAN><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">To learn more about Rancher Labs 2.3 and the functionality check out this link</SPAN><SPAN data-contrast="auto">: <A href="#" target="_blank">https://rancher.com/products/rancher/2.3</A></SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> Tue, 08 Oct 2019 16:05:41 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-containers-and-rancher-2-3/ba-p/874943 mkostersitz 2019-10-08T16:05:41Z Disabling Legacy TLS https://gorovian.000webhostapp.com/?exam=t5/networking-blog/disabling-legacy-tls/ba-p/887568 <P class="xxmsonormal">Hi, my name is Gabriel Montenegro and I am a Program Manager in Windows Core Networking working on web technologies.</P> <P class="xxmsonormal">&nbsp;</P> <P class="xxmsonormal">The secure web (HTTP over TLS, also known as “HTTPS”) is increasingly prevalent. Good.</P> <P class="xxmsonormal">&nbsp;</P> <P class="xxmsonormal">However, there are several versions of TLS and there are many ciphers that it can use, and some of those combinations are not nearly as strong any more. Bad.</P> <P class="xxmsonormal">&nbsp;</P> <P class="xxmsonormal">It is imperative to be able to specify a minimum level of security&nbsp;for your connections, for example, at least version 1.2 of TLS, disallow weaker and outdated ciphers.</P> <P class="xxmsonormal">&nbsp;</P> <P class="xxmsonormal">Please read our blog on new capabilities in Windows Server 2019 to enforce improved security on connections:</P> <P class="xxmsonormal"><A href="#" target="_blank" rel="noopener">https://www.microsoft.com/security/blog/2019/09/30/tls-version-enforcement-capabilities-now-available-certificate-binding-windows-server-2019/</A></P> Wed, 02 Oct 2019 13:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/disabling-legacy-tls/ba-p/887568 Gabriel Montenegro 2019-10-02T13:00:00Z Evolution of Timekeeping in Windows https://gorovian.000webhostapp.com/?exam=t5/networking-blog/evolution-of-timekeeping-in-windows/ba-p/778020 <P><EM><SPAN style="font-family: inherit;" data-contrast="auto">(Thanks to Sarath Madakasira for writing this post)</SPAN></EM></P> <P>&nbsp;</P> <P><SPAN style="font-family: inherit;" data-contrast="auto">Do you e</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">ver wonder&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">how</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">Windows computers&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">keep&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">the&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">current&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">time</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">&nbsp;and&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">how it is maintained on a day to day basis</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">?</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">&nbsp;Did you know that your computer needs to keep the correct time&nbsp;</SPAN><SPAN style="font-family: inherit;" data-contrast="auto">for you to access certain resources over the network?&nbsp;</SPAN><SPAN style="font-family: inherit;" data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Are you aware of regulations on time synchronization in certain industries in several regions across the globe and why that may be required?&nbsp;</SPAN><SPAN data-contrast="auto">Did you know that some of the recent releases of Windows OS support&nbsp;</SPAN><STRONG><SPAN data-contrast="auto">sub-millisecond&nbsp;</SPAN></STRONG><SPAN data-contrast="auto">time synchronization accuracy?</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Do you know what Precision Time Protocol is?&nbsp;</SPAN><SPAN data-contrast="auto">You may have seen&nbsp;</SPAN><SPAN data-contrast="auto">recent posts about&nbsp;</SPAN><SPAN data-contrast="auto">various aspects of&nbsp;</SPAN><SPAN data-contrast="auto">accurate</SPAN><SPAN data-contrast="auto">&nbsp;time</SPAN><SPAN data-contrast="auto">keeping</SPAN><SPAN data-contrast="auto">&nbsp;on the&nbsp;</SPAN><SPAN data-contrast="auto">Windows Networking&nbsp;</SPAN><SPAN data-contrast="auto">blog</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">(e.g.&nbsp;&nbsp;</SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener"><SPAN data-contrast="none">1</SPAN></A><SPAN data-contrast="auto">,&nbsp;</SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Windows-Subsystem-for-Linux-for-testing-Windows-10-PTP-Client/ba-p/389181" target="_blank" rel="noopener"><SPAN data-contrast="none">2)</SPAN></A><SPAN data-contrast="auto">.</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">E</SPAN><SPAN data-contrast="auto">ver wonder&nbsp;</SPAN><SPAN data-contrast="auto">what&nbsp;</SPAN><SPAN data-contrast="auto">Windows Networking has&nbsp;</SPAN><SPAN data-contrast="auto">anything&nbsp;</SPAN><SPAN data-contrast="auto">to do with&nbsp;</SPAN><SPAN data-contrast="auto">timekeeping</SPAN><SPAN data-contrast="auto">?&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Would you like to about&nbsp;</SPAN><SPAN data-contrast="auto">the underlying technology that enables sub-millisecond time accuracy in Windows OS?&nbsp;</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Read on to know more…</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">Accurate timekeeping has become a requirement&nbsp;</SPAN><SPAN data-contrast="auto">for&nbsp;</SPAN><SPAN data-contrast="auto">a</SPAN><SPAN data-contrast="auto">&nbsp;variety of applications and Windows OS provides native support for accurate timekeeping and synchronization&nbsp;</SPAN><SPAN data-contrast="auto">in a</SPAN><SPAN data-contrast="auto">&nbsp;range of&nbsp;</SPAN><SPAN data-contrast="auto">deployment&nbsp;</SPAN><SPAN data-contrast="auto">topologies</SPAN><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">We are publishing a document</SPAN><SPAN data-contrast="auto">&nbsp;(attached below)</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">that paint</SPAN><SPAN data-contrast="auto">s</SPAN><SPAN data-contrast="auto">&nbsp;an overall picture of&nbsp;</SPAN><SPAN data-contrast="auto">timekeeping in operating systems</SPAN><SPAN data-contrast="auto">,&nbsp;</SPAN><SPAN data-contrast="auto">the&nbsp;</SPAN><SPAN data-contrast="auto">various&nbsp;</SPAN><SPAN data-contrast="auto">improvements&nbsp;</SPAN><SPAN data-contrast="auto">to timekeeping&nbsp;</SPAN><SPAN data-contrast="auto">in&nbsp;</SPAN><SPAN data-contrast="auto">the&nbsp;</SPAN><SPAN data-contrast="auto">Windows OS&nbsp;</SPAN><SPAN data-contrast="auto">over the past&nbsp;</SPAN><SPAN data-contrast="auto">several releases&nbsp;</SPAN><SPAN data-contrast="auto">and how these improvements helped us realize&nbsp;</SPAN><SPAN data-contrast="auto">improved&nbsp;</SPAN><SPAN data-contrast="auto">time accuracy</SPAN><SPAN data-contrast="auto">&nbsp;in&nbsp;</SPAN><SPAN data-contrast="auto">the&nbsp;</SPAN><SPAN data-contrast="auto">Windows OS</SPAN><SPAN data-contrast="auto">. We also have some notes on&nbsp;</SPAN><SPAN data-contrast="auto">where we&nbsp;</SPAN><SPAN data-contrast="auto">may be&nbsp;</SPAN><SPAN data-contrast="auto">headed next</SPAN><SPAN data-contrast="auto">&nbsp;in this space</SPAN><SPAN data-contrast="auto">.</SPAN><SPAN data-contrast="auto">&nbsp;Fair warning that&nbsp;</SPAN><SPAN data-contrast="auto">the document&nbsp;</SPAN><SPAN data-contrast="auto">is a long read.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">We&nbsp;</SPAN><SPAN data-contrast="auto">hope that this&nbsp;</SPAN><SPAN data-contrast="auto">informational&nbsp;</SPAN><SPAN data-contrast="auto">document&nbsp;</SPAN><SPAN data-contrast="auto">help</SPAN><SPAN data-contrast="auto">s</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">the readers&nbsp;</SPAN><SPAN data-contrast="auto">understand the complexities of timekeeping</SPAN><SPAN data-contrast="auto">, as well as&nbsp;</SPAN><SPAN data-contrast="auto">our passion, commitment and successes</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">with&nbsp;</SPAN><SPAN data-contrast="auto">accurate timekeeping in&nbsp;</SPAN><SPAN data-contrast="auto">the&nbsp;</SPAN><SPAN data-contrast="auto">Windows</SPAN><SPAN data-contrast="auto">&nbsp;OS</SPAN><SPAN data-contrast="auto">.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P>&nbsp;</P> <P><SPAN data-contrast="auto">The</SPAN><SPAN data-contrast="auto">&nbsp;released</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">features&nbsp;</SPAN><SPAN data-contrast="auto">that are&nbsp;</SPAN><SPAN data-contrast="auto">described in&nbsp;</SPAN><SPAN data-contrast="auto">this document&nbsp;</SPAN><SPAN data-contrast="auto">are&nbsp;</SPAN><SPAN data-contrast="auto">covered in detail by&nbsp;</SPAN><SPAN data-contrast="auto">our</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><A href="#" target="_blank" rel="noopener"><SPAN data-contrast="none">publications</SPAN></A><SPAN data-contrast="auto">&nbsp;and&nbsp;</SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener"><SPAN data-contrast="none">blog posts</SPAN></A><SPAN data-contrast="auto">&nbsp;and ha</SPAN><SPAN data-contrast="auto">ve</SPAN><SPAN data-contrast="auto">&nbsp;been available to customers who sought to take advantage of high accuracy time keeping in Windows.</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">We&nbsp;</SPAN><SPAN data-contrast="auto">will&nbsp;</SPAN><SPAN data-contrast="auto">publis</SPAN><SPAN data-contrast="auto">h</SPAN><SPAN data-contrast="auto">&nbsp;</SPAN><SPAN data-contrast="auto">details of&nbsp;</SPAN><SPAN data-contrast="auto">any&nbsp;</SPAN><SPAN data-contrast="auto">new time sync features/improvements in upcoming Windows releases when they are available for public use.</SPAN><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> <P><SPAN data-ccp-props="{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}">&nbsp;</SPAN></P> Mon, 05 Aug 2019 13:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/evolution-of-timekeeping-in-windows/ba-p/778020 tojens 2019-08-05T13:00:00Z Direct Server Return (DSR) in a nutshell https://gorovian.000webhostapp.com/?exam=t5/networking-blog/direct-server-return-dsr-in-a-nutshell/ba-p/693710 <DIV class="lia-message-subject-wrapper lia-component-subject lia-component-message-view-widget-subject-with-options">&nbsp;</DIV> <DIV class="lia-message-body-wrapper lia-component-message-view-widget-body"> <DIV id="bodyDisplay" class="lia-message-body"> <DIV class="lia-message-body-content"> <P><EM><STRONG>TL;DR;</STRONG></EM></P> <P><EM>A question we get asked quite a bit is: "What is Microsoft doing to improve Networking performance?"</EM>.</P> <P>To answer that we have today’s blog post, which is about load balancing performance enhancements in SDN networks for Windows Containers and Kubernetes Networks, especially for services.</P> <P>One of these enhancements is Direct Server Return (DSR) routing for overlay and l2bridge networks.</P> <P>DSR is available in Windows Server 19H1 or later.</P> <H2 id="toc-hId-1767504680">What it is</H2> <P>DSR is an implementation of asymmetric network load distribution in load balanced systems, meaning that the request and response traffic use a different network path.</P> <P>The use of different network paths helps avoid extra hops and reduces the latency by which not only speeds up the response time between the client and the service but also removes some extra load from the load balancer.<SPAN>&nbsp;</SPAN><BR />Using DSR is a transparent way to achieve increased network performance for your applications with little to no infrastructure changes.</P> <P>While DSR will improve your application’s network performance, there are a couple of things to keep in mind when enabling DSR:</P> <UL> <LI>Persistence is limited to source IP or destination IP (no cookie persistence)</LI> <LI>SSL offloading on the load balancer is not going work as they need to see both inbound and outbound traffic.</LI> <LI>There might be some ARP issues with some operating systems</LI> </UL> <P>Both the outbound and inbound packets pass through the filtering/load balancing layer and the root name space to reach their destination Service VIP resulting in delays due to the time it takes to process the packet in each layer both ways.</P> <P>In the below diagram you can see the typical non-DSR flow. As you can see both the inbound as well as the outbound packets between POD1 and POD2 are traversing through all components of the network.</P> <OL> <LI>Packet leaves POD1 addressed from 192.168.1.3:5555 to the lService VIP address 10.0.0.10:53</LI> <LI>Packet enters VMswitch at Port 3 and a SNAT rule is applied changing the destination address to 192.168.0.5:6666 to route the package through the ROOT namespace.</LI> <LI>The packet is then routed back to 192.168.1.4:53 through the VMSwitch Port 4 carrying the source address 192.168.0.5 of the ROOT Namespace.</LI> <LI>The return path follows the steps in reverse order and the packet will have to travel through the ROOT Namespace again to get back to the original source.</LI> </OL> <P><SPAN class="lia-inline-image-display-wrapper lia-image-align-inline"><SPAN class="lia-message-image-wrapper lia-message-image-actions-narrow lia-message-image-actions-below"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="nonDSRFlow1280.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/118161iD6804A5B84D64C19/image-size/large?v=v2&amp;px=999" role="button" title="nonDSRFlow1280.gif" alt="Non DSR enabled flow" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Non DSR enabled flow</span></span></SPAN><SPAN class="lia-inline-image-caption">Non DSR enabled flow</SPAN></SPAN></P> <P>In DSR enabled configurations only the outbound packets from the “client” to the server pass through the load balancing layer and are changed to convey the “real” address:port of the client to the server which in turn will use that on the return path to forward the packet directly to the “client” bypassing the Root Namespace steps and saving time along the way.</P> <P>The diagram below illustrates the flow of packets in a DSR enabled network.</P> <OL> <LI>Packet leaves POD1 addressed from 192.168.1.3:5555 to the load balancer address 10.0.0.10:53</LI> <LI>Packet enters VMswitch at Port 3 and rule is applied changing the destination address 192.168.1.4:53 which is POD2<BR />An example of an LBNAT rule is below</LI> </OL> <P>&nbsp;&nbsp;</P> <PRE>RULE : &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Friendly name : LB_DSR_E5B0F_10.231.111.97_13.0.0.13_8000_80_6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Priority : 100 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Flags : 1 terminating &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Type : lbnat &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Conditions: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Protocols : 6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Destination IP : 13.0.0.13 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Destination ports : 8000 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Flow TTL: 240 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Rule Data: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Decrementing TTL &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Fixing MAC &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Modifying destination IP &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Modifying destination port &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Creating a flow pair &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Map space : 82F1AFA2-1B42-4A38-81C5-B414B7541171 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Count of DIP Ranges: 2 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DIP Range(s) : &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; { 10.0.0.15 : 80 } &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; { 10.0.0.14 : 80 } &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FlagsEx : 0</PRE> <OL start="3"> <LI>The packet drops to the Forwarding layer in VFP where the MAC address is updated</LI> <LI>Forwarding rules look up the destination MAC address in the DSR cache</LI> <LI>Packet is forwarded to Port 4 of the VMswitch with the POD1 IP address as the source address and POD2 as the destination address</LI> <LI>Packet reaches the service in POD2 at 192.168.1.4:53</LI> <LI>On the return path the packet bypasses the Root Namespace and mux and is routed directly from Port 4 to Port 3 of the VMSwitch</LI> </OL> <P><SPAN class="lia-inline-image-display-wrapper lia-image-align-inline"><SPAN class="lia-message-image-wrapper lia-message-image-actions-narrow lia-message-image-actions-below"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DSRFlow1280.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/118163iABE4EC74B4134953/image-size/large?v=v2&amp;px=999" role="button" title="DSRFlow1280.gif" alt="DSR enabled flow" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">DSR enabled flow</span></span></SPAN><SPAN class="lia-inline-image-caption">DSR enabled flow</SPAN></SPAN></P> <H2 id="toc-hId--784652281">How to try it out</H2> <P>To enable DSR in Windows Container networking you need to know the feature is in preview and you will need to run Windows Server 19H1 or later including the latest insider builds.&nbsp;</P> <P>When starting kube-proxy provide the --enable-ds switch set to true and an additional –feature-gate WinDSR set to true.</P> <P>To do that you will have to use overlay networking as outlined in this<SPAN>&nbsp;</SPAN><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Introducing-Kubernetes-Overlay-Networking-for-Windows/ba-p/363082" target="_blank" rel="noopener">blog post</A><SPAN>&nbsp;</SPAN>by David Schott.</P> <P>Enabling DSR, follow the instructions<SPAN>&nbsp;</SPAN><A href="#" target="_blank" rel="noopener noopener noreferrer">here</A><SPAN>&nbsp;</SPAN>in the manual approach section with two &nbsp;changes:</P> <PRE>PS C:\k&gt; nssm install kube-proxy C:\k\kube-proxy.exe PS C:\k&gt; nssm set kube-proxy AppDirectory c:\k PS C:\k&gt; nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --feature-gates="WinOverlay=true,WinDSR=true” --hostname-override= --kubeconfig=c:\k\config --network-name=vxlan0 --source-vip= --enable-dsr=true --log-dir= --logtostderr=false PS C:\k&gt; nssm set kube-proxy DependOnService Kubelet PS C:\k&gt; nssm start kube-proxy </PRE> <H1 id="toc-hId-1154671559">Verify that DSR is working as expected</H1> <P>Verification involves a few steps. These steps can also help in general troubleshooting of the Windows networking components.</P> <UL> <LI>On the Windows work node download collectlogs.ps1 from here<SPAN>&nbsp;</SPAN><A href="#" target="_blank" rel="noopener noopener noreferrer">https://github.com/microsoft/SDN/tree/master/Kubernetes/windows/debug</A></LI> <LI>Run the script in an elevated PowerShell session</LI> <LI>This will generate a few output txt files.</LI> <LI>Open policy.txt.</LI> <LI>Below is part of a policy entry. Each policy will have a block like that.<BR />Check the IsDSR setting if it is set to TRUE</LI> </UL> <PRE>"ID":"2A34FF08-3F32-4C75-A6E9-F44D966324C1", "ILB":false, "InternalPort":6443, "IsDSR":true, "IsPolicy":true, "LocalRoutedVip":false, "Protocol":6, "SourceVIPMappingRe":1, "SourceVip":"", "State":3, "TCPPortAllocate":true, "Tag":"VFP ELB Policy", "UDPPortAllocate":false, "VIP_0":"10.96.0.1"</PRE> <P>If you see that on every policy, then DSR has been enabled successfully</P> <H1 id="toc-hId--1397485402">In Closing</H1> <P>Since DSR load balancing for Kubernetes on Windows is brand new, we’d love to hear any feedback in trying it out at&nbsp;<A href="#" target="_blank" rel="noopener noopener noreferrer">SIG-Windows</A>&nbsp;or in the comments below!</P> <P>&nbsp;</P> <P>Thanks for reading this far</P> <P>Mike Kostersitz</P> <P>&nbsp;</P> <P>*Special thanks to our Kalya Subramanian, Pradip Dhara, Madhan Raj Mookkandy, Buck Buckley and in our engineering team for designing and implementing DSR in overlay networking for Windows containers, as well as providing materials to help create content for this blog!</P> <P>&nbsp;</P> <P><FONT face="arial,helvetica,sans-serif" size="2">Editor's notes:</FONT></P> <P><FONT face="arial,helvetica,sans-serif" size="2">8/26/19: few fixes to content and typo corrections.</FONT></P> </DIV> </DIV> </DIV> Fri, 03 Apr 2020 20:33:50 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/direct-server-return-dsr-in-a-nutshell/ba-p/693710 mkostersitz 2020-04-03T20:33:50Z Synthetic Accelerations in a Nutshell – Windows Server 2019 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2019/ba-p/653976 <P>Hi folks,</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Dan Cuomo</A>&nbsp;here for our final installment in this blog series on synthetic accelerations covering Windows Server 2019.&nbsp; In Server 2019, we took learnings and expanded on the work that began in Server 2012 R2 with Dynamic VMQ and Server 2016 with VMMQ, to bring Dynamic VMMQ (d.VMMQ).</P> <P>&nbsp;</P> <P>The multi-release journey is designed to achieve one primary goal; improving your (and your tenant’s) networking experience in the Software Defined Data Center.&nbsp; This may come in the form of reducing CPU processing for network traffic and/or ensuring a smooth and consistent experience for the virtual machines on your host which ultimately means happy tenants running more virtual machines (and no midnight calls to troubleshoot the all-to-common “network slow-down”</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><FONT size="3"><STRONG>Public Service Announcement: </STRONG>Most of what you see below will not apply if you’re using an LBFO team.&nbsp; Microsoft recommends using Switch Embedded Teaming (SET) as the default teaming mechanism whenever possible, particularly when using Hyper-V.</FONT></P> </BLOCKQUOTE> <P>Before we get to the good stuff, here are the pointers to the previous blogs:</P> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012/ba-p/447792" target="_blank" rel="noopener">Synthetic Accelerations in a Nutshell – Windows Server 2012</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012-R2/ba-p/481428" target="_blank" rel="noopener">Synthetic Accelerations in a Nutshell – Windows Server 2012 R2</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2016/ba-p/535571" target="_blank" rel="noopener">Synthetic Accelerations in a Nutshell – Windows Server 2016</A></LI> </UL> <H2><FONT size="5">Dynamic VMMQ</FONT></H2> <P>As a quick refresher, Virtual Receive Side Scaling (on the host) creates an indirection table which enables packets to be processed by multiple, separate processors. &nbsp;The distribution of these packets to separate processors can be done in the OS, or offloaded to the NIC.&nbsp; While the indirection table is always established by the OS, we can offload the packet distribution to the NIC; when offloaded to the NIC, we call this VMMQ.</P> <P>&nbsp;</P> <P>Originally, we enabled the dynamic updating of the indirection table, called Dynamic VMQ, in Windows Server 2012 R2.&nbsp; However, in part due to the rearchitected design in Windows Server 2016 to bring VMMQ, Dynamic VMQ was not available in Windows Server 2016.</P> <P>&nbsp;</P> <P>Now in Windows Server 2019 we can dynamically remap VMMQ’s placement of packets onto different processors.&nbsp; We had three primary goals:</P> <UL> <LI>Optimize host efficiency</LI> <LI>Automatic tuning of the indirection table (so the VM can meet and maintain the desired throughput)</LI> <LI>Handling of bursty workloads</LI> </UL> <P>I’m starting to think those midnight network slow-downs may be a thing of the past!</P> <P>&nbsp;</P> <H3><FONT size="5">Optimizing Host Efficiency</FONT></H3> <P>When network&nbsp;throughput is low, Dynamic VMMQ enables the system to coalesce traffic received on a virtual NIC to as few CPUs as possible; we call this <STRONG>queue packing</STRONG> because we’re packing the queues onto as few CPU cores as is necessary to sustain the workload.&nbsp; Queue packing is more optimal for the host as the system would otherwise need to manage the distribution of packets across more CPUs; the more CPUs are engaged, the more the system must work to ensure all packets are properly handled.</P> <P>&nbsp;</P> <P>The picture below shows a virtual NIC receiving a low amount of network traffic.&nbsp; You can see we’re using the performance counter <STRONG>Hyper-V Virtual Switch Processor</STRONG> &gt; <STRONG>Packets from External/sec</STRONG> and there is one bar for each CPU core engaged.&nbsp; Only one CPU core (the green bar) is processing packets destined for a virtual NIC.&nbsp; The system has coalesced or packed all the queues onto one CPU core as was necessary to sustain the workload.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="1.png" style="width: 684px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/116362i6A102DBAF5104486/image-dimensions/684x338?v=v2" width="684" height="338" role="button" title="1.png" alt="1.png" /></span></P> <P>&nbsp;</P> <P>Here’s a video showing the Dynamic Coalescing.&nbsp; Note, the video is sped up to show the process occurring a bit quicker than normal.</P> <P>&nbsp;</P> <P><LI-VIDEO vid="https://youtu.be/CpUic2GaqSc" align="center" size="small" width="200" height="113" uploading="false" thumbnail="https://i.ytimg.com/vi/CpUic2GaqSc/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <H3><FONT size="5">Automatic Tuning of the System</FONT></H3> <P>After a hard day’s work, you head home for the day.&nbsp; Little did you know, your CIO is a night-owl and a few hours later begins working right as some backups begin on the file servers hosting the user profile.</P> <P>&nbsp;</P> <P>I think we all know the story that’s about to unfold.&nbsp; Your CIO calls in the support team after-hours because of the terrible performance.&nbsp; The following day, you’ll be asked to root cause what happened and develop an action plan to ensure the CIO never has this experience again.&nbsp;&nbsp;You think to yourself</P> <P>&nbsp;</P> <P style="text-align: center;">“<STRONG>this would be about the best place in the entire world to work, if it weren’t for all these complainers…</STRONG>” ;)</img></P> <P>&nbsp;</P> <P>One of the challenges with VMMQ in Windows Server 2016 (Static VMMQ) is that the indirection table – the assignment of a VMQ to be processed by a specific processor – cannot be updated once established.</P> <P>&nbsp;</P> <P>If another workload (for example VM B) starts receiving more traffic and one of its queues are mapped to the same processor as a queue from VM A, one of them may suffer.&nbsp; This is what happened to your CIO, the queues for the file server hosting his/her user profile was on the same processors as another workload performing backups.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG>Note</STRONG>: I’ve seen folks try to avoid this by preventing a NIC from using the same processors used by other NICs (overlapping).&nbsp; In practice, we’ve seen this provide very little value if any with SET teams.&nbsp; First, most people misconfigure this.&nbsp; Even if they have it configured correctly, you’re forced into constraining your adapters to using less processors.&nbsp; This only compounds the original problem.&nbsp; We do not recommend changing the default RSS Processor Array (which governs the indirection table creation) unless directed by Microsoft Support.</P> </BLOCKQUOTE> <P>With Windows Server 2019 and Dynamic VMMQ, we can now automatically move queues on an overburdened processor to other processors that aren’t doing as much work.&nbsp; Now workloads will have a more consistent and performant experience.</P> <P>&nbsp;</P> <P>In the following video, (sorry, no sound) we show a running network workload.&nbsp; Eventually we start a new process that competes and consumes for the CPU that is processing packets.&nbsp; In Windows Server 2016, the virtual machine would start receiving less packets affecting the throughput into the VM and your sleep patterns as your CIO calls you into the office to troubleshoot.</P> <P>&nbsp;</P> <P>However, in this video you can see that the system dynamically updates the indirection table and moves the processing of network traffic from CPU3 to an available processor (CPU1) when another workload starts consuming the CPU cycles.&nbsp; This allows the VM to continue receiving the same amount of traffic despite having a competing workload.</P> <P>&nbsp;</P> <P><LI-VIDEO vid="https://youtu.be/WsrVS3LCCNM" align="center" size="small" width="200" height="113" uploading="false" thumbnail="https://i.ytimg.com/vi/WsrVS3LCCNM/hqdefault.jpg" external="url"></LI-VIDEO></P> <P>&nbsp;</P> <H3><FONT size="5">Optimizing for bursty workloads</FONT></H3> <P>When a virtual NIC is idle, it doesn’t need any receive queues.&nbsp; However, if no queues are allocated (or perhaps only a bare minimum), and a burst of traffic comes in destined for that virtual NIC, it won’t be able to process all the data because we can’t just allocate queues all willy-nilly.&nbsp; Willy-nilly is bad...</P> <P>&nbsp;</P> <P>To ensure that we can meet an immediate burst of traffic, we pre-allocate queues for an idle workload.&nbsp; We call this <STRONG>queue parking</STRONG> (not to be confused with core parking).</P> <P>&nbsp;</P> <P>You can see the allocation of queues across a receive processor for a particular virtual NIC using the perfmon counter<STRONG> Hyper-V Virtual Network Adapter VRSS</STRONG> &gt; <STRONG>Instance (per virtual NIC)</STRONG> &gt; <STRONG>Receive Processor</STRONG>.&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="2.png" style="width: 944px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/116363i9DA4A0A4EC77B6C6/image-size/large?v=v2&amp;px=999" role="button" title="2.png" alt="2.png" /></span></P> <P>&nbsp;</P> <P>It’s important to note that there are always 16 entries shown and if you look closely, you’ll note that there are two bars of the same height.&nbsp; You can control how many receive queues per processor for all virtual NICs (although we recommend that you stick with the defaults) by modifying the MaxProcessors on the physical adapter.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="3.png" style="width: 817px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/116364i640E6BBD0826E5D7/image-size/large?v=v2&amp;px=999" role="button" title="3.png" alt="3.png" /></span></P> <P>&nbsp;</P> <P>The setting on the physical adapters cap the processors to be used by a virtual NIC.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 716px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/116365i9339490B76073ACA/image-size/large?v=v2&amp;px=999" role="button" title="4.png" alt="4.png" /></span></P> <P>&nbsp;</P> <P>If you only want to cap certain virtual NICs then instead of setting the value on the physical adapters, just set it on the virtual NIC using <STRONG>Set-VMNetworkAdapter -VRSSMaxQueuePairs &lt;value&gt;</STRONG></P> <P>&nbsp;</P> <P>Then review the updates to the vNIC as shown below.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 716px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/116366i2F87D5C9C5342637/image-size/large?v=v2&amp;px=999" role="button" title="5.png" alt="5.png" /></span></P> <P>&nbsp;</P> <H2><FONT size="5">Summary of Requirements</FONT></H2> <P>As you can see, the requirements to implement and manage the feature are greatly reduced.</P> <UL> <LI>Install latest drivers and firmware – Dynamic VMMQ is available on Premium Certified devices with non-inbox drivers.</LI> </UL> <UL> <LI><SPAN style="text-decoration: line-through;">Processor Array engaged by default – CPU0&nbsp;</SPAN><STRONG style="font-size: medium; font-family: inherit;">&nbsp;</STRONG><SPAN style="font-size: medium; font-family: inherit;">This was originally changed in 2012 R2 to enable VRSS (on the host) and you are no longer required to change the processor array (as is also the case in 2016).</SPAN></LI> </UL> <UL> <LI><SPAN style="text-decoration: line-through;">Configure the system to avoid CPU0 on non-hyperthreaded systems and CPU0 and CPU1 on hyperthreaded systems (e.g.&nbsp;BaseProcessorNumbershould be 1 or 2 depending on hyperthreading)</SPAN><SPAN style="font-size: medium; font-family: inherit;">&nbsp;While not explicitly required any longer as the dynamic algorithm will move workloads away from a burdened core, it would still be a best practice to do this in case of a driver bug.</SPAN></LI> </UL> <UL> <LI><SPAN style="text-decoration: line-through;">Configure the&nbsp;MaxProcessorNumber&nbsp;to establish that an adapter cannot use a processor higher than this</SPAN>.&nbsp;We recommend you let the system manage this now.</LI> </UL> <UL> <LI><SPAN style="text-decoration: line-through;">Configure&nbsp;MaxProcessors&nbsp;to establish how many processors out of the available list a NIC can spread VMQs across simultaneously</SPAN>&nbsp;&nbsp;This is unnecessary due to the enhancements in the default queue implemented in Windows Server 2016.&nbsp; You may still choose to do this if you’re limiting the queues as a rudimentary QoS mechanism as noted earlier.</LI> </UL> <UL> <LI>Test customer workload</LI> </UL> <H2>Summary of Advantages</H2> <UL> <LI><STRONG>All the benefits of VMMQ from Windows Server 2016</STRONG> (highlighted in the previous article)</LI> </UL> <UL> <LI><STRONG>Host efficiency is optimized</STRONG> – Through queue packing</LI> </UL> <UL> <LI><STRONG>Automatic tuning of the indirection table</STRONG> allows a VM to maintain stable throughput by reallocating queues to available processors</LI> </UL> <UL> <LI><STRONG>Handling of bursty workloads</STRONG> – Through queue parking</LI> </UL> <H2>Summary of Disadvantages</H2> <UL> <LI><STRONG>Requires a driver update and Premium certified device</STRONG></LI> </UL> <P>I hope you have enjoyed this series on synthetic accelerations and found it useful.&nbsp; As you can see, we’ve steadily worked towards reducing the setup complexity, improve the stability, and increase the performance for your virtualized workloads.&nbsp; Previously you had setup complicated adapters schemes, tune the system, avoid processors, and more…Now you simply install Windows and Hyper-V, test, and monitor.</P> <P>&nbsp;</P> <P>Please let us know in the comments if you have any questions!</P> <P>Thanks,<BR />Dan</P> Wed, 29 May 2019 13:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2019/ba-p/653976 Dan Cuomo 2019-05-29T13:00:00Z DPDK releases v19.05, introduces Windows Support! https://gorovian.000webhostapp.com/?exam=t5/networking-blog/dpdk-releases-v19-05-introduces-windows-support/ba-p/633927 <P><A href="#" target="_blank" rel="noopener"><STRONG>D</STRONG>ata <STRONG>P</STRONG>lane <STRONG>D</STRONG>evelopment <STRONG>K</STRONG>it</A>&nbsp;just announced the release of <A href="https://gorovian.000webhostapp.com/?exam=DPDK%20v19.05" target="_blank" rel="noopener">DPDK v19.05</A>.</P> <P>&nbsp;</P> <P><STRONG>We are thrilled to share that this release marks the introduction of Windows Support in the community-maintained upstream repository!</STRONG> This exciting development paves the way for more core libraries and networking hardware to be supported on Windows lighting up new use cases.</P> <P dir="ltr">&nbsp;</P> <P dir="ltr"><STRONG>D</STRONG>ata <STRONG>P</STRONG>lane <STRONG>D</STRONG>evelopment <STRONG>K</STRONG>it (DPDK) is a set of fast packet-processing libraries and drivers for user-mode applications looking to optimize network performance.</P> <P dir="ltr">&nbsp;</P> <P style="text-align: center;" dir="ltr"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DPDK logo.PNG" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/115677i3BB00EC0D238238F/image-size/medium?v=v2&amp;px=400" role="button" title="DPDK logo.PNG" alt="DPDK logo.PNG" /></span></P> <P dir="ltr">&nbsp;</P> <P dir="ltr">The Linux foundation hosted DPDK project is a vibrant, thriving community of developers from over 25 organizations spanning networking hardware vendors, independent software vendors, OS distros and consuming open source projects.</P> <P style="text-align: justify;">&nbsp;</P> <P style="text-align: justify;">For over a year now, we’ve had the ability to run DPDK on Windows through libraries available in the <A href="#" target="_blank" rel="noopener">DPDK Windows draft</A><SPAN> repository</SPAN>. However, this meant that the Windows port needed a separate development, build and testing pipeline, consequentially trailing behind the DPDK community project by multiple releases.</P> <P>&nbsp;</P> <P>With the initiation of the merge, DPDK libraries for Windows will benefit from the participation, contribution and leadership of the DPDK community. For instance, as part of this integration, DPDK libraries for Windows moved away from dependency on proprietary tool chain to using Clang-LLVM C compiler and Meson Build system.</P> <P>&nbsp;</P> <P><STRONG>What Next?</STRONG></P> <P>&nbsp;</P> <P>Wait, does this mean we can retire the DPDK Windows draft repository? Not quite, yet!</P> <P>&nbsp;</P> <P style="text-align: justify;">The draft repository will continue to be the development vehicle for all contributions, until we attain parity in features at the main repository. The integration of Windows Platform support has been initiated with the release of DPDK v19.05 and is expected to continue through 2019.</P> <P style="text-align: justify;">&nbsp;</P> <P style="text-align: justify;">Watch the <A href="#" target="_blank" rel="noopener">Roadmap</A> page for announcements on Core libraries, Poll Mode drivers and features that will be added in the subsequent releases. <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-2-Propelling/ba-p/339785" target="_blank" rel="noopener">As shared</A><SPAN> before</SPAN>, we are partnering with multiple networking vendors to expand the hardware ecosystem for DPDK on Windows.</P> <P>&nbsp;</P> <P>Eventually, when the integration is complete, DPDK on Windows can remain stable, up to date enjoying the quality baseline as other platforms.</P> <P>&nbsp;</P> <P><STRONG>Ways to Contribute</STRONG></P> <P>&nbsp;</P> <P>Interested in participating? Help us make DPDK on Windows more stable!</P> <P>&nbsp;</P> <P>Test the DPDK libraries on Windows and share your feedback! Head over to <A href="#" target="_blank" rel="noopener">the getting started guide</A>.</P> <P>&nbsp;</P> <P>But wait this is just a Hello world! Looking for more? Try the Windows port at the <A href="#" target="_self">DPDK-draft-Windows</A> repository with the v18.08 branch and <A href="#" target="_blank" rel="noopener">readme</A>.</P> <P>&nbsp;</P> <P>Do you have questions, feedback to share or want to report bugs? Do you have new use cases to support or want to make feature requests?</P> <P>&nbsp;</P> <P>Write to us by registering for the DPDK development mail list <A href="https://gorovian.000webhostapp.com/?exam=mailto:dev@dpdk.org" target="_blank" rel="noopener">dev@dpdk.org</A>. Contribute patches under these <A href="#" target="_blank" rel="noopener">guidelines</A>, reference <EM>“dpdk-draft-windows</EM>” in contribution.</P> <P>&nbsp;</P> <P>While we do our best to follow the forums used by the DPDK community, for quick direct access to the Microsoft Windows DPDK team, drop us an email at <A href="https://gorovian.000webhostapp.com/?exam=mailto:dpdkwin@microsoft.com" target="_blank" rel="noopener">dpdkwin@microsoft.com</A>.</P> <P>&nbsp;</P> <P>Join us in the DPDK Windows Community call, under the guidance of the DPDK Technical Board to help shape the future of DPDK on Windows!</P> <P>&nbsp;</P> <P>Thanks to the contributions from our partners at Intel and the DPDK Technical board for the guidance and the leadership.</P> <P>&nbsp;</P> <P>Looking forward to hearing from you, Thanks for reading!</P> <P>&nbsp;</P> <P><SPAN style="font-family: inherit;">Harini Ramakrishnan</SPAN></P> <P>&nbsp;</P> Wed, 22 May 2019 18:44:25 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/dpdk-releases-v19-05-introduces-windows-support/ba-p/633927 Harini Ramakrishnan 2019-05-22T18:44:25Z Troubleshooting Kubernetes Networking on Windows: Part 1 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshooting-kubernetes-networking-on-windows-part-1/ba-p/508648 <P>We’ve all been there: Sometimes things just don’t work the way they should even though we followed everything down to a T.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="giphy" style="width: 500px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111261iDDC9194EAE0831DD/image-size/large?v=v2&amp;px=999" role="button" title="giphy" alt="giphy" /></span></P> <P>&nbsp;</P> <P>Kubernetes in particular, is not easy to troubleshoot – even if you’re an expert. There&nbsp;are&nbsp;multiple&nbsp;components involved&nbsp;in the creation/deletion of containers&nbsp;that must all harmoniously interoperate&nbsp;end-to-end. For example:</P> <UL> <LI>Inbox&nbsp;platform&nbsp;services&nbsp;(e.g.&nbsp;WinNAT, HNS/HCS, VFP)</LI> <LI>Container&nbsp;runtimes&nbsp;&amp;&nbsp;Go-wrappers (e.g.&nbsp;Dockershim,&nbsp;ContainerD,&nbsp;hcsshim)&nbsp;</LI> <LI>Container orchestrator processes&nbsp;(e.g. kube-proxy,&nbsp;kubelet)&nbsp;</LI> <LI>CNI&nbsp;network&nbsp;plugins&nbsp;(e.g.&nbsp;win-bridge, win-overlay, azure-cni)&nbsp;</LI> <LI>IPAM plugins (e.g. host-local)</LI> <LI>Any other&nbsp;host-agent processes/daemons&nbsp;(e.g. FlannelD, Calico-Felix, etc.)&nbsp;</LI> <LI>… (more to come!)</LI> </UL> <P>&nbsp;</P> <P>This, in turn, also means that the potential problem space to investigate can grow overwhelmingly large when things <STRONG>do</STRONG> end up breaking. We often hear the phrase:&nbsp;<EM>“I don’t even know where to begin.”</EM></P> <P>&nbsp;</P> <P>The intent of this blog post is to educate the reader on the available tools and resources that can help unpeel the first few layers of the onion; it is not intended to be a fully exhaustive guide to root-cause every possible bug for every possible configuration. However, by the end one should at least be able to narrow down on an observable symptom through a pipeline of analytical troubleshooting steps and come out with a better understanding of what the underlying issue could be.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P>NOTE: Most of the content in this blog is directly taken from the amazing Kubecon Shanghai ’18 video <A href="#" target="_blank" rel="noopener">“Understanding Windows Container Networking in Kubernetes Using a Real Story”</A> <SPAN>&nbsp;by Cindy Xing (Huawei) and Dinesh Kumar Govindasamy (Microsoft)</SPAN>.</P> </BLOCKQUOTE> <P>&nbsp;</P> <P><FONT size="6">Table of Contents</FONT></P> <OL> <LI><A href="https://gorovian.000webhostapp.com/?exam=#ensure-k8s-installed" target="_self">Ensure Kubernetes is installed and running correctly</A></LI> <LI><A style="font-family: inherit; background-color: #ffffff;" href="https://gorovian.000webhostapp.com/?exam=#use-validate-script" target="_self">Use a script to validate basic cluster connectivity</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#query-event-logs" target="_self">Query the built-in Kubernetes event logs</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#analyze-kube-logs" target="_self">Analyze kubelet, kube-proxy logs</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#inspect-cni-config" target="_self">Inspect CNI network plugin configuration</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#verify-hns-networking" target="_self">Verify HNS networking state</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#analyze-collectlogs" target="_self">Take a snapshot of your network using CollectLogs.ps1</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=#capture-packets" target="_self">Capture packets and analyze network flows</A></LI> </OL> <H2>&nbsp;</H2> <H2><A id="ensure-k8s-installed" target="_blank"></A>Step 1: Ensure Kubernetes is installed and running correc<FONT size="5">tly</FONT></H2> <P>As mentioned in the introduction, there are <EM>a lot</EM> of different platform and open-source actors that are needed to operate a Kubernetes cluster. It can be hard to keep track of all of them - especially given that they release at a different cadence.</P> <P>&nbsp;</P> <P>One quick sanity-check that can be done without any external help is to employ a <A href="#" target="_blank" rel="noopener">validation script</A> that verifies supported bits are installed:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="verify_installation.png" style="width: 878px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111265i47A0D04EB31F2765/image-size/large?v=v2&amp;px=999" role="button" title="verify_installation.png" alt="Verifying Kubernetes is installed" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Verifying Kubernetes is installed</span></span></P> <P>&nbsp;</P> <P>While trivial, another step that can be equally easily overlooked is ensuring that all the components are indeed running. Any piece of software can crash or enter a deadlock-like state, including host-agent processes such as kubelet.exe or kube-proxy.exe. This can result in unexpected cluster behavior and detached node/container states, but thankfully it’s easy to check. Running a simple ps command usually suffices:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ps.png" style="width: 630px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111264i205A93C038A8C849/image-size/large?v=v2&amp;px=999" role="button" title="ps.png" alt="Typical Kubernetes processes running" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Typical Kubernetes processes running</span></span></P> <P>Unfortunately, the above command won’t capture that the processes themselves could be stuck waiting in a deadlock-like state; we will cover this case in step 4.</P> <P>&nbsp;</P> <H2><A id="use-validate-script" target="_blank"></A>Step 2: Use a script to validate basic cluster connectivity</H2> <P>Before diving head-first into analyzing HNS resources and verbose logs, there is a handy Pester test suite which allows you to validate basic connectivity scenarios and report on success/failure <A href="#" target="_blank" rel="noopener">here</A>. The only pre-requisite in order to run it is that you are using Windows Server 2019 &nbsp;(requires minor fix-up otherwise) and that you have more than one node for the remote pod test:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="K8s_pester_tests.PNG" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111262i84532A697216F4CD/image-size/large?v=v2&amp;px=999" role="button" title="K8s_pester_tests.PNG" alt="Snippet from Kubernetes Connectivity Test Suite" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Snippet from Kubernetes Connectivity Test Suite</span></span></P> <P>The intent of running this script is to have a quick glance of overall networking health, as well as hopefully accelerate subsequent steps by knowing what to look for.</P> <P>&nbsp;</P> <H2><A id="query-event-logs" target="_blank"></A>Step 3: Query the built-in Kubernetes event logs</H2> <P>After verifying that all the processes are running as expected, the next step is to query the built-in Kubernetes event logs and see what the basic built-in health-checks that ship with K8s have to say:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="kubectl_get_pods.png" style="width: 926px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111267i35BD29DEAF55E0AC/image-size/large?v=v2&amp;px=999" role="button" title="kubectl_get_pods.png" alt="Kubernetes pods that are stuck in &quot;ContainerCreating&quot; state" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Kubernetes pods that are stuck in "ContainerCreating" state</span></span></P> <P>&nbsp;</P> <P>More Information about misbehaving Kubernetes resources such as event logs can be viewed using the “kubectl describe” command. For example, one frequent misconfiguration on Windows is having a misconfigured “pause” container with a kernel version that doesn’t match the host OS:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="pause_container.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111269i2709E6346C731D5B/image-size/large?v=v2&amp;px=999" role="button" title="pause_container.gif" alt="Shared pause container vNIC in a pod" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Shared pause container vNIC in a pod</span></span></P> <P>&nbsp;</P> <P>Here are the corresponding event logs from kubectl describe output, where we accidentally built our “kubeletwin/pause” image on top of a Windows Server, version 1803 container image and ran it on a Windows Server 2019 host:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="eventlogs - highlighted.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112988i10DB5A7D8FDE51A6/image-size/large?v=v2&amp;px=999" role="button" title="eventlogs - highlighted.png" alt="Erroneous Kubernetes event logs" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Erroneous Kubernetes event logs</span></span></P> <P><EM>(On a side note, this specific example issue can be avoided altogether if one references the&nbsp; multi-arch pause container image mcr.microsoft.com/k8s/core/pause:1.0.0 which will run on both Windows Server, version 1803 and Windows Server 2019).</EM></P> <P>&nbsp;</P> <H2><A id="analyze-kube-logs" target="_blank"></A>Step 4: Analyze kubelet, kube-proxy logs</H2> <P>Another useful source of information that can be leveraged to perform root-cause analysis for failing container creations is the kubelet, FlannelD, and kube-proxy logs.</P> <P>&nbsp;</P> <P>These components all have different responsibilities. Here is a very brief summary of what they do which should give you a rough idea on what to watch out for:</P> <P>&nbsp;</P> <TABLE style="width: 960px;"> <TBODY> <TR> <TD style="width: 100px;"> <P><STRONG>Component</STRONG></P> </TD> <TD style="width: 578.889px;"> <P><STRONG>Responsibility</STRONG></P> </TD> <TD style="width: 40px;"> <P><STRONG>When to inspect?</STRONG></P> </TD> </TR> <TR> <TD style="width: 100px;"> <P>Kubelet</P> </TD> <TD style="width: 578.889px;"> <P>Interacts with container runtime (e.g. Dockershim) to bring up containers and pods.</P> </TD> <TD style="width: 40px;"> <P>Erroneous pod creations/configurations</P> </TD> </TR> <TR> <TD style="width: 100px;"> <P>Kube-proxy</P> </TD> <TD style="width: 578.889px;"> <P>Manages network connectivity for containers (programming policies used for NAT’ing or load balancing).</P> </TD> <TD style="width: 40px;"> <P>Mysterious network glitches, in particular for service discovery and communication</P> </TD> </TR> <TR> <TD style="width: 100px;"> <P>FlannelD</P> </TD> <TD style="width: 578.889px;"> <P>Responsible for keeping all the nodes in sync with the rest of the cluster for events such as node removal/addition. This consists of assigning IP blocks (pod subnets) to nodes as well as plumbing routes for inter-node connectivity.</P> </TD> <TD style="width: 40px;"> <P>Failing inter-node connectivity</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>Log files for all of these components can be found in different locations; by default a log dump for kubelet and kube-proxy is generated in the C:\k directory, though some users opt to log to a different directory.</P> <P>&nbsp;</P> <P>If the logs appear to not have updated in a longer time, then perhaps the process is stuck, and a simple restart or sending the right signal can kick things back into place.</P> <P>&nbsp;</P> <H2><A id="inspect-cni-config" target="_blank"></A>Step 5: Inspect CNI network plugin configuration</H2> <P>Another common source of problems that can cause containers to fail to start with errors such as “FailedCreatePodSandbox” is having a misconfigured CNI plugin and/or config. This usually occurs whenever there are bugs or typos in the deployment scripts that are used to configure nodes:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="failedpodcreatesandbox.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112990iDAD9903605C126E8/image-size/large?v=v2&amp;px=999" role="button" title="failedpodcreatesandbox.png" alt="Example error due to misconfigured CNI config" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Example error due to misconfigured CNI config</span></span></P> <P>Thankfully, the network configuration that is passed to CNI plugins in order to plumb networking into containers is a very simple static file that is easy to access. On Windows, this configuration file is stored under the “C:\k\cni\config\” directory. On Linux, a similar file exists in “/etc/cni/net.d/”.</P> <P>&nbsp;</P> <P>Here is the corresponding typo that caused pods to fail to start due to degraded networking state:</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="invalid_cni_config_highlighted.png" style="width: 437px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111273i9AA1CA00DDE7B303/image-size/large?v=v2&amp;px=999" role="button" title="invalid_cni_config_highlighted.png" alt="Highlighted Typo in CNI Config" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Highlighted Typo in CNI Config</span></span></P> <P>Whenever there are failing pod creations or unexpected network plumbing, we should always inspect the CNI config file for typos and consult the CNI plugins documentation for more details on what is expected. Here are the docs for the Windows plugins:</P> <UL> <LI><A href="#" target="_blank" rel="noopener">win-bridge</A></LI> <LI><A href="#" target="_blank" rel="noopener">win-overlay</A></LI> <LI><A href="#" target="_blank" rel="noopener">flannel</A></LI> </UL> <P>&nbsp;</P> <H2><A id="verify-hns-networking" target="_blank"></A>Step 6: Verify HNS networking state</H2> <P>Having exhaustively examined Kubernetes-specific event logs and configuration files previously, the next step usually consists of collecting network information programmed on the networking stack (control plane and data plane) used by containers. All of the information can be collected conveniently by the “<A href="#" target="_blank" rel="noopener">CollectLogs.ps1</A>” script, which will be done in step 7.</P> <P>&nbsp;</P> <P>Before reviewing the contents of the “CollectLogs.ps1” tool, the Windows container networking architecture needs to be understood at a high-level.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="k8s-windows.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111274i2C8415404A94B48F/image-size/large?v=v2&amp;px=999" role="button" title="k8s-windows.gif" alt="Windows container networking Overview" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Windows container networking Overview</span></span></P> <P>&nbsp;</P> <TABLE style="width: 960px;"> <TBODY> <TR> <TD width="208"> <P><STRONG>Windows Component</STRONG></P> </TD> <TD width="208"> <P><STRONG>Responsibilities</STRONG></P> </TD> <TD width="208"> <P><STRONG>Linux Counterpart</STRONG></P> </TD> </TR> <TR> <TD width="208"> <P>Network Compartment</P> </TD> <TD width="208"> <UL> <LI>Logical separation in the TCP/IP stack.</LI> <LI>Packet forwarding between compartments is prevented (by default).</LI> <LI>All IP objects (addresses, routes, etc.) stay unique to the compartment.</LI> </UL> </TD> <TD width="208"> <P>Network namespace</P> </TD> </TR> <TR> <TD width="208"> <P>vSwitch and HNS networks</P> </TD> <TD width="208"> <UL> <LI>Provides L2 switching and L3 functionality.</LI> <LI>Each vSwitch has its own forwarding table and forwards packets based on MAC address or VLAN tag.</LI> <LI>Dynamically add/remove switch ports.</LI> <LI>1 (external) vSwitch / NIC.</LI> <LI>1 vSwitch / HNS network.</LI> </UL> </TD> <TD width="208"> <P>Bridge and IP routing</P> </TD> </TR> <TR> <TD width="208"> <P>vNICs, HNS endpoints, and vSwitch ports</P> </TD> <TD width="208"> <UL> <LI>Container NICs (vNICs) are bound to a corresponding port in the vSwitch.</LI> <LI>Endpoints are a HNS abstraction for a container vNIC.</LI> </UL> </TD> <TD width="208"> <P>IP Links and virtual network interfaces</P> </TD> </TR> <TR> <TD width="208"> <P>HNS policies, VFP rules, Firewall</P> </TD> <TD width="208"> <UL> <LI>VFP is the programmable, match-action based filtering engine.</LI> <LI>Applies rules to incoming outgoing packets from vPort.</LI> <LI>VFP rules are different for each vPort.</LI> </UL> </TD> <TD width="208"> <P>iptables</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>One particular component to highlight is VFP (Virtual Filtering Platform), which is a vSwitch extension containing most of the decision logic used to route packets correctly from source to destination by defining operations to be performed on packets such as:</P> <UL> <LI>Encapsulating/Decapsulating packets</LI> <LI>Load balancing packets</LI> <LI>Network Address Translation</LI> <LI>Network ACLs</LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="VFP_Overview.PNG" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111276i4F289B701335A572/image-size/large?v=v2&amp;px=999" role="button" title="VFP_Overview.PNG" alt="Overview of Virtual Filtering Platform (VFP)" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Overview of Virtual Filtering Platform (VFP)</span></span></P> <P>To read up more on this topic, many more details on VFP can be found <A href="#" target="_blank" rel="noopener">here</A>.</P> <P>&nbsp;</P> <P>Our first starting point should be to check that all the HNS resources indeed exist. Here is an example screenshot that shows the HNS networking state for a cluster with kube-DNS service (10.96.0.10) and a sample Windows web service (10.104.193.123) backed by 2 endpoint DIPs ("768b4bd1-774c-47e8-904f-91c007a4b183", "048cd973-b5db-45a6-9c65-16dec22e871d"):</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="hnsdiag_list_all - Copy.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111277iA9F5E56BDED4DBE8/image-size/large?v=v2&amp;px=999" role="button" title="hnsdiag_list_all - Copy.png" alt="Summary of typical Host Networking Service (HNS) objects" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Summary of typical Host Networking Service (HNS) objects</span></span></P> <P>We can take a closer look at the network object representing a given endpoint DIP using Get-HNS* cmdlets (this even works for remote endpoints!)</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="get-hnsendpoint.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111278i413EB669C15C12BE/image-size/large?v=v2&amp;px=999" role="button" title="get-hnsendpoint.png" alt="Typical HNS endpoint object" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Typical HNS endpoint object</span></span></P> <P>The information listed here (DNSSuffix, IPAddress, Type, VirtualNetworkName, and Policies should match what was passed in through the CNI config file.</P> <P>&nbsp;</P> <P>Digging deeper, to view VFP rules we can use the inbox “vfpctrl” cmdlet. For example, to view the layers of the endpoint:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="vfp_layers.PNG" style="width: 795px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111279i3A50451F50F46FAA/image-size/large?v=v2&amp;px=999" role="button" title="vfp_layers.PNG" alt="Listing VFP layers for a given container vPort" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Listing VFP layers for a given container vPort</span></span></P> <P>Similarly, to print the rules belonging to a specific layer (e.g. SLB_NAT_LAYER) that each packet goes through:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="vfp_rules1 - Copy.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111280i9893835D6F161611/image-size/large?v=v2&amp;px=999" role="button" title="vfp_rules1 - Copy.png" alt="Snippet of VFP rules for the SLB_NAT_LAYER of a container vPort" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Snippet of VFP rules for the SLB_NAT_LAYER of a container vPort</span></span></P> <P>The information programmed into VFP should match with what was specified in the CNI config file and HNS Policies.</P> <P>&nbsp;</P> <H2><A id="analyze-collectlogs" target="_blank"></A>Step 7: Analyze snapshot of network using CollectLogs.ps1</H2> <P>Now that we familiarized ourselves with the state of the network and its basic constituents let’s take a look at some common symptoms and correlate it against the likely locations where the culprit may be.</P> <P>&nbsp;</P> <P>Our tool of choice to take a snapshot of our network is <A href="#" target="_self">CollectLogs.ps1</A>. It collects the following information (amongst a few other things):</P> <TABLE style="width: 960px;"> <TBODY> <TR> <TD width="312"> <P><STRONG>File</STRONG></P> </TD> <TD width="312"> <P><STRONG>Contains</STRONG></P> </TD> </TR> <TR> <TD width="312"> <P>endpoint.txt</P> </TD> <TD width="312"> <P>Endpoint information and HNS Policies applied to endpoints.</P> </TD> </TR> <TR> <TD width="312"> <P>ip.txt</P> </TD> <TD width="312"> <P>All NICs in all network compartments (and which)</P> </TD> </TR> <TR> <TD width="312"> <P>network.txt</P> </TD> <TD width="312"> <P>Information about HNS networks</P> </TD> </TR> <TR> <TD width="312"> <P>policy.txt</P> </TD> <TD width="312"> <P>Information about HNS policies (e.g. VIP - &gt; DIP Load Balancers)</P> </TD> </TR> <TR> <TD width="312"> <P>ports.txt</P> </TD> <TD width="312"> <P>Information about vSwitch (ports)</P> </TD> </TR> <TR> <TD width="312"> <P>routes.txt</P> </TD> <TD width="312"> <P>Route tables</P> </TD> </TR> <TR> <TD width="312"> <P>hnsdiag.txt</P> </TD> <TD width="312"> <P>Summary of all HNS resources</P> </TD> </TR> <TR> <TD width="312"> <P>vfpOutput.txt</P> </TD> <TD width="312"> <P>Verbose dump of all the VFP ports used by containers listing all layers and associated rules</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <H3><FONT size="5">Example #1: Inter-node communication Issues</FONT></H3> <P>&nbsp;</P> <H4>L2bridge / Flannel (host-gw)</H4> <P>When dealing with inter-node communication issues such as pod-to-pod connectivity across hosts, it is important to check static routes are programmed. This can be achieved by inspecting the routes.txt or using Get-NetRoute:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="l2bridgeRoutes - internode.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111281iDF2559D557EEC4DE/image-size/large?v=v2&amp;px=999" role="button" title="l2bridgeRoutes - internode.png" alt="Get-NetRoute output highlighting static routes for pod networks" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Get-NetRoute output highlighting static routes for pod networks</span></span></P> <P>There should be routes programmed for each pod subnet (e.g. 10.244.18.0/24) =&gt; container host IP (e.g. 10.127.130.35).</P> <P>&nbsp;</P> <P>When using Flannel, users can also consult the FlannelD output to watch for the appropriate events for adding the pod subnets after launch:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="flannel_subnets.PNG" style="width: 752px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111283iBF77A61E18E94618/image-size/large?v=v2&amp;px=999" role="button" title="flannel_subnets.PNG" alt="Flannel &quot;subnet lease&quot; events" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Flannel "subnet lease" events</span></span></P> <H4><FONT size="4">Overlay (Flannel vxlan)<BR /></FONT></H4> <P>In overlay, inter-node connectivity is implemented using "REMOTESUBNETROUTE" rules in VFP. Instead of checking static routes, we can reference "REMOTESUBNETROUTE" rules directly in vfpoutput.txt, where each pod subnet (e.g. 10.244.2.0/24) assigned to a node should have its corresponding destination IP (e.g. 10.127.130.38) specified as the destination in the outer packet:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="remotesubnet - highlighted.PNG" style="width: 869px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111286iA9CD54FE2E1F019A/image-size/large?v=v2&amp;px=999" role="button" title="remotesubnet - highlighted.PNG" alt="VFP RemoteSubnetRoute rules used for VXLAN packet encapsulation" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">VFP RemoteSubnetRoute rules used for VXLAN packet encapsulation</span></span></P> <P>For additional details on inter-node container to container connectivity in overlay, please take a look at <A href="#" target="_blank" rel="noopener">this video</A>.</P> <P>&nbsp;</P> <H4><FONT size="4">When can I encounter this issue?</FONT></H4> <P>One common configuration problem that manifests in this symptom is having mismatched networking configuration on Linux/Windows.</P> <P>&nbsp;</P> <P>To double-check the network configuration on Linux, users can consult the CNI config file stored in /etc/cni/net.d/. In the case of Flannel on Linux, this file can also be embedded into the container, so users may need to exec into the Flannel pod itself to access it:</P> <P>&nbsp;</P> <PRE>kubectl exec -n kube-system kube-flannel-ds-amd64-&lt;someid&gt; cat /etc/kube-flannel/net-conf.json kubectl exec -n kube-system kube-flannel-ds-amd64-&lt;someid&gt; cat /etc/kube-flannel/cni-conf.json</PRE> <P>&nbsp;</P> <H3><FONT size="5">Example #2: Containers cannot reach the outside world</FONT></H3> <P>Whenever outbound connectivity does not work, one of the first starting points is to ensure that there exists a NIC in the container. For this, we can consult the “ip.txt” output and compare it with the output of an “docker exec &lt;id&gt; ipconfig /all” in the problematic (running) container itself:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ipconfig.PNG" style="width: 776px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111285i15070A4881B4280D/image-size/large?v=v2&amp;px=999" role="button" title="ipconfig.PNG" alt="Reference Container NIC in network compartment 2" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Reference Container NIC in network compartment 2</span></span></P> <P>In l2bridge networking (used by Flannel host-gw backend), the container gateway should be set to the 2. address exclusively reserved for the bridge endpoint (cbr0_ep) in the same pod subnet.</P> <P>&nbsp;</P> <P>In overlay networking (used by Flannel vxlan backend), the container gateway should be set to the .1 address exclusively reserved for the DR (distributed router) vNIC in the same pod subnet.</P> <P>&nbsp;</P> <H4><FONT size="4">L2bridge</FONT></H4> <P>Going outside of the container, on l2bridge one should also verify that the route tables on the node itself are setup correctly for the bridge endpoint. Here is a sample with the relevant entries containing quad-zero routes for a node with pod subnet 10.244.19.0/24:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="l2bridgeRoutes - cbr0ep.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111287i2FA49F4E43D1D2F4/image-size/large?v=v2&amp;px=999" role="button" title="l2bridgeRoutes - cbr0ep.png" alt="Quad-zero static routes for a node with pod subnet 10.244.19.0/24" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Quad-zero static routes for a node with pod subnet 10.244.19.0/24</span></span></P> <P>The next thing to check on l2bridge is verify that the OutboundNAT policy and the ExceptionList is programmed correctly. For a given endpoint (e.g. 10.244.4.7) we should verify in the endpoint.txt that there exists an OutboundNAT HNS Policy and that the ExceptionList matches with what we entered into the deployment scripts originally:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="endpoint-highlighted.png" style="width: 700px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111288i5562210974D8BE26/image-size/large?v=v2&amp;px=999" role="button" title="endpoint-highlighted.png" alt="HNS Policies for a typical l2bridge endpoint" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">HNS Policies for a typical l2bridge endpoint</span></span></P> <P>Finally, we can also consult the vfpOutput.txt to verify that the L2Rewrite rule exists so that the container MAC is rewritten to the host’s MAC as specified in the <A href="#" target="_blank" rel="noopener">l2bridge container networking docs</A>.</P> <P>&nbsp;</P> <P>In the EXTERNAL_L2_REWRITE layer, there should be a rule which matches the container’s source MAC (e.g. "00-15-5D-AA-87-B8") and rewrites it to match the host’s MAC address (e.g. "00-15-5D-05-C3-0C"):</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="l2bridge_rule.png" style="width: 462px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111289iC24422731726779A/image-size/large?v=v2&amp;px=999" role="button" title="l2bridge_rule.png" alt="Reference EXTERNAL_L2_REWRITE_LAYER with rules transposing a container MAC to a host MAC" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Reference EXTERNAL_L2_REWRITE_LAYER with rules transposing a container MAC to a host MAC</span></span></P> <H4>Overlay</H4> <P>For overlay, we can check whether there exists an ENCAP rule that encapsulates outgoing packets correctly with the hosts IP. For example, for a given pod subnet (10.244.3.0/24) with host IP 10.127.130.36:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="encap_overlay_highlighted.png" style="width: 471px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111291i62161B345775A6EE/image-size/large?v=v2&amp;px=999" role="button" title="encap_overlay_highlighted.png" alt="Reference encapsulation rule used by overlay container networks" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Reference encapsulation rule used by overlay container networks</span></span></P> <H4>&nbsp;</H4> <H4>When can I encounter this issue?</H4> <P>One example configuration error for Flannel (vxlan) overlay that may results in failing east/west connectivity is failing to <A href="#" target="_self">delete the old SourceVIP.json</A> file whenever the same node is deleted and re-joined to a cluster.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P>NOTE: When deploying L2bridge networks on Azure, user’s also need to configure <A href="#" target="_blank" rel="noopener">user-defined routes</A> for each pod subnet allocated to a node for networking to work. Some users opt to use overlay in public cloud environments for that reason instead, where this step isn’t needed.</P> </BLOCKQUOTE> <P>&nbsp;</P> <H3><FONT size="5">Example #3: Services / Load-balancing does not work</FONT></H3> <P>Let’s say we have created a Kubernetes service called “win-webserver” with VIP 10.102.220.146:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="services_example.PNG" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111292iE056F0991730E5A2/image-size/large?v=v2&amp;px=999" role="button" title="services_example.PNG" alt="Example Kubernetes service on Windows" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Example Kubernetes service on Windows</span></span></P> <P>Load Balancing is usually performed directly on the node itself by replacing the destination VIP (Service IP) with a specified DIP (pod IP). HNS Loadbalancers can be viewed using the “hnsdiag” cmdlet:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="loadbalancing_annotated.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111293i7B6DB2469ED96695/image-size/large?v=v2&amp;px=999" role="button" title="loadbalancing_annotated.png" alt="Typical HNS Loadbalancer objects on Windows" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Typical HNS Loadbalancer objects on Windows</span></span></P> <P>For a more verbose output, users can also inspect policy.txt to check for “ELB” policies (LoadBalancers) for additional information:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="loadbalancers_detailed.PNG" style="width: 620px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111295i9899B4FC0287B0F9/image-size/large?v=v2&amp;px=999" role="button" title="loadbalancers_detailed.PNG" alt="Example HNS LoadBalancer configuration in policy.txt" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Example HNS LoadBalancer configuration in policy.txt</span></span></P> <P>The next step usually consists of verifying that the endpoints (e.g. "4d1d1b8c-c12d-461a-a608-11825b6a9189") still exist in endpoint.txt and are reachable by IP from the same source:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dip.PNG" style="width: 711px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111296i446AC7CDAF4A2DCA/image-size/large?v=v2&amp;px=999" role="button" title="dip.PNG" alt="Example pod DIP endpoint referenced by a HNS Loadbalancer" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Example pod DIP endpoint referenced by a HNS Loadbalancer</span></span></P> <P>Finally, we can also check whether the VFP "lbnat" rules exist in the "LB" layer for our service IP 10.102.220.146 (with NodePort 31486):</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lbnat-highlighted.png" style="width: 574px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111297i25625ED0CA900FE1/image-size/large?v=v2&amp;px=999" role="button" title="lbnat-highlighted.png" alt="Reference VFP rules used for load-balancing containers" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Reference VFP rules used for load-balancing containers</span></span></P> <H4>&nbsp;</H4> <H4>When can I encounter this issue?</H4> <P>One possible issue that can cause erroneous load balancing is a misconfigured kube-proxy which is responsible for programming these policies. For example, one may <A href="#" target="_self">fail to pass in the --hostname-override parameter</A>, causing endpoints from the local host to be deleted.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P>NOTE that service VIP resolution from the Windows node itself is <A href="#" target="_blank" rel="noopener">not supported</A> on Windows Server 2019, but planned for Windows Server, version 1903.</P> </BLOCKQUOTE> <H3><FONT size="5">Example #4: DNS resolution is not working from within the container</FONT></H3> <P>For this example, let’s assume that the kube-DNS cluster addon is configured with service IP 10.96.0.10.</P> <P>Failing DNS resolution is often a symptom of one of the previous examples. For example, (external) DNS resolution would fail if outbound connectivity isn’t present or resolution could also fail if we cannot reach the kube-DNS service.</P> <P>Thus, the first troubleshooting step should be to analyze whether the kube-DNS service (e.g. 10.96.0.10) is programmed as a HNS LoadBalancer correctly on the problematic node:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DNS_loadbalancer.png" style="width: 616px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111298i3953773018E7F565/image-size/large?v=v2&amp;px=999" role="button" title="DNS_loadbalancer.png" alt="HNS LoadBalancer object representing kube-DNS service" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">HNS LoadBalancer object representing kube-DNS service</span></span></P> <P>Next, we should also check whether the DNS information is set correctly in the ip.txt entry for the container NIC itself:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dns-info-highlighted.png" style="width: 786px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111299iE86B172A18571034/image-size/large?v=v2&amp;px=999" role="button" title="dns-info-highlighted.png" alt="Reference DNS configuration in a Windows pod" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Reference DNS configuration in a Windows pod</span></span></P> <P>We should also check whether it’s possible to reach the kube-DNS pods directly and whether that works. This may indicate that there is some problem in resolving the DNS service VIP itself. For example, assuming that one of the DNS pods has IP 10.244.0.3:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dns-dip.PNG" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111300i45EBCBB9C5463443/image-size/large?v=v2&amp;px=999" role="button" title="dns-dip.PNG" alt="Sending a DNS request directly to the DNS pod endpoint" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Sending a DNS request directly to the DNS pod endpoint</span></span></P> <H4>&nbsp;</H4> <H4>When can I encounter this issue?</H4> <P>One possible misconfiguration that results in DNS resolution problems is an incorrect DNS suffix or DNS service IP which was specified in the CNI config <A href="#" target="_blank" rel="noopener">here</A> and <A href="#" target="_blank" rel="noopener">here</A>.</P> <P>&nbsp;</P> <H2><A id="capture-packets" target="_blank"></A>Step 8: Capture packets and analyze flows</H2> <P>The last step requires in-depth knowledge of the operations that packets from containers undergo and the network flow. As such, it is also the most time-consuming to perform and will vary depending on the observed issue. At a high level, it consists of:</P> <OL> <LI>Running <A href="#" target="_blank" rel="noopener">startpacketcapture</A> to start the trace</LI> <LI>Reproducing the issue – e.g. sending packets from source to destination</LI> <LI>Running <A href="#" target="_blank" rel="noopener">stoppacketcapture</A> to stop the trace</LI> <LI>Analyzing correct processing by the data path at each step</LI> </OL> <P>&nbsp;</P> <P>Here are a few example animations that showcase some common container network traffic flows:</P> <H3>Pod to Pod:</H3> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="pod2pod.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111301i67449EE494FF87CE/image-size/large?v=v2&amp;px=999" role="button" title="pod2pod.gif" alt="Animated visualization showing pod to pod connectivity" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Animated visualization showing pod to pod connectivity</span></span></P> <H3>&nbsp;</H3> <H3>Pod to Internet:</H3> <H3><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="pod_outbound.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111302iBACC612A7FD1A427/image-size/large?v=v2&amp;px=999" role="button" title="pod_outbound.gif" alt="Animated visualization showing pod to outbound connectivity" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Animated visualization showing pod to outbound connectivity</span></span></H3> <H3>&nbsp;</H3> <H3>Pod to Service:</H3> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="pod-serviceVIP.gif" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/111304i479D28762FEEB2E3/image-size/large?v=v2&amp;px=999" role="button" title="pod-serviceVIP.gif" alt="Animated visualization showing pod to service connectivity" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Animated visualization showing pod to service connectivity</span></span></P> <P>Showing how to analyze and debug these packet captures<SPAN>&nbsp;</SPAN>will be done in future part(s) of this blog post series through scenario-driven videos showing packet captures for supported networking flows on Windows.</P> <P>&nbsp;</P> <P>For a quick teaser, here is a video recording taken at KubeCon that shows debugging an issue live using startpacketcapture.cmd: <A href="#" target="_blank" rel="noopener">https://www.youtube.com/watch?v=tTZFoiLObX4&amp;t=1733</A></P> <P>&nbsp;</P> <H2>Summary</H2> <P>We looked at:</P> <OL> <LI>Automated scripts that can be used to verify basic connectivity and correct installation of Kubernetes</LI> <LI>HNS networking objects and VFP rules used to network containers</LI> <LI>How to query event logs from different Kubernetes components</LI> <LI>How to analyze the control path at a high level for common configuration errors using CollectLogs.ps1</LI> <LI>Typical network packet flows for common connectivity scenarios</LI> </OL> <P>Performing the above steps can go a great length towards understanding the underlying issue for an observed symptom, improve efficacy when it comes to implementing workarounds, and accelerate the speed at which fixes are implemented by others, having already performed the initial investigation work.</P> <P>&nbsp;</P> <H2>What’s next?</H2> <P>In the future, we will go over supported connectivity scenarios and specific steps on how to troubleshoot each one of them in-depth. These will build on top of the materials presented here but also contain videos analyzing packet captures, data-path analysis as well as other traces (e.g. HNS tracing).</P> <P>&nbsp;</P> <H2>We are looking for your feedback!</H2> <P>Last but not least, the Windows container networking team needs your feedback! What would you like to see next for container networking on Windows? Which bugs are preventing you from realizing your goals? Share your voice in the comments below, or fill out the following&nbsp;<A href="#" target="_blank" rel="noopener">survey</A> and influence our future investments!</P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Thank you for reading,</P> <P>David Schott</P> <P>&nbsp;</P> <P>*Special thanks to Dinesh Kumar Govindasamy (Microsoft) for his fantastic work creating &amp; presenting many of the materials used as a basis for this blog at KubeCon Shanghai '18!</P> Wed, 31 Mar 2021 16:54:47 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshooting-kubernetes-networking-on-windows-part-1/ba-p/508648 David Schott 2021-03-31T16:54:47Z Synthetic Accelerations in a Nutshell – Windows Server 2016 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2016/ba-p/535571 <P>Hi folks,</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Dan Cuomo</A>&nbsp;back for our next installment in this blog series on synthetic accelerations.&nbsp; Windows Server 2016 marked an inflection point in the synthetic acceleration world on Windows, so in this article we’ll talk about the architectural changes, new capabilities, and changes in configuration requirements compared to the last couple operating systems.</P> <P>&nbsp;</P> <P>Before we begin, here are the pointers to the previous blogs:</P> <UL> <LI><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012/ba-p/447792" target="_blank" rel="noopener">Synthetic Accelerations in a Nutshell – Windows Server 2012</A></LI> <LI><A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012-R2/ba-p/481428" target="_blank" rel="noopener">Synthetic Accelerations in a Nutshell - Windows Server 2012 R2</A></LI> </UL> <P>Keep in mind that due to changes in Windows Server 2016, many of the details in Windows Server 2012/R2 becomes irrelevant while some become even more important.&nbsp; For example, vRSS becomes more important while the benefits of Dynamic VMQ,&nbsp;as originally implemented in Windows Server 2012 R2, is surpassed in this release. We’ll wrap up this series in a future post that covers Windows Server 2019 (and the new and improved Dynamic VMMQ)!</P> <P>&nbsp;</P> <P>But before we get to the grand finale in Windows Server 2019, let’s talk about the groundwork that occurred in Windows Server 2016 and the big advances they brought in synthetic (through the virtual switch) network performance.&nbsp; As a quick review, this is the synthetic datapath; all ingress traffic must traverse the virtual switch (vSwitch) in the parent partition prior to being received by the guest:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="1.png" style="width: 798px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112300i29630B582BDC54FE/image-size/large?v=v2&amp;px=999" role="button" title="1.png" alt="1.png" /></span></P> <P>&nbsp;</P> <H2>NIC Architecture in Windows Server 2016</H2> <P>Windows Server 2016 brought a new architecture in the NIC that affects the implementation of VMQ.&nbsp; If you remember the article on <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012/ba-p/447792" target="_blank" rel="noopener">Windows Server 2012</A>, you may remember that the NIC creates a filter based on each vmNIC’s MAC and VLAN combination on the Hyper-V vSwitch.&nbsp; Ergo, every MAC and VLAN combination registered on the Hyper-V vSwitch would be passed to the NIC to request a VMQ be mapped.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG><EM>Note: </EM></STRONG><EM>Well not exactly <STRONG>every</STRONG> combination.&nbsp; VMQs must be requested using the virtual machine properties so only VMs that have the appropriate properties would be passed to the NIC for allocation of a queue.&nbsp; The rest, as you may recall, land in the default queue.</EM></P> </BLOCKQUOTE> <P>Now however, some added intelligence in the form of an embedded Ethernet switch (NIC Switch) was built into the physical network adapters.&nbsp; When Windows detects that a NIC has a NIC Switch, it asks the NIC to map a NIC Switch port (vPort) to a queue (instead of assigning a queue to the MAC + VLAN filter mentioned previously).&nbsp; Here’s a look at the Legacy VMQ (MAC + VLAN filtered) architecture:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="2.png" style="width: 865px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112302i6A844D68257B8F45/image-size/large?v=v2&amp;px=999" role="button" title="2.png" alt="2.png" /></span></P> <P>&nbsp;</P> <P>Here’s the new architecture.&nbsp; If you look at the NIC at the bottom of the picture, you can see a queue is now mapped to a vPort and that vPort maps to a processor.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="3.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112303iC5661D8FAF2CFA78/image-size/large?v=v2&amp;px=999" role="button" title="3.png" alt="3.png" /></span></P> <P>&nbsp;</P> <P>In previous operating systems, the NIC Switch was used only when SR-IOV was enabled on the virtual switch.&nbsp; However, in Windows Server 2016 we split the use of the NIC Switch away from SR-IOV and now leverage it to enable some great advantages.</P> <P>&nbsp;</P> <H2>VRSS without the offload</H2> <P>If you’re running Windows Server 2016, you’re likely <STRONG>NOT</STRONG> in this exact situation.&nbsp; However, this section is important to understand the great leap in performance you’ll see in the next section (so don’t skip this section :smiling_face_with_smiling_eyes:</img>).</P> <P>&nbsp;</P> <P>Virtual Receive Side Scaling (vRSS) was <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012-R2/ba-p/481428" target="_blank" rel="noopener">first introduced in Windows Server 2012 R2</A>.&nbsp; As a review, vRSS has two primary responsibilities on the host:</P> <UL> <LI>Creating the mapping of VMQs to processors (known as the indirection table)</LI> <LI>Packet distribution onto processors</LI> </UL> <P>With Legacy VMQ in Windows Server 2012 R2 packet distribution, vRSS performs RSS spreading onto additional processors surpassing the throughput that a single VMQ and CPU could deliver.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 733px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112304iCB4048A54C7DBC09/image-size/large?v=v2&amp;px=999" role="button" title="4.png" alt="4.png" /></span></P> <P>&nbsp;</P> <P>While this enabled much improved throughput over Windows Server 2012, the packet distribution (RSS spreading) was performed in software and incurred CPU processing (this is covered in the 2012 R2 article so I won’t spend much time on this here).&nbsp; This capped the potential throughput to a virtual NIC to about 15 Gbps in Windows Server 2012 R2.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG><EM>Note</EM></STRONG><EM>: If you simply compare vRSS and VMQ on Windows Server 2012 R2 to Windows Server 2016, you’ll notice that your throughput with the same accelerations enabled naturally improves.&nbsp; This is for a couple of reasons but predominantly boils down to the improvements in drivers and operating system efficiency. &nbsp;This is one benefit of upgrading to the latest operating system and installing the latest drivers/firmware.</EM></P> </BLOCKQUOTE> <H2>VMMQ</H2> <P>(I said don’t skip the last section!&nbsp; If you did, go back and read that first!)</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="familyguy.gif" style="width: 200px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112305iA1061334DF5F9BAA/image-size/large?v=v2&amp;px=999" role="button" title="familyguy.gif" alt="familyguy.gif" /></span></P> <P>&nbsp;</P> <P>The big advantage of the NIC Switch is the ability to offload the packet distribution function performed by vRSS to NIC.&nbsp; This capability is known as Virtual Machine Multi Queues (VMMQ) and allows us to assign multiple VMQs to the same vPort in the NIC Switch, which you can see in the picture below.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112306iD3A94C9DECCE7ECC/image-size/large?v=v2&amp;px=999" role="button" title="5.png" alt="5.png" /></span></P> <P>&nbsp;</P> <P>Now instead of a messy software solution that incurs overhead on the CPU and processes the packets in multiple places, the NIC simply places the packet directly onto the intended processor.&nbsp; RSS hashing is used to spread the incoming traffic between the queues assigned to the vPort.</P> <P>&nbsp;</P> <P>In the last article, we discussed how VMQ and RSS had a budding friendship (see the “Orange Mocha Frappuccinos” reference ;)</img>).&nbsp; Now you should see just how integral RSS is to VMQ’s vitality; They’re like a&nbsp;<A style="background-color: #ffffff;" href="#" target="_blank" rel="noopener">frog and a bear singing America</A>.&nbsp; They’re movin' right along; they're truly birds of a feather; they’re in this together.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="MuppetMovie.gif" style="width: 450px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112307i3D9E3B3ECD620A63/image-size/large?v=v2&amp;px=999" role="button" title="MuppetMovie.gif" alt="MuppetMovie.gif" /></span></P> <P>&nbsp;</P> <P>Now the packets only need to be processed once before heading towards their destination – the virtual NIC.&nbsp; The result is a much more performant host and increased throughput to the virtual NIC.&nbsp; With enough available processing power, you can exceed 50 Gbps.</P> <H2>Management of a Virtual NIC’s vRSS and VMMQ Properties</H2> <P>In Windows Server 2016, VMMQ was disabled by default.&nbsp; It was after all, a brand-new feature and after the “Great Windows Server 2012 VMQ” debacle, we thought it best to disable new offloads by default until the NIC and OS had an opportunity to prove they could play nicely together.</P> <H3>Verify vRSS is Enabled for a vNIC</H3> <P>This should already be enabled (remember this is the OS component that directs the hardware offload, VMMQ) but you can see the settings by running this command for a vmNIC:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="1.png" style="width: 771px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112310iE1EB7C9AA3BCF1FB/image-size/large?v=v2&amp;px=999" role="button" title="1.png" alt="1.png" /></span></P> <P>&nbsp;</P> <P>And now a host vNIC – Oh yea! We enabled vRSS for host vNICs in this release as well!&nbsp; This was not previously available on 2012 R2.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="2.png" style="width: 691px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112311iF4F8C467E1FFA070/image-dimensions/691x83?v=v2" width="691" height="83" role="button" title="2.png" alt="2.png" /></span></P> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG>Note</STRONG>: If this was accidentally disabled and you’d like to reenable it, please see <A href="#" target="_blank" rel="noopener">Set-VMNetworkAdapter</A>.</P> </BLOCKQUOTE> <H3>To Enable VMMQ</H3> <P>First, check to see if VMMQ is enabled.&nbsp; As previously mentioned, it is not enabled by default on Server 2016.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="3.png" style="width: 685px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112312i5F250D842CDE0EA6/image-dimensions/685x105?v=v2" width="685" height="105" role="button" title="3.png" alt="3.png" /></span></P> <P>&nbsp;</P> <P>Now let’s check that the prerequisite hardware features are enabled.&nbsp; To do this, use <A href="#" target="_self">Get-NetAdapterAdvancedProperty</A> to query the device (or use the device manager properties for the adapter) for the following three properties:</P> <P>&nbsp;</P> <P><STRONG>Virtual Switch RSS<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112315i924A68C955193947/image-size/large?v=v2&amp;px=999" role="button" title="4.png" alt="4.png" /></span></STRONG></P> <P>&nbsp;</P> <P><STRONG>Virtual Machine Queues<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112317i702FB8CA4445870D/image-size/large?v=v2&amp;px=999" role="button" title="5.png" alt="5.png" /></span></STRONG></P> <P>&nbsp;</P> <P><STRONG>Receive Side Scaling<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="6.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112318i93BFF235A96F8101/image-size/large?v=v2&amp;px=999" role="button" title="6.png" alt="6.png" /></span></STRONG></P> <P>&nbsp;</P> <P>If any of the above settings are disabled (0) make sure to enable them with <A href="#" target="_self">Set-NetAdapterAdvancedProperty</A>.</P> <P>&nbsp;</P> <P>Now let’s enable VMMQ on the vNIC by running Set-VMNetworkAdapter:<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="7.png" style="width: 742px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112320iF78BB4A3BC5B19A0/image-size/large?v=v2&amp;px=999" role="button" title="7.png" alt="7.png" /></span></P> <P>&nbsp;</P> <P>Finally, verify that VMMQ is enabled:</P> <P><STRONG><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="8.png" style="width: 775px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112322iDB7DD9B0DB540A09/image-size/large?v=v2&amp;px=999" role="button" title="8.png" alt="8.png" /></span></STRONG></P> <P>&nbsp;</P> <P>One last thing…In the picture above you can see the number of VMMQ Queue Pairs (see the 2012 article on <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012/ba-p/447792" target="_blank" rel="noopener">NIC Architecture</A> – a queue is technically a queue pair) assigned for this virtual NIC ☹.&nbsp; It shows the request was for 16, so why did only 8 get allocated?</P> <P>&nbsp;</P> <P>First, understand the perspective of each properties output.&nbsp; <STRONG>VMMQQueuePairRequested</STRONG> is what vRSS requested on your behalf – in this case 16.&nbsp; <STRONG>VMMQQueuePairs</STRONG> is the actual number granted by the hardware.</P> <P>&nbsp;</P> <P>As you know from our previous Get-NetAdapterAdvancedProperty cmdlet above, each network adapter has defined defaults that governs its properties.&nbsp; The <STRONG>*MaxRssProcessors</STRONG> property (intuitively) defines the maximum number of RSS processors that can be assigned for any adapter, virtual NICs included.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="9.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112323iB9D073F4B8A0ADCE/image-size/large?v=v2&amp;px=999" role="button" title="9.png" alt="9.png" /></span></P> <P>&nbsp;</P> <P>Lastly the <STRONG>*NumRSSQueues </STRONG>defines how many queues can be assigned.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="10.png" style="width: 822px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112324iDD0FC15B6D05F237/image-size/large?v=v2&amp;px=999" role="button" title="10.png" alt="10.png" /></span></P> <P>&nbsp;</P> <P>We can remedy this by changing these properties using Set-NetAdapterAdvancedProperty or device manager.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="1.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112327i890FCA936E015632/image-size/large?v=v2&amp;px=999" role="button" title="1.png" alt="1.png" /></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="2.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112328iB0B615B8169B8F10/image-size/large?v=v2&amp;px=999" role="button" title="2.png" alt="2.png" /></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="3.png" style="width: 837px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112325i6657E356952EDD3A/image-size/large?v=v2&amp;px=999" role="button" title="3.png" alt="3.png" /></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 822px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112326iFDE4A045E3FAE65D/image-size/large?v=v2&amp;px=999" role="button" title="4.png" alt="4.png" /></span></P> <P>&nbsp;</P> <P>Now check that the virtual NIC now has 16 queue pairs:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 589px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112329iE34FD2C08FBEE6EF/image-size/large?v=v2&amp;px=999" role="button" title="5.png" alt="5.png" /></span></P> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG><EM>Note:</EM></STRONG> <EM>Ever wonder why there is an asterisk (*) in front of the Get-NetAdapterAdvancedProperty properties?</EM></P> <P>&nbsp;</P> <P><EM>These are known as well-known advanced registry keywords which is standardized software contract between the operating system and the network adapter.&nbsp; Any keyword listed here without an asterisk in front of it is defined by the vendor and may be different (or not exist) depending on the adapter and driver you use.</EM></P> </BLOCKQUOTE> <P>By lowering the number of <STRONG>VMMQQueuePairsRequested</STRONG> you have an easy mechanism to manage the available throughput into a VM.&nbsp; You should assign 1 Queue Pair for every 4 Gbps you’d like a VM to have.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="6.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112330iBC3F9A355976E299/image-size/large?v=v2&amp;px=999" role="button" title="6.png" alt="6.png" /></span></P> <P>&nbsp;</P> <P>If you choose to do this, keep in mind two things.&nbsp; First, VMMQ is not a true QoS mechanism. &nbsp;Your mileage may vary as the actual throughput will depend on the system and available resources.</P> <P>&nbsp;</P> <P>Second, VMMQ scales considerably better than VMQ alone in large part because of the improvements to the default queue outlined in the next section so you may not need to manage the allocation of queues quite as stringently as you have in the past.</P> <P>&nbsp;</P> <H2>Management of default queues</H2> <P>In the last section, we enabled VMMQ for a specific virtual NIC.&nbsp; However, you may want to enable VMMQ for the default queue.&nbsp; As a general best practice, we recommend that you enable VMMQ for the default queue.</P> <P>&nbsp;</P> <P>As a reminder, the default queue is the queue that can apply to multiple devices.&nbsp; More specifically, if you don’t get a VMMQ, you’ll use these.&nbsp; In the past, all VMs that were unable to receive their own VMQ shared the default queue.&nbsp; Now, they share multiple queues (e.g. VMMQ).<BR /><BR /></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="7.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112331i3E13FD3F5F1420AF/image-size/large?v=v2&amp;px=999" role="button" title="7.png" alt="7.png" /></span></P> <P>&nbsp;</P> <P>You can see above the same setup is required for the <STRONG>DefaultQueueVMMQPairs</STRONG>.&nbsp; The only difference is that, instead of setting the configuration on the virtual NIC using Set-VMNetworkAdapter, you set the configuration on the virtual switch like this:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="8.png" style="width: 850px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112332i8285508CAB4147F1/image-size/large?v=v2&amp;px=999" role="button" title="8.png" alt="8.png" /></span></P> <P><BR /><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="9.png" style="width: 768px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112333iDBFC7C59AF7F762D/image-size/large?v=v2&amp;px=999" role="button" title="9.png" alt="9.png" /></span></P> <P>&nbsp;</P> <P>Now any virtual machine that is unable to receive its own queues will benefit from having 16 (or whatever you configure) available to share the workload.</P> <P>&nbsp;</P> <H2>Processor Arrays</H2> <P>In Windows Server 2016, you are no longer required to set the processor arrays with Set-NetAdapterVMQ or Set-NetAdapterRSS.&nbsp; I’ve been asked if you still can configure these settings if you have a desire to, and the answer is yes.&nbsp; However, the scenarios when this is useful are few and far between.&nbsp; For general use, this is no longer a requirement.</P> <P>&nbsp;</P> <P>As you can see in the picture below, the RSSProcessorArray includes processors by default and they are ordered by NUMA Distance.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="10.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112335iC6D8DD25989F0912/image-size/large?v=v2&amp;px=999" role="button" title="10.png" alt="10.png" /></span></P> <P>&nbsp;</P> <H2>#DownWithLBFO</H2> <P>You may have seen our <A href="#" target="_self">minor twitter campaign</A> about eliminating LBFO from your environment…</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="11.png" style="width: 426px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112336i68A6EE9E5A8BF5D0/image-dimensions/426x416?v=v2" width="426" height="416" role="button" title="11.png" alt="11.png" /></span></P> <P>&nbsp;</P> <P>There are numerous reasons for this recommendation, however it boils down to this:</P> <P style="text-align: center;"><EM>LBFO is our older teaming technology that will not see future investment, is not compatible with numerous advanced capabilities, and has been exceeded in both performance and stability by our new technologies (SET).</EM></P> <P style="text-align: center;">&nbsp;</P> <BLOCKQUOTE> <P style="text-align: left;"><EM><STRONG>Note</STRONG>: If you're new to Switch Embedded Teaming (SET) you can review <A href="#" target="_self">this guide</A> for an overview</EM></P> </BLOCKQUOTE> <P>We’ll have a blog that will unpack that statement quite a bit more but let’s talk about it in terms of synthetic accelerations.&nbsp; LBFO doesn’t support offloads like VMMQ.&nbsp; VMMQ is an <EM>advanced capability</EM> that lowers the host CPU consumption and enables <EM>far better network throughput</EM> to virtual machines. &nbsp;In other words, your users (or customers) will be happier with VMMQ in that they can generally get the throughput they want, when they want it, so long as you have the processing power and network hardware to meet their demands. &nbsp;If you want to use VMMQ and you’re looking for a teaming technology, you must use Switch Embedded Teaming (SET).</P> <P>&nbsp;</P> <P>To do this, simply add <STRONG>-EnableEmbeddedTeaming $true</STRONG> to your New-VMSwitch cmdlet.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="12.png" style="width: 739px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/112337i5C04CA3EB55B80ED/image-size/large?v=v2&amp;px=999" role="button" title="12.png" alt="12.png" /></span></P> <P>&nbsp;</P> <H2>Summary of Requirements</H2> <P>We continue to chip away at the requirements on the system administrator as compared to previous releases.</P> <P>&nbsp;</P> <UL> <LI>Install latest drivers and firmware</LI> </UL> <P>&nbsp;</P> <UL> <LI><SPAN style="text-decoration: line-through;">Processor Array engaged by default – CPU0</SPAN>.&nbsp;<SPAN style="font-family: inherit;">This was originally changed in 2012 R2 to enable VRSS (on the host), however this now includes host virtual NICs and hardware queues (VMQ/VMMQ) as well.</SPAN></LI> </UL> <P>&nbsp;</P> <UL> <LI>Configure the system to avoid CPU0 on non-hyperthreaded systems and CPU0 and CPU1 on hyperthreaded systems (e.g.&nbsp;BaseProcessorNumbershould be 1 or 2 depending on hyperthreading).</LI> </UL> <P>&nbsp;</P> <UL> <LI><SPAN style="text-decoration: line-through;">Configure the&nbsp;MaxProcessorNumber&nbsp;to establish that an adapter cannot use a processor higher than this.</SPAN>&nbsp; The system will now manage this automatically in Windows Server 2016 and so we recommend you do not modify the defaults.</LI> </UL> <P>&nbsp;</P> <UL> <LI><SPAN style="text-decoration: line-through;">Configure&nbsp;MaxProcessors&nbsp;to establish how many processors out of the available list a NIC can spread VMQs across simultaneously.</SPAN>&nbsp;&nbsp;This is unnecessary due to the enhancements in the default queue.&nbsp; You may still choose to do this if you’re limiting the queues as a rudimentary QoS mechanism as noted earlier but it is not required.</LI> </UL> <P>&nbsp;</P> <UL> <LI>Enable VMMQ on Virtual NICs and the Virtual Switch.&nbsp; This is new as this capability didn't exist prior to this release and as mentioned, we disabled new offloads.</LI> </UL> <P>&nbsp;</P> <UL> <LI>Test customer workload</LI> </UL> <P>&nbsp;</P> <H2>Summary of Advantages</H2> <UL> <LI><STRONG>Spreading across virtual CPUs (vRSS in the Guest)</STRONG>&nbsp;– The virtual processors have been removed as a bottleneck (originally implemented in 2012 R2).</LI> </UL> <UL> <LI><STRONG>vRSS Packet Placement Offload</STRONG> – Additional CPUs can be engaged by vRSS (creation of the indirection table implemented in 2012 R2). Now packet placement onto the correct processor can be done in the NIC improving the performance of an individual virtual NIC to +50 Gbps with adequate available resources. This represents another 3x improvement over Windows Server 2012 R2 (and 6x over Windows Server 2012)!</LI> </UL> <BLOCKQUOTE> <P><STRONG>Note: </STRONG>Some have commented that they have no need for an individual virtual machine to receive +50 Gbps on its own.&nbsp; While these scenarios are (currently) less common, it misses the point.&nbsp; The actual benefit is that +50 Gbps can be processed by the system whether that be 100 VMs / 50 Gbps == 2 Gbps per VM or +50 Gbps for 1 VM.&nbsp; You choose how to divvy up the available bandwidth.</P> </BLOCKQUOTE> <UL> <LI><STRONG>Multiple queues for the Default Queue&nbsp;</STRONG>– Previously the default queue was a bottleneck for all virtual machines that couldn’t receive their own dedicated queue.&nbsp; Now those virtual machines can leverage VMMQ in the default queue enabling greater scaling of the system.</LI> </UL> <P>&nbsp;</P> <UL> <LI><STRONG>Management of the Default Queue</STRONG> – You can choose how many queues for the default queue and each virtual NIC</LI> </UL> <P>&nbsp;</P> <UL> <LI><STRONG>Pre-allocated queues </STRONG>– By pre-allocating the queues to virtual machines, we’re able to meet the demand of bursty network workloads.</LI> </UL> <H2>Summary of Disadvantages</H2> <UL> <LI><STRONG>VMMQ is disabled by default</STRONG> – You need to enable VMMQ individually or use tooling to enable it.</LI> <LI><STRONG>No dynamic assignment of Queues </STRONG>– Dynamic VMQ is effectively deprecated with the use of VMMQ which means that once a queue has been mapped to a processor it will not be moved in response to changing system conditions.</LI> <LI><STRONG>Pre-allocated queues</STRONG> – This is also a disadvantage because we may be wasting system resources without the ability to reassign them.</LI> </UL> <P>The redesigned NIC architecture (NIC Switch) enabled VMMQ which represents another big jump in synthetic system performance and efficiency.&nbsp; If you’re using the synthetic datapath, you’ll receive a huge boost over what vRSS and Dynamic VMQ alone can bring alone enabling another 3x improvement in performance.&nbsp; In addition, VMMQ enables improved system density (packing more VMs onto the same host) and a more consistent user experience as they will be able to leverage VMMQ in the default queue.&nbsp; Lastly, Switch Embedded Teaming (SET) has officially becomes our recommended teaming option in this release, in part due to its support for advanced offloads like VMMQ.</P> <P>&nbsp;</P> <P>I hope you enjoyed the first three articles in this series.&nbsp; Next week we’ll wrap up with the final installment describing Dynamic VMMQ and our first acceleration that isn't named VMQ or RSS!</P> <P>&nbsp;</P> <P>Dan "Accelerating" Cuomo</P> Wed, 08 May 2019 13:00:00 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2016/ba-p/535571 Dan Cuomo 2019-05-08T13:00:00Z Synthetic Accelerations in a Nutshell - Windows Server 2012 R2 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2012-r2/ba-p/481428 <P>Hi folks,</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Dan Cuomo</A> back for our next installment in this blog series on synthetic accelerations.&nbsp; This time we’ll cover Windows Server 2012 R2.&nbsp; If you haven’t already read the first post on Server 2012, I’d recommend <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012/ba-p/447792" target="_blank" rel="noopener">reviewing that</A> before proceeding here.&nbsp; Following this post, we’ll tackle Windows Server 2016, and finally Windows Server 2019.</P> <P>&nbsp;</P> <H1><FONT size="6">Background</FONT></H1> <P>Windows Server 2012 R2 added a few improvements to VMQ, including Dynamic VMQ which enables the processor processing network traffic destined for a virtual machine to be updated based on some heuristics.</P> <P>&nbsp;</P> <P>However, the most important feature introduced in 2012 R2 was vRSS which is critical to understand as we move into the future operating systems like Windows Server 2016 and 2019.&nbsp; vRSS is the basis for how we exceed the bandwidth of a single CPU as you’ll see in this post.</P> <P>&nbsp;</P> <P>As such, we’ll focus this article on vRSS then cover Dynamic VMQ so you can understand the journey when we complete our series with Windows Server 2016 and 2019.</P> <P>&nbsp;</P> <H1><FONT size="6">Virtual Receive Side Scaling (vRSS)</FONT></H1> <P>Virtual Receive Side Scaling (vRSS) was introduced in Windows Server 2012 R2.&nbsp; vRSS enables improved performance over VMQ alone through a couple of new responsibilities.</P> <P>&nbsp;</P> <P><FONT size="5">In the Guest</FONT></P> <P>Virtual CPUs fill up just like any other CPU and can become a bottleneck just like CPUs on the host.&nbsp; If the virtual machine has multiple virtual CPUs, it can alleviate this chokepoint by splitting the processing for the received network traffic across each virtual CPU in the guest.</P> <P>&nbsp;</P> <P>If you want to visualize this architecturally, a virtual NIC attaches to a “port” on the Hyper-V Virtual switch (vSwitch).&nbsp; This is similar to a physical port on a physical switch (except that it’s all virtual). &nbsp;In the picture below you can see that the vSwitch operates at layer 2 (ethernet).&nbsp; If you remember from our first article, legacy VMQ operates similarly using a MAC + VLAN filter to separate traffic into different VMQs.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="1.png" style="width: 611px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110009i58E9367243208474/image-dimensions/611x362?v=v2" width="611" height="362" role="button" title="1.png" alt="1.png" /></span></P> <P>&nbsp;</P> <P>Once the traffic reaches the vmNIC, it needs to traverse the network stack and be processed by the Guest’s vCPUs.&nbsp; In the guest, you’ll see the network traffic distributed in task manager similar to this thanks to vRSS.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="2.png" style="width: 664px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110010iF4044547F3B52013/image-dimensions/664x298?v=v2" width="664" height="298" role="button" title="2.png" alt="2.png" /></span></P> <P>&nbsp;</P> <P>This operates by using RSS inside the guest, hence the intuitive name vRSS :smiling_face_with_smiling_eyes:</img>.&nbsp; Enabling or disabling this can be done by modifying the Receive Side Scaling property on the vNIC inside the guest.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="3.png" style="width: 601px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110011iA83F6FE45981DA0C/image-dimensions/601x274?v=v2" width="601" height="274" role="button" title="3.png" alt="3.png" /></span></P> <P>&nbsp;</P> <P><FONT size="5">On the Host</FONT></P> <P>&nbsp;</P> <P>But vRSS ain’t no one-trick pony! &nbsp;vRSS also has responsibilities on the host as well.&nbsp; vRSS’s core responsibilities are:</P> <UL> <LI>Creating the mapping of VMQs to Processors (known as the indirection table)</LI> <LI>Packet distribution onto processors</LI> </UL> <P>When VMQ interrupts a processor, it always interrupts the “base processor” for a specific VMQ.&nbsp; In the example below, the VM with the MAC Address ending in <STRONG>7A-35-0C</STRONG> always interrupts processor 17.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 661px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110013iB0190356836FB5EF/image-dimensions/661x125?v=v2" width="661" height="125" role="button" title="4.png" alt="4.png" /></span></P> <P>&nbsp;</P> <P>Next, we use RSS technology to spread the flows across other available processors in the processor array.&nbsp; While this is a bit of work for the system, the benefit is that the throughput can surpass the bandwidth of a single VMQ.&nbsp; Remember, the base processor must always receive the packets first, but the heavy lifting is performed by other available processors.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><FONT size="3"><STRONG>Note: </STRONG>You may recognize that vRSS and VMQ are working together.&nbsp; CPU spreading with vRSS requires VMQ.</FONT></P> </BLOCKQUOTE> <P>&nbsp;</P> <P>Now a VM can receive the bandwidth (approximately) equivalent to the number of processors engaged to do its bidding.&nbsp; Typically, the default is 8 processors.&nbsp; Excellent…</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 300px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110015i7CCD0DCF496CEC8E/image-dimensions/300x232?v=v2" width="300" height="232" role="button" title="5.png" alt="5.png" /></span></P> <P>&nbsp;</P> <P>Here’s a diagram that shows the packet flow.&nbsp; In the picture below:</P> <UL> <LI>All packets destined for VM01 are received by VMQ2 in the physical NIC</LI> <LI>VMQ2 Interrupts the CPU2 (the base processor)</LI> <LI>The virtual switch will use CPU2 to process as many packets as it can</LI> <LI>If required to keep up with the inbound workload, the virtual switch will “hand-out” packets to the other processors (in this case CPU5). It will engage no more than the value of <STRONG>MaxProcessors</STRONG> (in this case 3) set in the vRSS indirection table</LI> </UL> <BLOCKQUOTE> <P><FONT size="3"><STRONG>Note</STRONG>: The virtual switch from the picture above was replaced with the vRSS indirection table however this is the same feature; vRSS is a function of the virtual switch.</FONT></P> </BLOCKQUOTE> <P>&nbsp;</P> <P><FONT size="3"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="6.png" style="width: 698px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110016iE969BDE9A2271499/image-dimensions/698x298?v=v2" width="698" height="298" role="button" title="6.png" alt="6.png" /></span></FONT></P> <P>&nbsp;</P> <H2><FONT size="6"><SPAN style="background: white;">The Base CPU</SPAN></FONT></H2> <P>&nbsp;</P> <P>The base CPU is chosen out of the available processors in the RSS processor array on the system.&nbsp; In the screen shot below, we constrain the processor array to processors 12 – 16.&nbsp; You can see that the physical adapter (which is configured with a virtual switch), updates the <STRONG>RSSProcessorArray</STRONG> property accordingly.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="1.png" style="width: 688px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110017i95A58F7043360769/image-dimensions/688x247?v=v2" width="688" height="247" role="button" title="1.png" alt="1.png" /></span></P> <P>&nbsp;</P> <BLOCKQUOTE> <P><FONT size="3"><STRONG>Note: </STRONG>There is no RSS indirection table shown because there is a virtual switch attached and TCP/IP is no longer bound to this adapter.&nbsp; Instead the virtual switch manages the indirection table.</FONT></P> </BLOCKQUOTE> <P>&nbsp;</P> <P>Next, if we look at <STRONG>Get-NetAdapterVMQ</STRONG>, you can see that the BaseVmqProcessor for the adapter has been updated to one in the RSSProcessorArray.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="2.png" style="width: 677px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110019iFF79ADB77C2F0C05/image-dimensions/677x101?v=v2" width="677" height="101" role="button" title="2.png" alt="2.png" /></span></P> <P>&nbsp;</P> <P>And the processors listed below for each VM have been constrained between 12 and 16 as we set in a previous command.</P> <P>&nbsp;</P> <P><STRONG>Note: </STRONG>In this example, we use one possible command to get the VMQ assigned for a vNIC.&nbsp; You could also use Get-NetAdapterVMQQueue.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="3.png" style="width: 672px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110021i126380B469753E58/image-dimensions/672x91?v=v2" width="672" height="91" role="button" title="3.png" alt="3.png" /></span></P> <P>&nbsp;</P> <H1>Setting the Groundwork for 2016 and Beyond</H1> <P>As you can see, we did not need to use the Set-NetAdapterVMQ cmdlets in the previous blog post (although they would have worked as well).</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 639px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110022iBE30866300142C24/image-dimensions/639x217?v=v2" width="639" height="217" role="button" title="4.png" alt="4.png" /></span></P> <P>&nbsp;</P> <P>As we move forward to the next article on Windows Server 2016, you’ll begin to understand how integral RSS is to VMQ.&nbsp; In Windows Server 2012, the two technologies were mutually exclusive (per adapter).&nbsp; In Windows Server 2012 R2, RSS isn’t yet the peanut butter to VMQ’s jelly, but you can see there is a budding friendship starting to form…In the 2012 R2 days, probably closer to orange mocha frappuccinos…</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="asdf.gif" style="width: 489px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110025i7439C047D1226A69/image-dimensions/489x275?v=v2" width="489" height="275" role="button" title="asdf.gif" alt="asdf.gif" /></span></P> <P>&nbsp;</P> <H2><FONT size="5">High and Low Throughput</FONT></H2> <P>VRSS intends to engage the least number of processors necessary to process the received traffic.&nbsp; This is for efficiency as there is a tax to engage and spread across each additional CPU.&nbsp; On low throughput, this may mean that only the base CPU is engaged.&nbsp; However, on high-throughput environments, the number of CPUs that <STRONG>can be</STRONG> engaged is equal to the value of <EM>MaxProcessors</EM>.</P> <P>&nbsp;</P> <P>At maximum throughput, the base CPU may be receiving so many packets that it’s only able to process a minimal number of packets itself.&nbsp; Instead, it hands out packets to the others in the processor array.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 733px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110026i959F86A9FD31229A/image-dimensions/733x321?v=v2" width="733" height="321" role="button" title="5.png" alt="5.png" /></span></P> <P>&nbsp;</P> <P>In the example shown above, this VM will receive throughput equivalent to what approximately 2 – 3 processors can crank out (because the <EM>MaxProcessors </EM>value is limited to 3).&nbsp; As mentioned before, this is because most of the base CPU’s work is simply handing packets out to other processors.&nbsp; Despite engaging three processors, you’re getting a bit less than that.</P> <P>&nbsp;</P> <P>In fact, you may find that despite setting MaxProcessors to 8, you may only engage a few processors, this is likely because your Base Processor is pegged and cannot hand out any more packets to other processors.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><FONT size="3"><STRONG>Note: </STRONG>The defaults are specific to your NIC, however in most cases, MaxProcessors defaults to 8 and was changed in this example only for visualization purposes.</FONT></P> </BLOCKQUOTE> <P>&nbsp;</P> <P>As you can imagine, this is far better than VMQ alone.&nbsp; A single VM could receive approximately 15 Gbps using this mechanism – a nearly 3x improvement!&nbsp; In the next post on Windows Server 2016 we’ll explain how VMMQ enhances this to get even better throughput and lower the CPU cost, improving your system’s density.</P> <P>&nbsp;</P> <P>You can see which processors are being engaged for virtual traffic by opening perfmon, changing to the histogram view, and adding the <STRONG>Hyper-V Virtual Switch Processor</STRONG> Object with the <STRONG>Packets from External/sec </STRONG>counter.&nbsp; Next use NTTTCP or CTSTraffic to send traffic from another machine to your VM.</P> <P>&nbsp;</P> <P>In the picture below, you can see that two processors are engaged: CPU12 (shown in Red) and CPU13 (shown in green). &nbsp;Each CPU that is processing packets will be represented by a bar. The height of the bar indicates the number of packets being processed by that specific CPU.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="6.png" style="width: 505px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110027i969A2300ED42E3E0/image-dimensions/505x371?v=v2" width="505" height="371" role="button" title="6.png" alt="6.png" /></span></P> <P>&nbsp;</P> <H1><FONT size="6">Dynamic VMQ</FONT></H1> <P>The next major improvement in 2012 R2 was called Dynamic VMQ.&nbsp; To make better use of VMQ and available system resources, the OS was improved in Windows Server 2012 R2 to allow a VMQ to processor assignment be updated dynamically based on its CPU load.</P> <P>&nbsp;</P> <P>If you refer back to <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Synthetic-Accelerations-in-a-Nutshell-Windows-Server-2012/ba-p/447792" target="_blank" rel="noopener">the article on 2012</A>, you can see an obvious challenge where multiple VMs using the same processor can easily starve one another.&nbsp; In other words, the maximum throughput of a single CPU core does not change from around 5 or 6 Gbps.</P> <P>&nbsp;</P> <P>Imagine there are 3 VMs all with a VMQ but assigned to the same processor, and that processor can process a total of 6 Gbps. &nbsp;Each VM could potentially receive 2 Gbps if they were distributed evenly.&nbsp;</P> <P>&nbsp;</P> <P>However, network traffic isn’t that predictable.&nbsp; It’s just as likely that 5 Gbps goes to VM1 and the remaining processing power is split among the other VMs.&nbsp; When the other VMs suddenly need more throughput, it’s not available for them because VM1 is already dominating the core used to process the data.</P> <P>&nbsp;</P> <P>In this picture <STRONG><FONT color="#FF0000">VM03</FONT> </STRONG>is assigned to the same core as <STRONG><FONT color="#00FF00">VM01</FONT></STRONG>.&nbsp; In this way, VM01 and VM03 are in competition for the CPU cores.&nbsp; Their individual performance is gated by the VMs sharing that processor.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="7.png" style="width: 672px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110028iAFB8E2A109666BDD/image-dimensions/672x315?v=v2" width="672" height="315" role="button" title="7.png" alt="7.png" /></span></P> <P>&nbsp;</P> <P>With Dynamic VMQ however, if CPU1 is not able to process the incoming packets, one of the queues can be reassigned to interrupt a different processor if one is able to sustain the workload.</P> <P>&nbsp;</P> <P>The measurements determining that a queue should be reassigned is happening in Windows (this is another one of vRSS’s responsibilities).&nbsp; Since the queues can now move and they exist in the NIC you should pay special attention to keep your NIC’s driver and firmware up-to-date.</P> <P>&nbsp;</P> <P>Dynamic VMQ is all about enabling your workloads to have consistent performance.&nbsp; If you have a night-owl for a CEO, and he tries to login at night when the backups are running, you want him to have the same performance as when he works during the day.&nbsp; Otherwise, you might have to work the same hours!</P> <P>&nbsp;</P> <P>That said, there’s a couple of problems in this implementation which will be addressed in Windows Server 2019.&nbsp; Most notably, 2012 R2 does not handle workloads that burst very well.&nbsp; If you have a system that needs max throughput suddenly, Dynamic VMQ isn’t able to do that.&nbsp; Instead, you’ll notice that the system ramps up slowly, maxing out at around 15 Gbps.&nbsp; This is because the system only reserves the bare minimum resources required, pre-allocating only the single VMQ available when network throughput begins.&nbsp; As demand for throughput increases, the system takes some measurements, vRSS expands to more CPUs, it takes some more measurements, expands to more CPUs, and finally max throughput is reached.</P> <P>&nbsp;</P> <H2><FONT size="5">Implications of the Default Queue</FONT></H2> <P>As a refresher, the default queue is where all traffic that doesn’t match a filter land - this is a shared VMQ.</P> <P>&nbsp;</P> <P>With vRSS however, this is less painful because the packets from the default queue can be distributed to other available processors.&nbsp; While VMs that land on this VMQ will not reach the same performance metrics as VMs with a dedicated queue, the delta is not as severe as enabling a much higher density of VMs on the same hardware.</P> <P>&nbsp;</P> <H2><FONT size="5">Host Virtual NICs</FONT></H2> <P>Host virtual NICs cannot leverage vRSS or Dynamic VMQ.&nbsp; This will be addressed in later versions however if your host NICs need more than what a single VMQ can provide, Windows Server 2012 R2 can’t help you.</P> <P>&nbsp;</P> <P>If you look at the properties of a Host virtual NIC there is no RSS capability listed.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="9.png" style="width: 673px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/110029i10C132FBAB63AE16/image-dimensions/673x275?v=v2" width="673" height="275" role="button" title="9.png" alt="9.png" /></span></P> <H1><FONT size="5">Summary of Requirements</FONT></H1> <P>We had a similar list in the article on Windows Server 2012.&nbsp; I’ll keep this in each blog so you can see the progression.&nbsp; As you can see in 2012 R2, there were performance and workload stability improvements, however the management experience didn't change much.</P> <P>&nbsp;</P> <UL> <LI>Install latest drivers and firmware – Even more important now that queues are moving!</LI> <LI><STRIKE>Processor Array engaged by default – CPU0</STRIKE></LI> <LI>Configure the system to avoid CPU0 on non-hyperthreaded systems and CPU0 and CPU1 on hyperthreaded systems (e.g. <STRONG>BaseProcessorNumber</STRONG> should be 1 or 2 depending on hyperthreading)</LI> <LI>Configure the <STRONG>MaxProcessorNumber</STRONG> to establish that an adapter cannot use a processor higher than this.</LI> <LI>Configure <STRONG>MaxProcessors</STRONG> to establish how many processors out of the available list a NIC can spread VMQs across simultaneously</LI> <LI>Test customer workload</LI> </UL> <P>&nbsp;</P> <P><FONT size="5"><STRONG>Other best practices</STRONG></FONT></P> <P>&nbsp;</P> <P>All 2012 best practices move forward including that<SPAN>&nbsp;</SPAN><STRONG>NOT</STRONG><SPAN>&nbsp;</SPAN>disabling RSS unless directed by Microsoft support for troubleshooting -&nbsp;<SPAN>Disabling RSS is not a supported configuration on Windows.&nbsp;&nbsp;RSS can be temporarily disabled by Microsoft Support recommendation for troubleshooting whether there is an issue with the feature. RSS should not be permanently disabled, which would leave Windows in an unsupported state.</SPAN></P> <P>&nbsp;</P> <P><FONT size="5"><STRONG>Summary of Advantages</STRONG></FONT></P> <UL> <LI><STRONG>Spreading across virtual CPUs (vRSS in the Guest)</STRONG> – The virtual processors have been removed as a bottleneck.</LI> <LI><STRONG>Spreading across physical CPUs (vRSS on the host)</STRONG> – Additional CPUs can be engaged to improve the performance of an individual virtual NIC by ~3x (approximately 15 Gbps).</LI> <LI><STRONG>Dynamic Assignment </STRONG>– Overburdened processors can be moved to processors with less workload if the hardware (firmware/driver) supports it and an available processor is available.</LI> <LI><STRONG>Default Queue </STRONG>– An overprovisioned system with more VMs than queues can “ease the pain” of a shared default queue by spreading and balance traffic across multiple CPUs enabling higher VM density.</LI> </UL> <P><FONT size="5"><STRONG>Summary of Disadvantages</STRONG></FONT></P> <UL> <LI><STRONG>One VMQ per virtual NIC </STRONG>– One is the loneliest number, and that goes for VMQs as well. While vRSS eases the pain, scaling up the workload to achieve higher throughputs means a considerable tax in the form of a CPU penalty.</LI> <LI><STRONG>No Host vRSS </STRONG>– Host virtual NICs cannot take advantage of vRSS spreading on the host</LI> <LI><STRONG>No default queue management </STRONG>– There is minimal management of the default queue.</LI> <LI><STRONG>No management of vRSS</STRONG> –There is no management of vRSS</LI> <LI><STRONG>Bursty workloads cannot be satisfied</STRONG> – No preallocated resources means that the system must measure and react to the workload’s demands slowly</LI> </UL> <P>As you can see, vRSS and Dynamic VMQ brought significant enhancements to Hyper-V networking for its time.&nbsp; This allowed virtual machines to nearly triple the throughput into a virtual machine <EM>AND</EM> have some level of throughput consistency for the workload by balancing across available processors.&nbsp; In our next article, we’ll discuss the rearchitecture of network cards and how this enabled us to overcome some of the existing shortcomings in 2012 R2.</P> <P>&nbsp;</P> <P>Thanks for reading,<BR />Dan</P> <P>&nbsp;</P> Mon, 06 May 2019 21:56:26 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2012-r2/ba-p/481428 Dan Cuomo 2019-05-06T21:56:26Z Synthetic Accelerations in a Nutshell – Windows Server 2012 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2012/ba-p/447792 <P>Hi folks,</P> <P>&nbsp;</P> <P><A href="#" target="_blank" rel="noopener">Dan Cuomo</A> here to talk about the state of synthetic accelerations (sometimes synonymous with VMQ) on Windows Server.&nbsp; <EM>This is the first in a series of posts covering synthetic accelerations on 2012, 2012 R2, 2016, and 2019. &nbsp;We recommend that you read through the progression of posts (once available) to make sure you have the proper information for each operating system you’re managing.</EM></P> <P>&nbsp;</P> <P>Some things never change.&nbsp; They are what they are.&nbsp; For-ev-er.</P> <P>&nbsp;</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="a.png" style="width: 447px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108488i69C34B59F3852019/image-dimensions/447x317?v=v2" width="447" height="317" role="button" title="a.png" alt="a.png" /></span></P> <P>&nbsp;</P> <P>The sky will always be blue (unless you’re in Seattle), there will always be 60 seconds in every minute (<A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener">unless there’s a leap second</A>)…ok, well some things change. And due to the literary <A href="#" target="_blank" rel="noopener">rule of three</A>, technology changes too.</P> <P>&nbsp;</P> <P>Virtual Machine Queues (VMQ) is no exception.&nbsp; This technology was first introduced in Windows Server 2012 as a means to spread the processing of received network packets destined for a virtual NIC across different cores.&nbsp; Once 10 GbE adapters became prevalent, the network processing quickly consumed more than any one single core could handle.&nbsp; The first thing most folks do after getting their shiny new 10 GbE adapter would be to try and copy something over the network to see the blazing fast speed.&nbsp; All looked great, 9+ Gbps!&nbsp; So, you add a couple of VMs and repeat the experiment.&nbsp; Much to your chagrin you get an underwhelming 3 – 5 Gbps.&nbsp; What happened?&nbsp; Hyper-V must have a regression!?&nbsp;</P> <P>&nbsp;</P> <P>No, it’s just that you’re now pegging a single CPU core.&nbsp; <STRONG>**Enter VMQ**</STRONG></P> <P>&nbsp;</P> <P>I’ve seen many a blog about the correct way to poke it, prod it, and tweak VMQ into semi-willing submission.&nbsp; Windows Server releases have come and gone and as they have the best practices for VMQ have changed considerably.&nbsp; So, in this blog series we’ll try to dispel the myths and give the details you need to configure this properly across each of the in-market versions starting with today’s post on Windows Server 2012.&nbsp; We’ll also talk about some of the key changes in 2012 R2, 2016, and 2019 so you can understand how to configure your system if you’re running on each of these operating systems.</P> <P>&nbsp;</P> <P>But first, some background…</P> <P>&nbsp;</P> <H1>Background</H1> <P>First, you may have noticed that this blog is not titled VMQ Deep Dive 1, 2, or 3 – These previously published articles were awesome resources in their time (in fact, I’m borrowing some of the original content there for these articles).&nbsp; But time marches on and (coinciding with our migration to the new blogging site) there’s a need to set proper context for the available operating systems of the day.</P> <P>&nbsp;</P> <P>VMQ used to be an independent feature in the operating system, and as such, it became synonymous with network acceleration on Hyper-V.&nbsp; That’s no longer the case as you’ll see in the sections on Windows Server 2016 and 2019.&nbsp; In fact, we now call the VMQ described in this article “Legacy VMQ” (as compared to vPort-based VMQ which will be explained in the 2016 and 2019 articles).</P> <P>&nbsp;</P> <P>Credit where credit is due. The original articles that are being replaced/updated here are:</P> <UL> <LI>Gabe Silva - VMQ Deep Dive, 1 of 3</LI> <LI>Gabe Silva - VMQ Deep Dive, 2 of 3</LI> <LI>Gabe Silva - VMQ Deep Dive, 3 of 3</LI> <LI>Marco Cancillo - Virtual Machine Queue (VMQ) CPU assignment tips and tricks</LI> </UL> <P>In addition, this blog is so titled based on the data path that packets travel through the system to reach their intended destination, a virtual NIC in either the host or a guest.</P> <P>&nbsp;</P> <P>So, what do we mean by the “synthetic” data path?&nbsp; This means traffic that passes through the virtual switch.&nbsp; At a high-level, the data path looks like this:</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="b.png" style="width: 798px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108489iE894FC51BAA6EAD3/image-size/large?v=v2&amp;px=999" role="button" title="b.png" alt="b.png" /></span></P> <P>&nbsp;</P> <P>Packets come off the wire, into the NIC and miniport from the NIC vendor.&nbsp; The virtual switch processes the packets and sends the data to ride along the vmBus before reaching the vmNIC and being processed by the network stack in the VM.&nbsp; This is the most common data path for network communication in Hyper-V and, as you can see, the packets are passing through the virtual switch which requires use of the physical CPU cores.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><SUB><STRONG>Note: </STRONG>This does not apply to RDMA, or other Direct Memory Access technology such as SR-IOV.</SUB></P> </BLOCKQUOTE> <P>&nbsp;</P> <P>Now let’s zoom into the NIC architecture and interoperability with the 2012 OS family.</P> <P>&nbsp;</P> <H1>NIC Architecture in 2012</H1> <P>Every packet flow that enters a NIC is assigned a hardware queue in that NIC whether you’re using Hyper-V or not.&nbsp; How the adapter filters the packets is dependent on whether you have a virtual switch attached to that adapter.</P> <P>&nbsp;</P> <P>For example, if you have an adapter without Hyper-V attached, the NIC <A href="#" target="_blank" rel="noopener">calculates a hash</A> to distribute the packets to different queues in the NIC.&nbsp; The miniport driver can interrupt or DPC to CPU cores to have a CPU process the incoming packets. &nbsp;This is how <A href="#" target="_blank" rel="noopener">RSS works</A>.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><SUB><STRONG>Note: </STRONG>Each queue is a pair of queues; one send, one receive.&nbsp; The queues below are denoted as “QP” or queue pair.</SUB></P> </BLOCKQUOTE> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="c.png" style="width: 998px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108490i7A9EF16F37A7D7E4/image-size/large?v=v2&amp;px=999" role="button" title="c.png" alt="c.png" /></span></P> <P>&nbsp;</P> <P>However, when you have a virtual switch attached, the NIC creates a filter based on each vmNIC’s MAC and VLAN combination on the Hyper-V Virtual Switch (vSwitch).&nbsp; When a packet arrives in this context, a filter is applied that checks if the packet matches one of the already seen MAC and VLAN combinations; if there is a match, the packet is sent to the corresponding queue that has been assigned to that packet filter.&nbsp; If there is no match, it is sent to the default queue – There is always a default queue.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="d.png" style="width: 801px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108491iF692E48CE88BD1CD/image-dimensions/801x145?v=v2" width="801" height="145" role="button" title="d.png" alt="d.png" /></span></P> <P>&nbsp;</P> <H1>Disabling RSS</H1> <P>In this way, RSS was <STRONG>functionally</STRONG> disabled (for that adapter) the moment you attached a vSwitch.&nbsp; Regardless of what you see in <STRONG>Get-NetAdapterRSS</STRONG>, RSS is not in use with this version of the operating system. Due to some unfortunate support challenges, particularly with NIC drivers during the early days of VMQ, there was a common recommendation of manually disabling RSS when VMQ is enabled on the system.</P> <P>&nbsp;</P> <P>We do not support disabling RSS for any operating system for a couple of reasons:</P> <P>&nbsp;</P> <UL> <LI>Disabling RSS disables it for every adapter. Any adapter not attached to the virtual switch can still use RSS and as such would be negatively impacted by this change.</LI> </UL> <P>&nbsp;</P> <UL> <LI>As you’ll see in the future articles, there is a budding friendship between these features (RSS and VMQ). They’ll soon become fast friends…like peanut butter and jelly…</LI> </UL> <BLOCKQUOTE> <P><FONT size="3"><SUB><STRONG>Important: </STRONG>Disabling RSS is not a supported configuration on Windows.&nbsp;&nbsp;RSS can be temporarily disabled by Microsoft Support for troubleshooting whether there is an issue with the feature. RSS should not be permanently disabled, which would leave Windows in an unsupported state.</SUB></FONT></P> </BLOCKQUOTE> <H1>Available Processors</H1> <P>Virtual NICs on Windows Server 2012 will, by default, use one, and only one core to process all the network traffic that coming into the system. That core is CPU0.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="e.png" style="width: 625px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108492i47179EAAC2019777/image-size/large?v=v2&amp;px=999" role="button" title="e.png" alt="e.png" /></span></P> <P>&nbsp;</P> <P>This meant that without any changes from you, with a VMQ certified NIC and driver, your overall system throughput would still be limited to what CPU0 could attain.&nbsp; To remedy this, you had to tell Windows that it was OK for these queues to send their packets to other CPU cores as shown in the next couple of sections.</P> <P>&nbsp;</P> <H1>Configuration of Legacy VMQ</H1> <H2>Enabling / Disabling VMQ</H2> <P>To enable or disable VMQ use the <A href="#" target="_blank" rel="noopener">Enable-NetAdapterVmq</A> and <A href="#" target="_blank" rel="noopener">Disable-NetAdapterVmq</A> PowerShell cmdlets on the host adapters.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><SUB><STRONG>Note: </STRONG>Depending on your specific adapter, you may also need to review <STRONG>Get-NetAdapterAdvancedProperty </STRONG>and make sure the adapter is showing link up</SUB>.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="f.png" style="width: 822px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108493i563C412830326978/image-size/large?v=v2&amp;px=999" role="button" title="f.png" alt="f.png" /></span></P> </BLOCKQUOTE> <P>Next, identify the number of VMQs available. The number of queues available on your NIC can be found by running the PowerShell cmdlet <A href="#" target="_blank" rel="noopener">Get‑NetAdapterVmq</A> and looking at the column <STRONG>NumberOfReceiveQueues</STRONG> as shown here.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="g.png" style="width: 836px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108494i54B955877980283E/image-size/large?v=v2&amp;px=999" role="button" title="g.png" alt="g.png" /></span></P> <P>&nbsp;</P> <P>If your adapters are teamed, the total number of queues available to the devices attached to the team may be either the sum-of-queues from all adapters in the team, or could be limited to the min-of-queues available on any one single adapter.&nbsp; This depends on the teaming options selected when the team was created.&nbsp; For more information, please see <A href="#" target="_blank" rel="noopener">NIC Teaming documentation</A>.</P> <P>&nbsp;</P> <P>For the purposes of this guide, we’ll use the <STRONG><EM>recommended</EM></STRONG> sum-of-queues mechanism which requires a team to use switch-independent teaming mode and either Hyper-V port or Dynamic Load balancing.&nbsp;</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><SUB><STRONG>Note:</STRONG> In Windows Server 2012 and 2012 R2, only LBFO teaming was available.&nbsp; We do not recommend using LBFO on Windows Server 2016 and 2019 unless you are on a native host (without Hyper-V).</SUB></P> </BLOCKQUOTE> <P>&nbsp;</P> <H2>Engaging additional CPU Cores</H2> <P>As mentioned earlier, by default Windows Server 2012 engaged only CPU0 on NUMA0.&nbsp; To engage and toggle the CPU cores assignments used by VMQ use the Set-NetAdapterVmq command.&nbsp; This cmdlet uses similar parameters as Set-NetAdapterRSS.&nbsp; There are many possible configurations available (for more information, please see the documentation for <A href="#" target="_self">Set-NetAdapterVMQ</A>), however we’ll use a simple example to show how to use the cmdlets.</P> <P>&nbsp;</P> <P>Consider a server that has the following hardware and software configuration</P> <UL> <LI>2 CPUs with 10 cores and 10 logical processors; Hyperthreading is disable<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="h.png" style="width: 423px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108495i26143440EEB61EAF/image-size/large?v=v2&amp;px=999" role="button" title="h.png" alt="h.png" /></span> <P>&nbsp;</P> </LI> </UL> <BLOCKQUOTE> <P><SUB><STRONG>Note:</STRONG>&nbsp;&nbsp;Because hyperthreaded CPUs on the same core processor share the same execution engine, the effect is not the same as having multiple core processors. For this reason, RSS and VMQ do not use hyperthreaded processors.</SUB></P> </BLOCKQUOTE> <UL> <LI>2 physical NICs <UL> <LI>Names: <STRONG>pNIC01</STRONG>, <STRONG>pNIC02</STRONG></LI> <LI>60 VMQs (<EM>NumberOfReceiveQueues</EM>) per interface</LI> </UL> </LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="i.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108496i83371109501E4F02/image-size/large?v=v2&amp;px=999" role="button" title="i.png" alt="i.png" /></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="j.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108497i27AB788539BFE02C/image-size/large?v=v2&amp;px=999" role="button" title="j.png" alt="j.png" /></span></P> <P>&nbsp;</P> <UL> <LI>NICs are connected to an LBFO team intuitively named: <STRONG>LBFO</STRONG> <UL> <LI>Teaming Mode: Switch Independent</LI> <LI>LoadBalancingAlgorithm: Hyper-V Port</LI> </UL> </LI> </UL> <BLOCKQUOTE> <P dir="ltr"><SUB><STRONG>Note</STRONG>: In Windows Server 2016 and 2019, we no longer recommend using LBFO with the virtual switch</SUB></P> </BLOCKQUOTE> <UL> <LI>A virtual switch named LBFOvSwitch is attached to the LBFO team</LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="k.png" style="width: 635px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108498i36F52F20DB659393/image-size/large?v=v2&amp;px=999" role="button" title="k.png" alt="k.png" /></span></P> <P>&nbsp;</P> <P>Before we begin modifying anything, let’s check to see if there are any differences in the NUMA configuration of the adapters.&nbsp; First let’s see if we have multiple NUMA nodes.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="l.png" style="width: 446px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108499i181A1B4BE461BF9C/image-size/large?v=v2&amp;px=999" role="button" title="l.png" alt="l.png" /></span></P> <P>&nbsp;</P> <P>And how the pNICs are distributed across those NUMA nodes</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="m.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108500i78D2A2272BD3622E/image-size/large?v=v2&amp;px=999" role="button" title="m.png" alt="m.png" /></span></P> <P>&nbsp;</P> <P>As you can see above, both pNICs are on NumaNode 1.&nbsp; If your adapters have access to different NUMA Nodes, we recommend assigning the NIC to processors on the that NUMA Node for improved performance.</P> <P>Here’s the basic structure.&nbsp; As you can see, the virtual switch can currently “see” only one processor – <FONT color="#008000"><STRONG>CPU0</STRONG></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="n.png" style="width: 558px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108501i2AC1495540DC0199/image-dimensions/558x401?v=v2" width="558" height="401" role="button" title="n.png" alt="n.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>If you run Get-NetAdapterVmq on the system (now teamed), by default, it’ll look like this:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="o.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108502i9F4A81BAE9C23E88/image-size/large?v=v2&amp;px=999" role="button" title="o.png" alt="o.png" /></span></P> <P>&nbsp;</P> <P>Specifically, the <STRONG>MaxProcessorNumber</STRONG> is not defined.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="q.png" style="width: 671px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108503iD90DCFD86292235A/image-dimensions/671x127?v=v2" width="671" height="127" role="button" title="q.png" alt="q.png" /></span></P> <P>&nbsp;</P> <P>There are a couple of problems with this configuration:</P> <UL> <LI><STRONG>CPU0 is engaged</STRONG> – CPU0 is typically the most overburdened CPU on the system and so a virtual machine whose VMQ lands on this processor will compete with all other processing on that system. It’s best if we could avoid this processor.</LI> </UL> <P>&nbsp;</P> <UL> <LI><STRONG>Only CPU0 is engaged</STRONG> – Currently all VMs are using this processor to service received network traffic. Since we have a bunch of other cores, we should use them so the VMs can get better performance.</LI> </UL> <P>&nbsp;</P> <UL> <LI><STRONG>MaxProcessors</STRONG> is configured to 8 and there are 19 processors on the system (excluding CPU0); 10 per NUMA. This means that the adapter will only assign VMQs to 8 of the available processors.</LI> </UL> <P>&nbsp;</P> <UL> <LI><STRONG>Overlapping of processors</STRONG> is not ideal - Given we’re limited to a single VMQ per virtual NIC, and VMQs cannot be reassigned to a new processor if under load, we need to minimize the chances of different adapters assigning VMQs to the same processor.</LI> </UL> <BLOCKQUOTE> <P><SUB><STRONG>Note: </STRONG>If you’re in min-of-queues mode (e.g. using LACP or address hash), the processors assigned to the NICs in the team must be the same; you’ll need to run the same commands on each pNIC and cannot split the adapters in the manner shown below.&nbsp; We do not recommend running in min-of-queues mode.&nbsp; If you are misconfigured based on the defined queue mode, you will see event 106.&nbsp; Please see <A href="#" target="_blank" rel="noopener">this KB article</A> for more information.</SUB></P> </BLOCKQUOTE> <P>&nbsp;</P> <P>To remedy the previously outlined issues, let’s configure the following VMQ to processor layout like this:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="p.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108504iF7ED877DC87F36C8/image-size/large?v=v2&amp;px=999" role="button" title="p.png" alt="p.png" /></span></P> <P>&nbsp;</P> <P>If you have Hyperthreading enabled, you’d visualize it like this:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="[EDIT] Picture updated to include all applicable processors" style="width: 955px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/109353i289FE44E5A2DDE4D/image-size/large?v=v2&amp;px=999" role="button" title="Annotation 2019-04-17 100721.jpg" alt="[EDIT] Picture updated to include all applicable processors" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">[EDIT] Picture updated to include all applicable processors</span></span></P> <P>Generally, we’d recommend assigning enough processors to max out the adapter’s available throughput.&nbsp; For example, if you have a 10 GbE NIC, you’d be best served by assigning a minimum of 4 cores to ensure that in the worst case, where an adapter can only process about 3 Gbps on each core, you’re getting the maximum out of your NICs.</P> <P>&nbsp;</P> <BLOCKQUOTE> <P><SUB><STRONG>Note:</STRONG> Some customers choose to skip the first core on the additional NUMAs.&nbsp; First, this is simpler to automate and second guarantees similar performance to all VMs regardless of which adapter services their received network traffic.&nbsp; This is a customer choice and is not required as shown by this example.</SUB></P> </BLOCKQUOTE> <P>&nbsp;</P> <P>Next, run these commands to have pNIC01 assign VMQs to processors 1 - 9</P> <PRE>Set-NetAdapterVmq -Name pNIC01 -BaseProcessorNumber 1 -MaxProcessorNumber 9 -MaxProcessors 9</PRE> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="s.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108507iBD8290EBC372FD05/image-size/large?v=v2&amp;px=999" role="button" title="s.png" alt="s.png" /></span></P> <P>&nbsp;</P> <PRE>Set-NetAdapterVmq -Name pNIC02 -BaseProcessorNumber 10 -MaxProcessorNumber 19 -MaxProcessors 10 </PRE> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="t.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108508iC9B7301ED0FA38C8/image-size/large?v=v2&amp;px=999" role="button" title="t.png" alt="t.png" /></span></P> <P>&nbsp;</P> <P><STRONG>BaseProcessorNumber</STRONG>: Numerically the lowest CPU core that can be engaged by queues on this NIC</P> <UL> <LI>pNIC01 starts at core 1</LI> <LI>pNIC02 starts at core 10</LI> </UL> <P><STRONG>MaxProcessorNumber</STRONG>: Numerically the maximum CPU core that can be engaged by queues on this NIC</P> <UL> <LI>pNIC01 ends at core 9</LI> <LI>pNIC02 ends at core 19</LI> </UL> <P><STRONG>MaxProcessors:</STRONG> The maximum number of CPU cores from the range above that can have VMQs assigned</P> <UL> <LI>pNIC01 can assign VMQs to 9 processors</LI> <LI>pNIC02 can assign VMQs to 10 processors</LI> </UL> <P>Now the configuration looks like this:</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="u.png" style="width: 633px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108535i40487D8FCD7B7218/image-dimensions/633x452?v=v2" width="633" height="452" role="button" title="u.png" alt="u.png" /></span></P> <P>&nbsp;</P> <H1>Implications of the default queue</H1> <P>VMQs are a finite resource in the hardware.&nbsp; As you’ve likely seen from some of the earlier screen shots, NICs from those time periods did not have the same amount of resources as they do today and as a result could not provide as many queues.</P> <P>&nbsp;</P> <P>By default, any virtual machine created would request a VMQ, however if the host does not have any more available to hand out, the VM will land on the default queue which is shared by all VMs that don’t have a dedicated queue. This poses two problems:</P> <P>&nbsp;</P> <UL> <LI>If VMs share a queue they’re sharing the overall bandwidth that can be generated by the processing power on the selected CPU core</LI> <LI>As VMs live migrate, they will land on a new host that may not be able to provide them the same level of performance as the old host</LI> </UL> <P>Therefore, if you have a high amount of VMs, first make sure that you optimize the placement of your VMs.&nbsp; If you’ve “over-provisioned” the number of VMs compared to the number of queues available make sure to only allocate a queue to VMs that need it; decide which virtual adapters are the high bandwidth (e.g. 3 – 6 Gbps) or require consistent bandwidth and disable VMQ on the other VMs adapters.</P> <P>&nbsp;</P> <P>To mark a virtual adapter as ineligible for a VMQ, use the <A href="#" target="_blank" rel="noopener"><EM>Set-VMNetworkAdapter</EM></A> cmdlet and –VmqWeight 0 parameter.&nbsp; VmqWeight interprets the value 0 as disabled, and any non-zero value as enabled.&nbsp; There is no difference between different non-zero values.</P> <P>&nbsp;</P> <H1>Troubleshooting</H1> <P><STRONG>Allocated Queues</STRONG></P> <P>To verify that a queue was allocated for a virtual machine use <A href="#" target="_blank" rel="noopener">Get-NetAdapterVmqQueue</A> and review the VMFriendlyName or MAC Address in the tables output.</P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="a.png" style="width: 666px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/108536iEED41E5607249C1E/image-size/large?v=v2&amp;px=999" role="button" title="a.png" alt="a.png" /></span></P> <P>&nbsp;</P> <P><STRONG>Performance Monitor</STRONG></P> <P>Using performance monitor (start &gt; run &gt; perfmon) there are few counters that are helpful and can help evaluate VMQ.&nbsp; Add the following counters under the <STRONG>Hyper-V Virtual Switch Processor</STRONG> category:</P> <UL> <LI>Number of VMQs – The number of VMQ processors that affinitize to that processor</LI> <LI>Packets from External – Packets indicated to a processor from any external NIC</LI> <LI>Packets from Internal – Packets indicated to a processor from any internal NIC, such as a vmNIC or vNIC</LI> </UL> <P><STRONG>Packets processed on the wrong processor</STRONG></P> <P>During the 2012 timeframe, we often saw packets being processed on the wrong VMQ processors and not respecting the defined processor array.&nbsp; This is largely resolved through NIC driver and firmware updates.</P> <P>&nbsp;</P> <P>Symptoms include a drastic unexplained dropped in throughput or just low throughput. This can be easily troubleshot using the counters above.</P> <H1>Summary of Requirements</H1> <P>To summarize what we’ve covered here, there are several requirements</P> <UL> <LI>Install latest drivers and firmware</LI> <LI>Processor Array engaged by default – CPU0</LI> <LI>Configure system to avoid CPU0 on non-hyperthreaded systems and CPU0 and [Edit] <STRIKE>CPU2</STRIKE>&nbsp;CPU1 on hyperthreaded systems (e.g. <STRONG>BaseProcessorNumber</STRONG> should be 1 or 2)</LI> <LI>Configure the <STRONG>MaxProcessorNumber</STRONG> to establish that an adapter cannot use a processor higher than this</LI> <LI>Configure <STRONG>MaxProcessors</STRONG> to establish how many processors out of the available list a NIC can spread VMQs across simultaneously and maximize the adapters throughput. We recommend configuring this to the number of processors in the processor array</LI> <LI>Test the workload – Leave no stone unturned</LI> </UL> <P>&nbsp;</P> <P><STRONG>Other best practices</STRONG></P> <P>These best practices should be combined with the information in the <EM>Summary of Requirements</EM> section above.</P> <UL> <LI>Use sum-of-queues to maximize the number of available VMQs</LI> <LI>Don’t span a single adapters queues across NUMA nodes</LI> <LI>If using teaming, specify Switch Independent and Hyper-V Port or Dynamic to ensure you have the maximum number of VMQs available; we recommend Hyper-V Port</LI> <LI>If you have more VMs than VMQs, make sure to define which VMs get the VMQs so you have some semblance of consistent performance</LI> <LI>Do <STRONG>NOT</STRONG> disable RSS unless directed by Microsoft support for troubleshooting</LI> </UL> <P><STRONG>Summary of Advantages</STRONG></P> <UL> <LI><STRONG>Full Utilization of the NIC</STRONG> – Given adequate processing resources the full speed of the physical adapter attached to a virtual switch can be utilized if properly configured</LI> </UL> <P><STRONG>Summary of Disadvantages</STRONG></P> <UL> <LI><STRONG>Default Queue </STRONG>– There is only one default queue – All VMs that do not get their own queue will share this queue and will experience lower performance than other VMs</LI> <LI><STRONG>Static Assignment </STRONG>– Since queues are statically assigned and cannot be moved once established, if sharing a CPU core with other “noisy” VMs, performance can be inconsistent</LI> <LI><STRONG>One VMQ per virtual NIC </STRONG>– Only one queue is assigned to each vNIC or vmNIC – a single virtual adapter can reach a maximum of 6 Gbps on a well-tuned system</LI> </UL> <P>As you can see, there are a number of complex configuration options that must be considered when deploying VMQ in order to reach the maximum throughput of your system.&nbsp; In future articles, you'll see how we improve the reliability and performance, ultimately increasing your system's VM density in Windows Server 2012 R2, 2016, and 2019.</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>Dan Cuomo</P> <P>&nbsp;</P> Fri, 24 May 2019 18:26:10 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/synthetic-accelerations-in-a-nutshell-windows-server-2012/ba-p/447792 Dan Cuomo 2019-05-24T18:26:10Z How NOT to test the Windows Time Service https://gorovian.000webhostapp.com/?exam=t5/networking-blog/how-not-to-test-the-windows-time-service/ba-p/411592 <P>Hi Everybody, <A href="#" target="_blank" rel="noopener">Dan Cuomo</A> back to talk about testing the Windows Time service for High Accuracy time.</P> <P>&nbsp;</P> <P>The Windows Time service has grown exponentially during our last couple releases.&nbsp; In “the old days” time was merely an enabler for Active Directory; Kerberos tickets required systems to be within &lt; 5 minutes of accuracy.&nbsp; If you ask your friendly neighborhood metrology expert, this is outrageous by today’s standards.&nbsp; Modern applications need time accuracy measured in the milliseconds (ms), microseconds (us), nanoseconds (ns) – There are other applications but for our areas of interest, we’ll stop at nanoseconds :smiling_face_with_smiling_eyes:</img>.</P> <P>&nbsp;</P> <P>Unfortunately, there’s a lot of misinformation about the Windows Time service.&nbsp; Due to our past requirements, it’s often assumed that Windows can’t obtain high accuracy time on a modern operating system.&nbsp; While you may not be able to consistently hit in the nanosecond range just yet, you can obtain accuracy in the order of microseconds as demonstrated <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener">here</A> and validated by NIST (see the section on <STRONG>Clock Source Stability</STRONG> where we were validated at 41 microseconds!).&nbsp; However, it’s important to use proper testing methodologies, otherwise, you might create a self-fulfilling prophecy.</P> <P>&nbsp;</P> <P>So here are a few things you <STRONG>SHOULD NOT </STRONG>do when testing the Windows Time service.</P> <H2>&nbsp;</H2> <H2>Don’t compare systems “Out of the Box”</H2> <P>I’ve heard from more than a few customers that they want to see the native time accuracy performance Windows can obtain.&nbsp; This is a bad idea for a few reasons.</P> <P>&nbsp;</P> <P><STRONG><EM>First</EM></STRONG>, Microsoft has a lot of customers.&nbsp; They range from our high-performance Azure Data Centers, to high accuracy customers in the Intelligent Edge, to your mom and dad’s computer they just bought at Best Buy.&nbsp;</P> <P>&nbsp;</P> <P>Our time service <STRONG><EM>must</EM></STRONG>&nbsp;work in all of these cases.&nbsp; You can imagine that a home computer doesn’t (and shouldn’t) synchronize nearly as much as one of the aforementioned high accuracy systems.&nbsp; This is wasteful of system resources such as CPU processing, but more importantly, power consumption considering these devices are mobile and often low powered.</P> <P>&nbsp;</P> <P>Long story short, Windows is optimized to make the most common scenarios work as best as possible – “Common” in this case means the 100’s of millions of every day users with basic network connectivity to the internet.</P> <P>&nbsp;</P> <P><STRONG><EM>Secondly</EM></STRONG>, the typical Out of the Box comparison attempts to compare Linux to Windows.&nbsp; The problem is that they downloaded and configured Linux packages (not updates) to better synchronize their system because “that’s how Linux does it.” &nbsp;Did you also download our software timestamping capability from the PowerShell gallery (<EM>Install-Module SoftwareTimestamping)</EM>?</P> <P>&nbsp;</P> <P>Needless to say, this is not an apples-to-apples comparison and if you truly want to understand what the platforms can do natively, you need to only configure what comes with a specific operating system.</P> <P>&nbsp;</P> <P><STRONG><EM>Finally</EM></STRONG>, Why?&nbsp; What other system in your datacenter do you not tweak or optimize?</P> <P>&nbsp;</P> <P>If you run Active Directory, you setup your site links, change replication intervals, or configure group policies to meet your needs.&nbsp; If you’re running Hyper-V you upgrade drivers, enable advanced features like RDMA and VMMQ, and tweak live migration settings to squeeze out the best possible performance and reliability.&nbsp; So, if you’re serious about high-accuracy time, why not put the best foot forward?</P> <P>&nbsp;</P> <P>To reach high accuracy (sub 1-second or sub 1-millisecond), you should modify your configuration using our <A href="#" target="_blank" rel="noopener">high accuracy settings</A>.&nbsp;</P> <P>&nbsp;</P> <H2>Don’t use different tools</H2> <P>I recently attended the 2019 WSTS conference in San Jose, Ca – and I’m American.&nbsp; Everyone at the conference spoke English, however it wasn’t everyone’s first language (in fact our friends from the United Kingdom would argue it wasn’t my primary language either ;)</img>).&nbsp; As a result, there were different interpretations of same information and colloquialisms that where “lost in translation.”</P> <P>&nbsp;</P> <P>Each tool is written by a developer and that tool compares the time it gets from an operating system with a reference clock.&nbsp; As a simple example, how long does that tool take to query the time on the system?&nbsp; This is up to the tool and APIs implemented and while this may seem insignificant, it’s crucial if you’re counting accuracy in the microseconds.</P> <P>&nbsp;</P> <P>If you want to compare one thing to another, you need to use the same tooling and APIs to eliminate the variability in the data; the time service is no different.&nbsp; If you use w32tm.exe to assess the accuracy of Windows, you should use w32tm.exe to assess the accuracy of other systems.&nbsp; If you have a 3<SUP>rd</SUP>-party tool, then use that tool across Windows and any other system you’re testing.</P> <P>&nbsp;</P> <P>This problem is exacerbated by the next problem…</P> <P>&nbsp;</P> <H2>Don’t test from the system you’re testing</H2> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG>“When you're good at something, you'll tell everyone. When you're great at something, they'll tell you.”</STRONG> – NFL Hall-of-Famer, Walter Payton</P> </BLOCKQUOTE> <P>As previously mentioned, Windows has an inbox component called w32tm.exe.&nbsp; You can use this tool with the <EM>/stripchart</EM> parameter to query the clock on a remote machine.&nbsp; If this returns a positive value, the remote clock is ahead of your clock by the specified offset.&nbsp; If a negative value was returned, your system is ahead of the reference clock.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="a.png" style="width: 513px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/106211iBFEA3838901D1925/image-dimensions/513x150?v=v2" width="513" height="150" role="button" title="a.png" alt="a.png" /></span></P> <P>&nbsp;</P> <P>Because of the convenience of using the inbox tool, I’ve often seen customers setup a test that looks a lot like this:<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="b.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/106213i6C00DE865344E328/image-size/medium?v=v2&amp;px=400" role="button" title="b.png" alt="b.png" /></span></P> <P>&nbsp;</P> <P>They run w32tm.exe from the computer they’re comparing against the reference clock and conclude that their system’s offset is 650 microseconds.</P> <P>&nbsp;</P> <P>Satisfied, you call success. This seems all well and good until one of your snarky co-workers walks in sipping their morning cup ‘O Joe and says, “But if the system knows it’s offset from the reference clock, why doesn’t it just correct the time?”</P> <P>&nbsp;</P> <P>As you begin to eloquently explain to your co-worker why they don’t understand, your oversight dawns on you and you realize you you’re going to need a double facepalm.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="c.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/106214i998563D7C6BD650B/image-size/medium?v=v2&amp;px=400" role="button" title="c.png" alt="c.png" /></span></P> <P>&nbsp;</P> <P>The very problem that causes your system to be offset from the reference clock is affecting the measurements you’re taking from the client.&nbsp; Put another way, you’re asking the system to self-verify its accuracy against the reference clock.&nbsp; That’s like telling the dentist, “trust me, I don’t have any cavities.”&nbsp; If that dentist doesn’t poke around your mouth for a bit, you may have more than one issue…</P> <P>&nbsp;</P> <P style="text-align: center;"><STRONG>There’s a better way</STRONG></P> <P>&nbsp;</P> <P>From the numerous stratum, to network hops, to operating system network stack latency, there’s a considerable amount of noise in the measurements.&nbsp; There are some good testing tools and methods available for assessing time accuracy available, but I can assure you most are not free – Trust me, I’ve looked (I’m also open to some suggestions in the comments ;)</img>).</P> <P>&nbsp;</P> <P>It would be better if you had another system verify your accuracy as compared to the reference clock in a way that could take into account the asymmetry in the network.&nbsp; Below you see a picture of a measurement system that simultaneously takes stripcharts of the reference clock and the system under test.&nbsp; In this scenario, you would output stripcharts to their own respective CSV file and import into excel to compare them.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="d.png" style="width: 400px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/106215iB8C46D1C0D4A39B2/image-size/medium?v=v2&amp;px=400" role="button" title="d.png" alt="d.png" /></span></P> <P><STRONG>Note:</STRONG> It’s a little more complicated than that, but that’s the gist.</P> <P>&nbsp;</P> <P>This approach has numerous benefits:</P> <UL> <LI>By putting a measurement system in between the reference clock and the system under test, you get an external opinion about your system's accuracy.</LI> <LI>This enables you to use the same tooling from the measurement system to measure different operating systems. w32tm /stripchart only requires that the node respond to NTP requests.</LI> <LI>If you place your observation system equidistant from the reference clock and the system under test, with enough samples, you can statistically minimize (not eliminate) the effect of network asymmetry.</LI> </UL> <P>While this is certainly not foolproof, this is a much-improved method of testing that begins to whittle down the variability affecting measurements.</P> <P>&nbsp;</P> <P style="text-align: center;"><STRONG>Summary</STRONG>: Listen to Walter; get a second opinion</P> <P style="text-align: center;">&nbsp;</P> <H2>Don’t use 2012 R2 or Windows 8.1 (or below)</H2> <P>When Windows Server 2012 R2 and the corresponding clients were released, the only in-market time-dependent scenario we had was Active Directory.&nbsp; As previously mentioned, this required only sub-5 minutes of accuracy. &nbsp;If you’re interested in testing Windows for time accuracy, you need to upgrade to a modern operating system.</P> <P>&nbsp;</P> <P><STRONG>*sigh*</STRONG> I fully recognize that many customers are just not able to upgrade their systems to the latest and greatest operating system.&nbsp; There are many reasons why customers may choose not to upgrade, but ultimately, we just didn’t bake in all the high accuracy goodness that allows Windows to reach sub-millisecond accuracy into 2012 R2 (or Windows 8)</img> and below.</P> <P>&nbsp;</P> <P>If you want to see what Windows can do in time accuracy, you need to use at least Windows 10 version 1607 or Windows Server 2016.&nbsp; The best experience will be in Windows 10 version 1809 and Windows Server 2019 (or whatever the latest release is when you read this).&nbsp; Our high accuracy <A href="#" target="_blank" rel="noopener">supportability guidelines</A> outline all of this for you.</P> <P>&nbsp;</P> <H2>Don’t test “a little”</H2> <P>&nbsp;</P> <P style="text-align: center;"><STRONG>I’m thinking of a number between 1 and…</STRONG></P> <P>If you want to measure something, it’s important to obtain a statistically significant sample size.&nbsp; Time accuracy is no different.&nbsp; As you’re aware, there is a variable amount of latency and asymmetry as packets move throughout your network.&nbsp; This changes constantly and can be affected by a number of network related aspects like Head-of-Line blocking, prioritization of traffic (e.g. quality of service), and the sheer number of packets moving through some switches and not others.</P> <P>&nbsp;</P> <P>As a result, we’d recommend using a large sample size.&nbsp; In a perfect world, you should take samples over a couple of days at least. Imagine you start a measurement at 8 AM in the morning, a surge of network traffic from logins and network profiles begins at 9 AM, or the backup job kicks off at 3 AM Sunday night and a maintenance window that live migrates a bunch of machines.</P> <P>&nbsp;</P> <P>…needless to say, there are so many irregularities that you know and some you probably don’t know.&nbsp; To get the best results, you should take samples over a couple of weeks to eliminate irregularities in network traffic.</P> <P>&nbsp;</P> <P style="text-align: center;"><STRONG>Summary</STRONG>: “Do or do not there is no try” - Yoda</P> <P style="text-align: center;">&nbsp;</P> <H2>Test the whole scenario</H2> <P>&nbsp;</P> <BLOCKQUOTE> <P style="text-align: left;"><STRONG>“How long is forever? Sometimes, just one second”</STRONG> – Lewis Carroll, Alice in Wonderland</P> </BLOCKQUOTE> <P>While rare, leap seconds are a fact of life. If your business requires accuracy below 1-second you MUST handle leap seconds and handle them properly.</P> <P>&nbsp;</P> <P>As discussed in some detail <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener">here</A>, leap second smearing (spreading the additional second across the entire day) is not acceptable for regulated customers like the financial services industry for two major reasons:</P> <UL> <LI>Leap second smearing is not traceable – Traceability (proof of your time accuracy back to a national reference timescale) is not possible according to the brainiacs that run the timing laboratories at NIST, NPL, etc.</LI> <LI>You’ve blown your accuracy target – Smearing works by carving up a second into smaller units and inserting those smaller units into the time scale throughout the day which means that by noon (12 PM), you’re approximately a ½ second offset from UTC.</LI> </UL> <P>If you need to keep time through a leap second, read <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Leap-Seconds-for-the-IT-Pro-What-you-need-to-know/ba-p/339811" target="_blank" rel="noopener">this</A> next.&nbsp; We also have some <A href="#" target="_blank" rel="noopener">validation guides</A> for you.</P> <P>&nbsp;</P> <P style="text-align: center;"><STRONG>Summary:</STRONG> If you need to keep accurate time through leap seconds, make sure you test them.</P> <H2>&nbsp;</H2> <H2>Don’t test things you don’t care about</H2> <P>Some anecdotal information may be interesting but if you care about accuracy, then follow the recommendations in this article and test accuracy.&nbsp; Be careful not to put too much weight into things you don’t really care about (if you do care about things listed below, then as we recommended in the last section, have at it…).&nbsp; Here are a couple of examples of “rabbit holes” I’ve seen people spend far too much time on:</P> <P>&nbsp;</P> <P><STRONG>Startup convergence: </STRONG>How quickly a system becomes accurate is <EM><STRONG>usually</STRONG> </EM>not too important for servers (I’m not aware of <STRONG>any</STRONG> client scenarios that are affected by this).&nbsp; This can be nice anecdotal information, but it should not be a primary metric you capture if your goal is to understand accuracy and stability.&nbsp; You certainly shouldn’t glean a systems overall ability because of how quickly it became accurate.</P> <P>&nbsp;</P> <P>That said, it’s certainly a pain to have to wait around for a system to become accurate prior to enabling a workload on them.&nbsp; While this could be automated, it’s certainly not ideal.</P> <P>&nbsp;</P> <P>The sweet spot is to make sure that you measure long-enough to where it does matter.&nbsp; If your workload has a time dependency, is not redundantly available, and must be running within a few minutes of a reboot, then make sure you understand convergence.&nbsp; However most modern workloads are built with availability in mind which likely means that they can wait a bit to take on the workload.</P> <P>&nbsp;</P> <P><STRONG>Bare-metal vs Virtual Machines</STRONG></P> <P>This one might be obvious, but your tests should reflect the actual profile of your target workload, which is to say, that if you’re using virtual machines, test virtual machines.&nbsp; If your systems are bare-metal, test bare-metal.</P> <P>&nbsp;</P> <P>With virtual machines, receive traffic is intended to optimize host CPU performance and as such may see a bit more variability in processing of all packets.&nbsp; This affects the observed time accuracy at a granularity of microseconds.</P> <P>&nbsp;</P> <P>Profiling can go a bit deeper than just bare-metal vs .&nbsp; Other things you might want to profile as they can vary your time accuracy are network adapter driver, firmware, and settings (e.g. Interrupt Moderation), OEM hardware (Dell vs HPE vs Lenovo vs…)…Your mileage may vary so test your scenarios!</P> <P>&nbsp;</P> <P><STRONG>Live Migration</STRONG></P> <P>Recently, I’ve heard an influx of comments about live migration.&nbsp; For most of us, live migration is a given; a basic function of a modern-day datacenter.</P> <P>&nbsp;</P> <P>The challenge with time accuracy is that virtual machines typically get time accuracy from their host.&nbsp; In turn, the virtual machines accuracy is reflective of their host.&nbsp; If Host A and Host B are not closely synchronized, then the virtual machines time will be offset when it reaches the new system.&nbsp; In addition, the virtual machine doesn’t know how long it was in transit to the new host.&nbsp; It has no point of reference for the time it was off.&nbsp; It only knows what the destination tells it when it wakes up.</P> <P>&nbsp;</P> <P>However, not everyone actually cares about this.&nbsp; <STRONG><EM>First</EM></STRONG>, there’s a risk reward with live migration that some customers and applications just don’t tolerate. I’ve even seen some customers require changes requests to be filed before moving a virtual machine to a new host (it was exhausting…but those VMs were vital to their business goals).</P> <P>&nbsp;</P> <P><STRONG><EM>Second</EM></STRONG>, as previously mentioned, many customers with high accuracy requirements are using bare-metal systems but for simplicity are testing their accuracy with virtual machines.</P> <P>The point is that if your systems don’t migrate, then you don’t need to worry about the accuracy after a live migration.&nbsp; If they do, then make sure you those scenarios.</P> <P>&nbsp;</P> <H2>Summary</H2> <P>There are a lot of things to consider when testing something as fine-grained as time accuracy.&nbsp; This is an area that most of us just take for granted and a few seconds here or there doesn’t affect our workload.&nbsp; If, however your workload is time-sensitive, make sure you properly scope out the scenarios and perform as high-fidelity tests as possible.</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>Dan Cuomo</P> Thu, 11 Apr 2019 20:55:42 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/how-not-to-test-the-windows-time-service/ba-p/411592 Dan Cuomo 2019-04-11T20:55:42Z Windows Subsystem for Linux for testing Windows 10 PTP Client https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-subsystem-for-linux-for-testing-windows-10-ptp-client/ba-p/389181 <P>Hi Folks,</P> <P>&nbsp;</P> <P>Program Manager <A href="#" target="_blank" rel="noopener">Dan Cuomo</A> here to chat about setting up one of the most important new features outlined in <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener">#10 on our Top 10 Networking features in Windows Server 2019</A>, Precision Time Protocol (PTP).</P> <P>&nbsp;</P> <P>As a quick refresher, you can visualize the benefit of PTP (IEEE 1588v2) by thinking back to the last thunderstorm you experienced.&nbsp; The further away you are from the lightning, the larger the audible delay in the thunder (you don’t see AND hear at the same time unless you’re very close to the lightning).&nbsp; This is not just the difference in speed of sound vs the speed of light, but the delay introduced by the environment such as nearby cars and buses, buildings, and many others.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="PTPthunderstorm.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100207iFB49E2E04C7816C0/image-size/large?v=v2&amp;px=999" role="button" title="PTPthunderstorm.png" alt="PTPthunderstorm.png" /></span></P> <P>&nbsp;</P> <P>In timing, latency (delay) is a killer – If you’re in the financial services, video broadcasting, gaming, or numerous other industries you’re painfully aware of this.&nbsp; As timing is distributed in the network, the accuracy and certainty of that measurement degrades like the sound of the lightning as you move further from the source.</P> <P>&nbsp;</P> <P>PTP is a new (to Windows) time synchronization protocol that helps to remove the noise and asymmetry in the network that reduces the accuracy of a traditional time synchronization protocol.&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="PTPSwitch.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100208iF045AB8A2DA74394/image-size/large?v=v2&amp;px=999" role="button" title="PTPSwitch.png" alt="PTPSwitch.png" /></span></P> <P>&nbsp;</P> <P>One of the challenges customers have is setting up and testing the Windows Client as it requires a time server that can speak PTP; you cannot use an NTP server to speak to a PTP client.&nbsp; So, in this blog, we’ll show you how to easily setup and test PTP with a linux grandmaster – All on a single physical machine!</P> <P>&nbsp;</P> <P><STRONG>Disclaimer</STRONG>: These instructions are point-in-time; at the time of writing (3/27/19), these instructions work however future changes to the various components included here might affect your success in the future.</P> <P>&nbsp;</P> <H1>Video</H1> <P>Let me first prove this works then you can use the instructions below to try it out yourself!</P> <P><LI-VIDEO vid="https://youtu.be/W58_JZLBbVw" align="center" size="small" width="200" height="150" uploading="false" thumbnail="https://i.ytimg.com/vi/W58_JZLBbVw/hqdefault.jpg" external="url"></LI-VIDEO></P> <H1>Instructions</H1> <P>These instructions reference three different locations where you will need to perform an action.</P> <P>&nbsp;</P> <P><STRONG>Windows 10 Client</STRONG>: This is a Windows 10 version 1809 or later physical host system.&nbsp; This is the only physical system needed for this exercise.</P> <P>&nbsp;</P> <P><STRONG>Windows Server 2019 VM</STRONG>: This is a virtual machine installed on the <STRONG>Windows 10 Client</STRONG> and will be the PTPv2 (IEEE 1588v2) subordinate.&nbsp; In this example we use a Windows Server 2019 system as the PTP subordinate, however a Windows 10 system can also work.&nbsp; Creating this virtual machine is outside of the scope of these instructions.</P> <P>&nbsp;</P> <P><STRONG>Ubuntu on WSL</STRONG>: This is a Windows Store app that runs the Windows Subsystem for Linux on the&nbsp;<STRONG>Windows 10 Client</STRONG> and will operate as the PTP Grandmaster.</P> <P>&nbsp;</P> <P>On the <STRONG>Windows 10 Client </STRONG>navigate to our <A href="#" target="_blank" rel="noopener">GitHub Repo</A>&nbsp;- In this example we will use the unicast configuration file in this repo.</P> <P>&nbsp;</P> <P>Click on the unicast configuration for PTPd (last one in the list shown below)<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Annotation 2019-03-31 154900.jpg" style="width: 453px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100699iCC580AA1B1FA709C/image-dimensions/453x283?v=v2" width="453" height="283" role="button" title="Annotation 2019-03-31 154900.jpg" alt="Annotation 2019-03-31 154900.jpg" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>Next, click <STRONG>Raw</STRONG> and copy the contents into notepad and save the file in your preferred location with a .conf extension. In this example, we’ve saved the file to c:\temp\PTPd\PTPdUnicast.conf.&nbsp; You will later access this file directly, from within the /mnt folder on the <STRONG>Ubuntu on WSL</STRONG> instance.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="14.jpg" style="width: 565px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100700iAF583009F9DA9DD6/image-dimensions/565x344?v=v2" width="565" height="344" role="button" title="14.jpg" alt="14.jpg" /></span></P> <P>&nbsp;</P> <P style="text-align: center;"><STRONG>Important Note: </STRONG>Please make sure that the file has a blank line at the end of the file</P> <P>&nbsp;</P> <P>Open Hyper-V Manager on your Windows 10 Client and attach the VM to a virtual switch. In this example, we use the default switch however any virtual switch can be used so long as it is not a private virtual switch.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="4.png" style="width: 174px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100248i40CE5FD1C7111737/image-dimensions/174x289?v=v2" width="174" height="289" role="button" title="4.png" alt="4.png" /></span></P> <P>&nbsp;</P> <P>In Hyper-V Manager, select the virtual machine and note the assigned IP Address. If you’re not using the default virtual switch, you may need to assign one.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="5.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100249i8CD3EDADA3FBB81B/image-size/large?v=v2&amp;px=999" role="button" title="5.png" alt="5.png" /></span></P> <P>&nbsp;</P> <P>Open the Windows Store on the <STRONG>Windows 10 Client</STRONG> and search for <EM>Linux</EM> or <EM>Ubuntu</EM> and install the Windows Subsystem for Linux application<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="6.png" style="width: 605px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100250i69EF5ECDA20F29CF/image-dimensions/605x426?v=v2" width="605" height="426" role="button" title="6.png" alt="6.png" /></span></P> <P>&nbsp;</P> <P>Once installed, select start and launch Ubuntu<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="7.png" style="width: 417px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100251i8AA992BBE338685F/image-dimensions/417x164?v=v2" width="417" height="164" role="button" title="7.png" alt="7.png" /></span></P> <P>&nbsp;</P> <P><STRONG>Ubuntu on WSL</STRONG></P> <P>&nbsp;</P> <P>First launch will take a few minutes; please be patient.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="8.png" style="width: 466px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100253i08D8262E39964F80/image-dimensions/466x186?v=v2" width="466" height="186" role="button" title="8.png" alt="8.png" /></span></P> <P>&nbsp;</P> <P>Enter a username, then a password<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="9.png" style="width: 563px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100254i167A6D11430BF65B/image-dimensions/563x130?v=v2" width="563" height="130" role="button" title="9.png" alt="9.png" /></span></P> <P>&nbsp;</P> <P>Run <EM>sudo apt-get update.&nbsp;&nbsp;</EM>This updates the packages that are on the system and will require internet access.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="10.png" style="width: 463px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100281i879B6BCA0179C007/image-dimensions/463x94?v=v2" width="463" height="94" role="button" title="10.png" alt="10.png" /></span></P> <P>&nbsp;</P> <P>Run <EM>sudo apt-get install ptpd.&nbsp;&nbsp;</EM>This installs the PTP package that will operate as the PTP grandmaster<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="11.png" style="width: 518px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100282i7B46E032EE0A3D07/image-dimensions/518x25?v=v2" width="518" height="25" role="button" title="11.png" alt="11.png" /></span></P> <P>&nbsp;</P> <P>Run<EM> ls /mnt.</EM> WSL mounts the local file system inside the Ubuntu instance.&nbsp;&nbsp;<STRONG style="font-family: inherit;"><FONT color="#0000FF">c</FONT> </STRONG><SPAN style="font-family: inherit;">indicates the drive letter that is shared between this WSL Linux system and your Windows 10 client.</SPAN><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="12.png" style="width: 345px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100283i7CB4BD8810687B03/image-dimensions/345x76?v=v2" width="345" height="76" role="button" title="12.png" alt="12.png" /></span></P> <P>&nbsp;</P> <P>Next, run <EM><EM>ls /mnt/c/temp/PTPd<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="13.png" style="width: 430px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100284i24DD7AC0CDDB4EA8/image-dimensions/430x61?v=v2" width="430" height="61" role="button" title="13.png" alt="13.png" /></span></EM></EM></P> <P>&nbsp;</P> <P>Run <EM>ifconfig</EM></P> <P><EM><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="ifconfig.png" style="width: 325px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100319i79518EE5AE1A2B39/image-dimensions/325x26?v=v2" width="325" height="26" role="button" title="ifconfig.png" alt="ifconfig.png" /></span></EM></P> <P>&nbsp;</P> <P>Identify the interface on the same subnet as your <STRONG><STRONG>Windows Server 2019 VM<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="interface.png" style="width: 524px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100320iAE268B7A6891C6D1/image-dimensions/524x69?v=v2" width="524" height="69" role="button" title="interface.png" alt="interface.png" /></span></STRONG></STRONG></P> <P>&nbsp;</P> <P><STRONG>Windows 10 Client</STRONG></P> <P>Modify the line <EM>ptpengine:interface = eth0</EM> to use the interface identified in the last step<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="a.png" style="width: 365px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100321i43E51485A8287EBF/image-dimensions/365x292?v=v2" width="365" height="292" role="button" title="a.png" alt="a.png" /></span></P> <P>&nbsp;</P> <P>Next modify the <EM>ptpengine:unicast_destinations</EM> value with the IP address of the <STRONG><STRONG>Windows Server 2019 VM<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="b.png" style="width: 498px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100322iD603CA6ABF9D1CDA/image-dimensions/498x99?v=v2" width="498" height="99" role="button" title="b.png" alt="b.png" /></span></STRONG></STRONG></P> <P>&nbsp;</P> <P><STRONG>Ubuntu on WSL</STRONG></P> <P>Run <EM>sudo ptpd -c /mnt/c/temp/PTPd/PTPdUnicast.conf</EM> to start the PTPd grandmaster.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="c.png" style="width: 611px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100323i43D8D18EF162582E/image-dimensions/611x25?v=v2" width="611" height="25" role="button" title="c.png" alt="c.png" /></span></P> <P>&nbsp;</P> <P>Next, run <EM>ps -aux | grep ptpd</EM> to identify the PID of the PTPd.&nbsp; If you want to restart PTPd, you use this PID to kill the process.</P> <P>&nbsp;</P> <P><STRONG>Windows Server 2019 VM</STRONG></P> <P>Modify the registry to Configure PTP. This step also disables the VMIC and NTP Client in order to guarantee we’re getting time using the correct provider.</P> <P>&nbsp;</P> <P><FONT color="#0000FF">This </FONT>is the IP from <STRONG>Ubuntu on WSL </STRONG>(eth2 in the example above)</P> <TABLE> <TBODY> <TR> <TD width="113"> <P><STRONG>Root Key</STRONG></P> </TD> <TD colspan="2" width="509"> <P>HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\PtpClient</P> </TD> </TR> <TR> <TD width="113"> <P><STRONG>Type</STRONG></P> </TD> <TD width="240"> <P><STRONG>Name</STRONG></P> </TD> <TD width="269"> <P><STRONG>Value</STRONG></P> </TD> </TR> <TR> <TD width="113"> <P>REG_SZ</P> </TD> <TD width="240"> <P>PtpMasters</P> </TD> <TD width="269"> <P><FONT color="#0000FF">172.17.75.17</FONT></P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>Enabled</P> </TD> <TD width="269"> <P>1</P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>InputProvider</P> </TD> <TD width="269"> <P>1</P> </TD> </TR> <TR> <TD width="113"> <P>REG_SZ</P> </TD> <TD width="240"> <P>DllName</P> </TD> <TD width="269"> <P>"c:\windows\system32\ptpprov.dll"</P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>DelayPollInterval</P> </TD> <TD width="269"> <P>0x3e80</P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>AnnounceInterval</P> </TD> <TD width="269"> <P>0x0fa0</P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>EnableMulticastRx</P> </TD> <TD width="269"> <P>0</P> </TD> </TR> <TR> <TD width="113"> <P><STRONG>Root Key</STRONG></P> </TD> <TD colspan="2" width="509"> <P>HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient</P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>Enabled</P> </TD> <TD width="269"> <P>0</P> </TD> </TR> <TR> <TD width="113"> <P><STRONG>Root Key</STRONG></P> </TD> <TD colspan="2" width="509"> <P>HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider</P> </TD> </TR> <TR> <TD width="113"> <P>REG_DWORD</P> </TD> <TD width="240"> <P>Enabled</P> </TD> <TD width="269"> <P>0</P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>Next, restart the Windows Time Service and open the PTP ports in the Windows Firewall using an elevated PowerShell prompt.</P> <BLOCKQUOTE> <P><EM style="font-family: inherit;">New-NetFirewallRule -DisplayName 'PTP-319' -Name 'PTP-319' -LocalPort 319 -Direction Inbound -Action Allow<EM> -Protocol UDP</EM></EM></P> <P dir="ltr">&nbsp;</P> <P dir="ltr"><EM>New-NetFirewallRule -DisplayName 'PTP-320' -Name 'PTP-320' -LocalPort 320 -Direction Inbound -Action Allow -Protocol UDP</EM></P> </BLOCKQUOTE> <P dir="ltr"><SPAN style="font-family: inherit;">Install and Launch Wireshark.&nbsp;&nbsp;</SPAN>Start a capture and filter using the following filter: <EM>udp.<EM>port in {319 320}<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="d.png" style="width: 366px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100324i0B85C797DD6DEAA0/image-dimensions/366x203?v=v2" width="366" height="203" role="button" title="d.png" alt="d.png" /></span></EM></EM></P> <P>&nbsp;</P> <P>You should see <EM>Announce, Sync, </EM>and <EM>Delay_Req, Delay_Resp</EM> messages.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="e.png" style="width: 891px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100325i9FA422E2538863A3/image-size/large?v=v2&amp;px=999" role="button" title="e.png" alt="e.png" /></span></P> <P>&nbsp;</P> <BLOCKQUOTE> <P><STRONG>Note</STRONG>: Be patient :smiling_face_with_smiling_eyes:</img> You should see Announce and Sync messages immediately however you may need to wait a few minutes to see the Delay_Req and Delay_Resp.&nbsp; While you’re waiting, you can go on to the next step.</P> </BLOCKQUOTE> <P>&nbsp;</P> <P><SPAN style="font-family: inherit;">Open the Event Viewer &gt; </SPAN><STRONG style="font-family: inherit;">Applications and Services</STRONG><SPAN style="font-family: inherit;"> &gt; </SPAN><STRONG style="font-family: inherit;">Microsoft </STRONG><SPAN style="font-family: inherit;">&gt; </SPAN><STRONG style="font-family: inherit;">Windows </STRONG><SPAN style="font-family: inherit;">&gt;</SPAN><STRONG style="font-family: inherit;"> Time-Service-PTP-Provider</STRONG><SPAN style="font-family: inherit;"> &gt; </SPAN><STRONG style="font-family: inherit;">PTP-Operational.&nbsp;&nbsp;</STRONG><SPAN style="font-family: inherit;">First verify you see event 512 which will occur once announce messages are received.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="f.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100327i8487B99EB39A669C/image-size/large?v=v2&amp;px=999" role="button" title="f.png" alt="f.png" /></span></P> <P>&nbsp;</P> <P>Next verify that you receive event 513 which indicates that you have successfully selected the PTP Master as the source of time.<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="g.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100328i794AE677934C13EE/image-size/large?v=v2&amp;px=999" role="button" title="g.png" alt="g.png" /></span></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>That’s it!&nbsp; Now that you’re system has chosen the PTP master, you’re synchronizing time.&nbsp; You can further verify this by running <EM>w32tm /query /status /verbose</EM> to verify that the last synchronization was successful.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="h.png" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100329i677F0108C5BDF0F3/image-size/large?v=v2&amp;px=999" role="button" title="h.png" alt="h.png" /></span></P> <P>&nbsp;</P> <P>If you’re trying to test out PTP, you can get your feet wet using the Windows Subsystem for Linux which is a nifty little tool that enables Linux to run natively on Windows.&nbsp; Obviously this approach will not yield high accuracy however if you’re looking to easily demonstrate the functionality, this might be the easiest approach for you!</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>Dan Cuomo</P> Mon, 13 Jul 2020 17:00:54 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-subsystem-for-linux-for-testing-windows-10-ptp-client/ba-p/389181 Dan Cuomo 2020-07-13T17:00:54Z Introducing: Kubernetes Overlay Networking for Windows https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-kubernetes-overlay-networking-for-windows/ba-p/363082 <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">With the release of Kubernetes v1.14, Windows server nodes are officially <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">declared</A> “stable”. But how is network connectivity provided between the pods and services which consist a modern application? Whenever a DevOps team deploys a new Kubernetes (K8s) cluster or adds a Windows node to an existing cluster, they want networking to <EM style="box-sizing: border-box;">just work</EM> with the equivalent or even better network management capabilities on containers than on existing infrastructure. Windows Server 2019 now includes <EM style="box-sizing: border-box;">a simpler</EM> and <EM style="box-sizing: border-box;">more scalable</EM> <EM style="box-sizing: border-box;">overlay networking solution for Kubernetes clusters via Windows update </EM><EM style="box-sizing: border-box;"><A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">KB4489899</A>,</EM> <EM style="box-sizing: border-box;">including integration</EM> with <EM style="box-sizing: border-box;">the latest release of the <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">Flannel</A> network control-plane, CNI plugins, and <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">kube-proxy</A>.</EM></P> <H2 style="box-sizing: border-box; color: inherit; font-family: inherit; font-size: 24px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 1.2; margin-bottom: 12px; margin-top: 24px; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;">Why overlay networking?</H2> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Overlay networking uses encapsulation to create a virtual network on top of the existing physical network without requiring any configuration changes to the physical network infrastructure. Overlay networking on Windows containers brings the following benefits:</P> <UL style="box-sizing: border-box; clear: left; color: #333333; font-family: &amp;quot; segoeui&amp;quot;,&amp;quot;lato&amp;quot;,&amp;quot;helvetica neue&amp;quot;,helvetica,arial,sans-serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; list-style-image: none; list-style-position: outside; list-style-type: disc; margin-bottom: 12px; margin-top: 0px; orphans: 2; padding-left: 2.5em; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;"> <LI style="box-sizing: border-box; font-family: &amp;quot;"><STRONG style="box-sizing: border-box; font-weight: bold;">Decoupled container networking:</STRONG> Logical separation of container networks from the underlay networks used to connect the K8s nodes in the cluster</LI> <LI style="box-sizing: border-box; font-family: &amp;quot;"><STRONG style="box-sizing: border-box; font-weight: bold;">Simplified management:</STRONG> Improved IP Address Management (IPAM) simplicity and support for IP re-use between different applications and namespaces</LI> <LI style="box-sizing: border-box; font-family: &amp;quot;"><STRONG style="box-sizing: border-box; font-weight: bold;">Scalability:</STRONG> Reduced stress on forwarding tables of host vSwitches (container MACs need not be learned) and reduced burden on control path components such as Host Networking Service (HNS) on a per-container basis</LI> <LI style="box-sizing: border-box; font-family: &amp;quot;"><STRONG style="box-sizing: border-box; font-weight: bold;">Better network security</STRONG>: Network isolation provided by VXLAN encapsulation using Virtual Network Identifier (VNI) – similar to VLANs – to segregate network traffic</LI> </UL> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;"><FONT size="4" style="box-sizing: border-box;"><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Figure 1: Overlay for containers" style="width: 808px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/100302iA949F8A130209F6E/image-size/large?v=v2&amp;px=999" role="button" title="overlay2.png" alt="Figure 1: Overlay for containers" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Figure 1: Overlay for containers</span></span></FONT></P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Overlay networks work by encapsulating networks packets with (outer) packets to form a new network topology independent of the underlying network. This allows for simpler communication paths between entities which were originally out of scope. By tunneling network subnets between individual hosts, it allows containers to communicate with each other as if they were on the same machine, thereby creating one network that spans multiple hosts.</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Without overlays, network administrators and other users are faced with cumbersome networking requirements on their underlying infrastructure when trying to adopt Kubernetes on Windows. This includes having L2 adjacency between container hosts/nodes, route tables in other networking modes (l2bridge/l2tunnel) falling out of sync as clusters grow in size, or network glitches on transparent networks waiting for switches to learn MAC addresses of sometimes short-lived containers.</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Now, with improved overlay networking support for containers, these constraints no longer apply, meaning users have a elastic way to deploy Kubernetes on Windows which is more agnostic and decoupled from the underlying infrastructure and its network configuration.</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Another particular source of excitement is that we’ve also revised the platform overlay networking design specifically for multi-node clustering scenarios in order to improve scalability as nodes and subnets in a cluster grow.</P> <H2 style="box-sizing: border-box; color: inherit; font-family: inherit; font-size: 24px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 1.2; margin-bottom: 12px; margin-top: 24px; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;">How does it work?</H2> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">In overlay networks, L3 IP connectivity between hosts is all that is required for containers scheduled on machines to interact with each other as if they had direct L2 connectivity (or in other words: lived on the same machine). Conceptually, this is achieved by encapsulating network packets coming from containers with an outer header. Each machine also programs HNS “RemoteSubnet” policies so that each node knows which container network is assigned to whom.</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Here is how the updated workflow using “RemoteSubnets” looks like on Windows Server 2019:</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;"><LI-VIDEO width="200" align="center" size="small" height="113" vid="https://www.youtube.com/watch?v=qY_84rrYYYY" uploading="false" thumbnail="https://i.ytimg.com/vi/qY_84rrYYYY/hqdefault.jpg" external="url"></LI-VIDEO></P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Here is an short explanation of some of the fields used in the video:</P> <UL> <LI style="text-align: left; color: #333333; text-transform: none; line-height: 1.7142; text-indent: 0px; letter-spacing: normal; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; text-decoration: none; word-spacing: 0px; white-space: normal; box-sizing: border-box; orphans: 2; -webkit-text-stroke-width: 0px;"><STRONG>"VFP":</STRONG> <A href="#" target="_self">Virtual Filtering Platform&nbsp;</A>is the vSwitch extension policy engine that allows programming which operations network packets undergo (e.g. encapsulation, NAT, ACLs, etc.)</LI> <LI style="text-align: left; color: #333333; text-transform: none; line-height: 1.7142; text-indent: 0px; letter-spacing: normal; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; text-decoration: none; word-spacing: 0px; white-space: normal; box-sizing: border-box; orphans: 2; -webkit-text-stroke-width: 0px;"><STRONG>"DR vNIC":</STRONG> The vNIC of the DR (distributed router) on a machine; this is also the container gateway</LI> <LI style="text-align: left; color: #333333; text-transform: none; line-height: 1.7142; text-indent: 0px; letter-spacing: normal; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; text-decoration: none; word-spacing: 0px; white-space: normal; box-sizing: border-box; orphans: 2; -webkit-text-stroke-width: 0px;"><STRONG>"vxlan0":</STRONG> The name of the overlay HNS network</LI> </UL> <H2 style="box-sizing: border-box; color: inherit; font-family: inherit; font-size: 24px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 1.2; margin-bottom: 12px; margin-top: 24px; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;">I want to try it out!</H2> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">Great! Building on top of the platform enrichment's described above, we’re also incredibly excited to announce Windows support for the popular open-source <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">Flannel</A> CNI plugin in overlay network mode using <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">VXLAN</A> encapsulation. This is the recommended way to get started with overlay networking on Kubernetes as it offers the simplest management experience available on Windows today.</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">The Kubernetes getting started guide has been updated with everything that you need. Here is how you can try it out yourself:</P> <OL style="box-sizing: border-box; color: #333333; font-family: &amp;quot; segoeui&amp;quot;,&amp;quot;lato&amp;quot;,&amp;quot;helvetica neue&amp;quot;,helvetica,arial,sans-serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; margin-bottom: 12px; margin-top: 0px; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;"> <LI style="box-sizing: border-box; font-family: &amp;quot;">Install Windows Server 2019 (or higher) with <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">KB4489899</A></LI> <LI style="box-sizing: border-box; font-family: &amp;quot;">Download Kube-proxy binaries from <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">v1.14</A> (or higher) for Windows</LI> <LI style="box-sizing: border-box; font-family: &amp;quot;">Download <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">Flannel v0.11.0</A> (or above)</LI> <LI style="box-sizing: border-box; font-family: &amp;quot;">Follow instructions in the <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_blank" rel="noopener">Getting Started Guide</A></LI> </OL> <H2 style="box-sizing: border-box; color: inherit; font-family: inherit; font-size: 24px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 1.2; margin-bottom: 12px; margin-top: 24px; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;">When should I use overlay vs. l2bridge networking?</H2> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">In short, the answer to this question really depends on a user’s goals and requirements. There are many reasons why attaching containers to an underlay network via a network mode like “l2bridge” may be desired. For example, users may wish for containers be able to talk to an existing on-premise service, rather than being isolated from the underlying network. One may also wish to fully onboard containers onto existing SDN features such as network security groups (NSG), which may not be possible otherwise. In performance-sensitive scenarios users should also keep in mind that overlay networks incur a performance hit, as nodes need to setup tunnels between hosts, which reduces available MTUs, and the data path need additional time and CPU cycles to encapsulate/decapsulate packets.</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">The reasons why a user may prefer overlays are largely covered above already; users <SPAN style="background-color: #ffffff; box-sizing: border-box; color: #333333; cursor: text; display: inline; float: none; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;">deploying using Flannel should also consult the <A style="background-color: transparent; box-sizing: border-box; color: #146cac; text-decoration: underline;" href="#" target="_self">Flannel backend docs</A> for additional guidance.</SPAN></P> <P style="box-sizing: border-box; color: #333333; font-family: inherit; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 300; letter-spacing: normal; line-height: 1.7142; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; margin: 0px;">&nbsp;</P> <H2>What’s next?</H2> <P>Since overlay networking for Kubernetes has just launched, we are working on incremental improvements as well as overcoming a <A href="#" target="_blank" rel="noopener">few limitations</A> to make it more customizable and useful. Here is a short teaser of what’s next:</P> <OL> <LI>Ability to customize and set an arbitrary VNID in the <A href="#" target="_blank" rel="noopener">Flannel backend configuration</A></LI> <LI>Improving inter-node overlay traffic performance (e.g. by removing extra encapsulation steps, see steps 7 and 9 in the video)</LI> <LI>Accelerate load-balancing performance for services using Direct Server Return (DSR)</LI> <LI>Stabilize and battle-harden overlay to graduate it from alpha to stable</LI> </OL> <P>Finally, since overlay networking for Kubernetes on Windows is brand new, we’d love to hear any feedback in trying it out at <A href="#" target="_blank" rel="noopener">SIG-Windows</A> or in the comments below!</P> <P>&nbsp;</P> <P>Thanks for reading,</P> <P>David Schott</P> <P>&nbsp;</P> <P>*Special thanks to Kalya <FONT style="background-color: #ffffff;">Subramanian</FONT> &amp; Pradip Dhara for designing and implementing overlay networking for Windows containers, as well as providing materials to help create content for this blog!</P> Tue, 02 Apr 2019 16:39:35 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-kubernetes-overlay-networking-for-windows/ba-p/363082 David Schott 2019-04-02T16:39:35Z Microsoft SDN patch available: Issue where SDN REST service stops working after installing updates https://gorovian.000webhostapp.com/?exam=t5/networking-blog/microsoft-sdn-patch-available-issue-where-sdn-rest-service-stops/ba-p/357828 <P><STRONG>Summary</STRONG></P> <P>Recently, a few SDN customers have hit an issue where after updating their SDN environments with patches, SDN stops working. This issue has been fixed in the latest update (<A href="#" target="_blank" rel="noopener">KB4487006</A>). All customers are strongly recommended to this update to this KB if they have already encountered this issue, or as a part of their next regular patch cycle.</P> <P>&nbsp;</P> <P><STRONG>Issue Details</STRONG></P> <P><EM>Applicable to</EM>: Windows Server 2016 customers who have deployed SDN</P> <P>&nbsp;</P> <P><EM>When will the issue occur</EM>: Any SDN customers who were on KB4343884, KB4457131, KB4457127, KB4462917 and have upgraded to any newer patches will no longer be able to communicate with their network controller.&nbsp; This will be visible as errors when using Network Controller PowerShell commands, or Network Controller errors in SC VMM.</P> <P>&nbsp;</P> <P><EM>How to identify that you have hit this issue</EM>: If you have hit this issue, the sdnapi microservice replica state keeps moving from "<EM>Primary</EM>" to "<EM>Down</EM>" to "<EM>InBuild</EM>". The long term/steady state should be "Ready". To get the status of the sdnapi service, run the following Powershell command on any of the Network Controller VMs: <EM>Get-NetworkControllerReplica -ServiceTypeName "ApiService"</EM></P> <P>&nbsp;</P> <P>If you look at Network Controller logs (located at C:\Windows\tracing\SDNDiagnostics\logs), you will see a pattern similar to the below:</P> <P><EM>SDNAPI... Primary Recovery Failed: System.ArgumentException: Unable to deserialize unknown property "storage_Epoch" in class "Microsoft.Windows.Networking.NetworkController.EventCoordinator.VirtualInterfaceState".</EM></P> <P>&nbsp;</P> <P><EM>Impact of the issue</EM>: Communication between customer tenant VMs will stop working if any of the VMs are moved to a different host or if the host goes down. Management and configuration of the fabric or tenant networks will be unavailable.</P> <P>&nbsp;</P> <P><EM>Resolution</EM>: The issue has been fixed in the latest Microsoft update (<A href="#" target="_blank" rel="noopener">KB4487006</A>). All customers are strongly recommended to move to the latest updates.&nbsp;&nbsp; After installing this update, no further steps are required to resolve this issue.</P> Wed, 27 Feb 2019 22:45:06 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/microsoft-sdn-patch-available-issue-where-sdn-rest-service-stops/ba-p/357828 AnirbanPaul 2019-02-27T22:45:06Z S2D Networking Best Practices @ MVP Days https://gorovian.000webhostapp.com/?exam=t5/networking-blog/s2d-networking-best-practices-mvp-days/ba-p/354282 <P>Hi Folks,</P> <P>&nbsp;</P> <P><A href="#" target="_self">Dan</A>&nbsp;here with a quick post to let you know about a recent presentation given at Storage Spaces Direct MVP Days.&nbsp; During this session, I discussed some of the requirements and best practices for RDMA, and also covered (very quickly) some of the synthetic best practices.</P> <P>&nbsp;</P> <P>As a quick refresher:</P> <UL> <LI>RDMA: Network traffic is offloaded to the NIC</LI> <LI>Synthetic: Network traffic is traverses the OS network stack and is processed by the CPUs requiring vRSS, VMQ, VMMQ, and now Dynamic VMMQ</LI> </UL> <P>Take a look at the video and check out the <A href="#" target="_self">rest of the playlist!</A></P> <P>&nbsp;</P> <P>Thanks,<BR />Dan</P> <P>&nbsp;</P> Wed, 20 Feb 2019 22:22:42 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/s2d-networking-best-practices-mvp-days/ba-p/354282 Dan Cuomo 2019-02-20T22:22:42Z We've moved! https://gorovian.000webhostapp.com/?exam=t5/networking-blog/we-ve-moved/ba-p/353420 <P>This is the new home of the Microsoft Windows Core Networking team blog!&nbsp; Congratulations on your (hopefully) redirected or short-linked journey from TechNet.</P> <P>&nbsp;</P> <P>For simplicity you can always find this blog by navigating to <STRONG><A href="#" target="_self">https://aka.ms/MSFTNetworkBlog</A></STRONG></P> <P>&nbsp;</P> <P>As mentioned on our TechNet blog, We'll aim to post updates on Wednesday when we have them.&nbsp; Please also consider following&nbsp;<STRONG><A href="#" target="_blank" rel="noopener">@Microsoft_SDN</A></STRONG> on Twitter!</P> <P>&nbsp;</P> <P>For the Core Networking Team,</P> <P>~<A href="#" target="_self">Dan Cuomo</A></P> <P>&nbsp;</P> Tue, 19 Feb 2019 17:52:06 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/we-ve-moved/ba-p/353420 Dan Cuomo 2019-02-19T17:52:06Z Networking in Red Hat OpenShift for Windows https://gorovian.000webhostapp.com/?exam=t5/networking-blog/networking-in-red-hat-openshift-for-windows/ba-p/339825 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Dec 06, 2018 </STRONG> <BR /> Hello again, <BR /> <BR /> Today we will be drilling into a more complex topic following the introduction to Red Hat OpenShift for Windows on premises two weeks ago. We will expand into the networking layer of the architecture that we have chosen for the current developer previews. <BR /> <BR /> You may ask yourself "Why do I care about how networking works?" <BR /> The obvious answer would be "Without it your container cannot listen or talk much to others." <BR /> What do I mean by that; networking is the backbone of any IT infrastructure and container deployments are no different from that. The various networking components allow communication of containers, nodes, pods, clusters amongst each other and the outside world. <BR /> <BR /> As a DevOps you will need to have a core understanding of the networking infrastructure pieces that are deployed in your container infrastructure and how they interact, be it bare-metal, VMs on a virtualization host or in one of the many cloud services provided so you can tailor the network setup to your needs. <BR /> <H3> Terminology </H3> <BR /> First let's cover a few buzzwords, TLAs (Three letter acronyms) and other complex things so we are all on the same page <BR /> <TABLE> <TBODY><TR> <TD> Terminology </TD> <TD> Description </TD> </TR> <TR> <TD> CNI </TD> <TD> Container Networking Interface, a specification of a standardized interface defining the container endpoint and its interaction with the node the container runs on. </TD> </TR> <TR> <TD> Docker </TD> <TD> A popular container runtime. </TD> </TR> <TR> <TD> vSwitch </TD> <TD> Virtual Switch, the central component in container networking. Every container host has one. It serves up the basic connectivity for each container endpoint. On the Linux side it resembles somewhat to a Linux Bridge. </TD> </TR> <TR> <TD> NAT </TD> <TD> Network Address Translation. A way to isolate private IP address spaces across multiple hosts and nodes behind a public IP Address space </TD> </TR> <TR> <TD> Pod </TD> <TD> the smallest atomic unit in a Kubernetes Cluster. A Pod can host one or more containers. All Containers in a pod share the same IP address </TD> </TR> <TR> <TD> Node </TD> <TD> An infrastructure component hosting one or more pods. </TD> </TR> <TR> <TD> Cluster </TD> <TD> An infrastructure component comprised of multiple nodes. </TD> </TR> <TR> <TD> HNS </TD> <TD> Host Network Service, a windows component interacting with the networking aspects of the Windows container infrastructure </TD> </TR> <TR> <TD> HCS </TD> <TD> Host Compute Service, a Windows component supporting the interactions of the container runtime with the rest of the operating system </TD> </TR> <TR> <TD> OVN </TD> <TD> Open Virtual Network. OVN provides network virtualization to containers. In the "overlay" mode, OVN can create a logical network amongst containers running on multiple hosts. In this mode, OVN programs the Open vSwitch instances running inside your hosts. These hosts can be bare-metal machines or vanilla VMs. OVN uses two data stores the Northbound (OVN-NB) and the Southbound&nbsp; (OVN-SB) data store. <BR /> <EM> ovn-northbound </EM> <BR /> <UL> <BR /> <LI> OpenStack/CMS integration point </LI> <BR /> <LI> High-level, desired state <BR /> <UL> <BR /> <LI> Logical ports -&gt; logical switches -&gt; logical routers </LI> <BR /> </UL> <BR /> </LI> <BR /> </UL> <BR /> <EM> ovn-southbound </EM> <BR /> <UL> <BR /> <LI> Run-time state </LI> <BR /> <LI> Location of logical ports </LI> <BR /> <LI> Location of physical endpoints </LI> <BR /> <LI> Logical pipeline generated based on configured and run-time state </LI> <BR /> </UL> <BR /> </TD> </TR> <TR> <TD> OVS </TD> <TD> Open Virtual Switch. Open vSwitch is well suited to function as a virtual switch in VM environments. In addition to exposing standard control and visibility interfaces to the virtual networking layer, it was designed to support distribution across multiple physical servers. </TD> </TR> </TBODY></TABLE> <BR /> Here is how all these components fit into the architecture on the Windows worker node. I will talk more about them through out the post. <BR /> <BR /> [caption id="attachment_7815" align="aligncenter" width="258"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75292iFC22D86C96F4F132" /> OpenShift for Windows Networking components[/caption] <BR /> <BR /> OK, now that we are on the same page let's dive in. <BR /> <H3> Setup </H3> <BR /> To recap from the last post, we will have a Linux Red Hat OpenShift Master node which also serves as the Kubernetes Master and a Windows Server Core Worker node which is joined to the Master. The deployment will also use the Docker container runtime on both the Linux and the Windows Node to instantiate and execute the containers. <BR /> You can deploy the nodes in one VM host, across multiple VM hosts, bare metal and also deploy more than two nodes in this environment. For the purpose of this discussion we have deployed a separate VM host and will use it to host both the Linux and the Windows Node. <BR /> Next lets dig into the networking and how the networks are created and how the traffic flows. <BR /> <H3> Networking Architecture </H3> <BR /> The image below shows the networking architecture in more detail and zooms into the above picture both on the Linux node and the Windows node. <BR /> Looking at the diagram below we can see that there are several components making up the networking layer <BR /> <BR /> [caption id="attachment_7835" align="aligncenter" width="879"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75293iCE60222BD49D7B43" /> OpenShift for Windows Networking Architecture[/caption] <BR /> <BR /> The components can be grouped into several groups: <BR /> <UL> <BR /> <LI> Parts which are Open Source components (light orange) </LI> <BR /> <LI> Parts which are in the core Windows Operating System (bright blue). </LI> <BR /> <LI> Parts which are Open Source and Microsoft made specific changes to the code and shared them with the community (light blue). </LI> <BR /> </UL> <BR /> On the Linux side Open Source Components are the container runtime like the Docker Engine, Kubernetes components such as <BR /> <UL> <BR /> <LI> kube-proxy - (Kubernetes network proxy) which runs on each node and reflects services as defined in the Kubernetes API on each node for traffic forwarding across a set of backends. </LI> <BR /> <LI> kubelet -&nbsp;is the primary “node agent” that runs on each node. The kubelet works by reading a PodSpec object which is a YAML or JSON document that describes a pod. </LI> <BR /> <LI> To find out more about Kubernetes components on Linux check the Kubernetes documentation <A href="#" target="_blank"> here </A> . </LI> <BR /> </UL> <BR /> On the Windows side some of these components like the kube-proxy and the kubelet have been enhanced by Microsoft to work with the Microsoft networking components such the Host Compute Service (HCS) and the Host Network Service (HNS). These changes are made to allow the interoperability with Windows core services and also the abstraction of the differences in the underlying architecture. <BR /> <BR /> On the Windows side some of these components like the kube-proxy and the kubelet have been enhanced by Microsoft to work with the Microsoft networking components such the Host Compute Service (HCS) and the Host Network Service (HNS). These changes are made to allow the interoperability with Windows core services and the abstraction of the differences in the underlying architecture. <BR /> <BR /> One of the differences between Linux Nodes and Windows Nodes in this system is the way the nodes are joined to the Kubernetes cluster. In Linux you would use a command like <BR /> kubeadm join 10.127.132.215:6443 --token &lt;token&gt; --discovery-token-ca-cert-hash &lt;cert hash&gt; <BR /> <BR /> On Windows where the kubeadm command is not available the join is handled by the Host Compute Service when the resource is created. <BR /> <BR /> The key takeaway of the discussion here is that overall the underlying architectural differences between Linux and Windows are abstracted and the process of setting up Kubernetes for Windows and managing the networking components of the environment is going to be straight forward and mostly familiar if you have done it on Linux before. <BR /> Also since Red Hat OpenShift calls into Kubernetes the administrative experience will be uniform across Windows and Linux Nodes. <BR /> That being said, be what we are discussing today is the architecture of the currently available developer preview. Microsoft and Red Hat are working to completed work to integrate the Windows CNI into the flow to replace OVN/OVS. We will keep the support for OVN/OVS and also add other CNI plugins as we progress but will switch to Windows CNI during the first half of 2019. So be on the lookout for an update on that. <BR /> <BR /> To say it with a famous cartoon character of my childhood "That's all folks!" <BR /> <BR /> Thanks for reading this far and see you next time. <BR /> <BR /> Mike Kostersitz <BR /> <BR /> P.S.: If this post was too basic or too high-level. Stay tuned for a deeper dive into Windows Container Networking Architecture and troubleshooting common issues coming soon to this blog near you. <BR /> <BR /> Editors Note: Fixed a typo </BODY></HTML> Thu, 14 Feb 2019 18:12:13 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/networking-in-red-hat-openshift-for-windows/ba-p/339825 mkostersitz 2019-02-14T18:12:13Z Turkey Day Mailbag https://gorovian.000webhostapp.com/?exam=t5/networking-blog/turkey-day-mailbag/ba-p/339822 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Nov 21, 2018 </STRONG> <BR /> Hello Networking Enthusiasts - Tomorrow, the US will celebrate Thanksgiving and since we're so close to a holiday we decided to keep this week's blog fairly simple and answer some common questions and information we've seen over the last few months. <BR /> <BR /> If you have follow-up questions you'd like answered (or more details on what's below), hit us up on Twitter @ <A href="#" title="https://twitter.com/Microsoft_SDN" target="_blank"> Microsoft SDN </A> ! <BR /> <H2> <STRONG> RDMA and HCI </STRONG> </H2> <BR /> Q. Network traffic from Live Migrations takes valuable CPU cycles from my tenant VMs.&nbsp;How can I reduce the impact of a live migration for tenants,&nbsp;increase the number of live migrations I&nbsp;can perform, and/or increase the speed of the live migrations? <BR /> <BR /> Answer from RDMA PM, <A href="#" title="https://twitter.com/Dan2_2023" target="_blank"> Dan Cuomo </A> : <BR /> <P> Although not the default option, SMB&nbsp;can be selected as the live migration mechanism.&nbsp; If selected, SMB can use RDMA under the hood (in this context, known as SMBDirect), which avoids the need to process the GB's (yes, Gigabytes not bits)&nbsp;of network traffic produced from the live migration (e.g VM Memory or VHDX Storage). </P> <BR /> <P> RDMA by-passes the host operating system and removes the&nbsp;processing burden of the live migrations.&nbsp; Since host networking is most commonly constrained by host CPU spreading (remember your VMs are competing for access to the same cores processing network traffic), RDMA eases the effect of the live migration on VMs on the same host as they can now continue to focus on the VMs CPU scheduling needs. </P> <BR /> <P> The net effect is an&nbsp;increase&nbsp;to the number of live migrations you can perform at once because the CPU is no longer the bottleneck for the network or affecting your tenant VMs. </P> <BR /> <BR /> <H2> <STRONG> Software Defined Networking </STRONG> </H2> <BR /> <DIV> Q. How do I get support deploying Software Define Networking? </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> Answer from SDN PM, <A href="#" title="https://twitter.com/Microsoft_SDN" target="_blank"> Schumann Ge </A> </DIV> <BR /> <DIV> </DIV> <BR /> <DIV> There are a ton of resources available and we'd recommend you'd start with our documentation <A href="#" title="https://docs.microsoft.com/en-us/windows-server/networking/sdn/" target="_blank"> here </A> .&nbsp; However if you'd like to speak to an expert, our field engineers would be glad to assist.&nbsp; Contact them at SDNBlackbelt@Microsoft.com or hit-up <A href="#" title="https://twitter.com/Microsoft_SDN" target="_blank"> Microsoft SDN </A> on twitter! </DIV> <BR /> <H2> <STRONG> Containers </STRONG> </H2> <BR /> <DIV> <BR /> <DIV> Q. Does Red Hat OpenShift support Windows Containers?&nbsp; Where can I find out more about Red Hat OpenShift?&nbsp; What is the roadmap of supporting Windows Containers with Kubernetes? </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> We're posting this answer from Containers PM, <A href="#" title="https://twitter.com/huskyat" target="_blank"> Mike Kosteritz </A> under protest because it's technically three questions... </DIV> <BR /> <DIV> </DIV> <BR /> <DIV> See the blog post " <A href="#" title="https://blogs.technet.microsoft.com/networking/2018/11/14/managing-containers-with-red-hat-openshift-container-platform-3-11-and-microsoft-windows-server-nodes/" target="_blank"> Managing Windows containers with Red Hat OpenShift Container Platform 3.11 </A> " for an overview what is coming in this space. </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> General information on OpenShift is available on the <A href="#" title="https://www.openshift.com/products" target="_blank"> https://www.openshift.com/products </A> website. If you have Windows specific questions please post a comment to the blog post at " <A href="#" title="https://blogs.technet.microsoft.com/networking/2018/11/14/managing-containers-with-red-hat-openshift-container-platform-3-11-and-microsoft-windows-server-nodes/" target="_blank"> Managing Windows containers with Red Hat OpenShift Container Platform 3.11 </A> " </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> For more information, please see the blog post here " <A href="#" title="https://blogs.technet.microsoft.com/networking/2018/09/19/ws2019-kubernetes/" target="_blank"> Top 10 Networking Features in Windows Server 2019: #1 Container Networking with Kubernetes </A> " </DIV> <BR /> </DIV> <BR /> <H2> <STRONG> Networking Diagnosis Tools </STRONG> </H2> <BR /> <DIV> <BR /> <DIV> Q. How do I review all the pertinent Networking information on my system.&nbsp; I'm not sure I know all the cmdlet's I need or how to put the data together into a cohesive view of my system. </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> Answer from Datapath PM, <A href="#" title="https://twitter.com/Dan2_2023" target="_blank"> Dan Cuomo </A> : </DIV> <BR /> <DIV> </DIV> <BR /> <DIV> Get-NetView is a nifty script that curates all the pertinent networking information into a single zip file for portability.&nbsp; It even grabs the data about the VMs sitting on system.&nbsp; If you're one of the many customer's we've worked with over the last year or so, you've no doubt had to run this command and send us the output for review.&nbsp; Also, this tool is integrated into Get-SDDCDiagnosticInfo cmdlet you've no doubt run when troubleshooting Storage Spaces Direct. </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> Once extracted to a folder, we'd recommend using <A href="#" title="https://code.visualstudio.com/" target="_blank"> Visual Studio Code </A> for review of the contents of the folder.&nbsp; Check out <A href="#" title="https://aka.ms/Get-Netview" target="_blank"> Get-NetView </A> on GitHub </DIV> <BR /> <DIV> </DIV> <BR /> <P> </P> <BR /> <DIV> Happy Turkey Day, </DIV> <BR /> </DIV> <BR /> <DIV> Windows Core Networking Team </DIV> </BODY></HTML> Thu, 14 Feb 2019 18:11:53 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/turkey-day-mailbag/ba-p/339822 Dan Cuomo 2019-02-14T18:11:53Z Managing Windows containers with Red Hat OpenShift Container Platform 3.11 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/managing-windows-containers-with-red-hat-openshift-container/ba-p/339821 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Nov 14, 2018 </STRONG> <BR /> <H3> Who is the new guy blogging? </H3> <BR /> Before getting into the topic, I wanted to introduce myself. <BR /> My name is Mike Kostersitz, I am Principal Program Manager and just joined the <A href="#" target="_blank"> core networking team </A> in the Cloud and AI organization. I will be focusing on expanding the Windows container networking ecosystem, work with partners to bring their solutions to Windows and bridge the gap between Linux and Windows systems in the container space. <BR /> Enough of the intro. Let’s jump in. <BR /> <H3> What is Red Hat OpenShift </H3> <BR /> In short Red Hat OpenShift enables build and deployment automation as well as continuous integration and continuous delivery for container systems built on kubernetes. At a high level Red Hat OpenShift is an open source container management platform which sits on top of the Kubernetes container orchestration system and the container runtime. <BR /> <BR /> My colleague David Schott has covered the current state of <A href="#" target="_blank"> Kubernetes support for Windows </A> Microsoft is working with the RedHat OpenShift team to enable management of mixed clusters using the Red Hat OpenShift platform toolset and allow for deployment of mixed clusters using OpenShift. <BR /> <BR /> Red Hat OpenShift for Windows will enable managing Windows server 2019 Nodes and containers in a mixed Linux and Red Hat Windows OpenShift deployment. <BR /> The solution will be run on RedHat Enterprise Linux 7.x and use for now Windows Server 1803 worker nodes. The nodes can be physical or virtual. <BR /> <BR /> In the simple example below, we deploy two virtual machines, one to run the Red Hat OpenShift master node and one to be the worker node for the Windows containers running on Windows Server Core. <BR /> To enable network connectivity the solution will use the <A href="#" target="_blank"> Cloudbase OVN/OVS CNI plugin </A> and to allow seamless setup the cluster will require a DNS and a DHCP server. The DHCP server will be used to assign IP addresses to the Windows worker node and all pods in the system. <BR /> ( <EM> Note to self: </EM> Don’t deploy a DHCP server in your corporate network. Bad things might happen, such as everyone in the local area getting a non-routable IP address from your server blocking internet and corporate resource access.) <BR /> <UL> <BR /> <LI> The high-level deployment of the components looks like the below diagram. </LI> <BR /> <LI> The master node currently runs Red Hat OpenShift version 3.11 on top of RHEL 7.5. </LI> <BR /> <LI> The worker node runs Windows Server Core, currently 1803 but soon on Windows Server 2019. </LI> <BR /> <LI> Both use the <A href="#" target="_blank"> Cloudbase Solutions developed OVS/OVN plug in </A> for networking. We are working on adding other CNI plug ins before release. </LI> <BR /> <LI> The networking mode is setup as an overlay network but will support other modes too. </LI> <BR /> </UL> <BR /> [caption id="attachment_7655" align="aligncenter" width="583"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75291iBC7823050BA6A7D1" /> OpenShift for Windows example deployment[/caption] <BR /> <H3> Summary </H3> <BR /> Red Hat OpenShift support for Windows is coming and will provide the build, deployment and CI/CD capabilities of Red Hat OpenShift on Linux to Windows Server. <BR /> While we have not set a final release date yet for Red Hat OpenShift on Windows we are working closely with the OpenShift team at Red Hat and are looking forward to release a preview of what is to come sometime in the first half of next year. <BR /> <BR /> Stay tuned for more on this topic as things develop. <BR /> <BR /> Thanks for reading this far and keep an eye out for our next post on OpenShift for Windows <BR /> Editors Note: 11/15/18: Updated formatting and fixed a few typos. <BR /> </BODY></HTML> Thu, 14 Feb 2019 18:11:50 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/managing-windows-containers-with-red-hat-openshift-container/ba-p/339821 mkostersitz 2019-02-14T18:11:50Z Windows Transport converges on two Congestion Providers: Cubic and LEDBAT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-transport-converges-on-two-congestion-providers-cubic/ba-p/339819 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Nov 07, 2018 </STRONG> <BR /> <A href="#" target="_blank"> #LEDBAT </A> <A href="#" target="_blank"> @Win10Transports </A> <BR /> <BR /> Why don't we dive right&nbsp; in?&nbsp;&nbsp; What is a Congestion Provider and why do you (the reader) care? <BR /> <UL> <BR /> <LI> What is it? A Congestion Provider is an algorithm that controls the flow of data from a Windows computer to any other computer. </LI> <BR /> <LI> Congestion Provider defined: Because Cubic is for humans and LEDBAT is for unattended scenarios. </LI> <BR /> </UL> <BR /> How does that impact me? &nbsp;In the heart of the Windows kernel there is a networking stack. At the heart of the networking stack there is a layer called Transport and Transport contains a suite of algorithms called Congestion Providers. &nbsp;the data flow across the network/Internet. <BR /> <BR /> Let’s take a look at the difference between the two. Cubic is optimized for throughput while LEDBAT is optimized for low-latency and non-interference. Now the picture is becoming clear. LEDBAT is for unattended scenarios (meaning that there is not a person actively waiting for the transaction to complete) because these bots should not interfere with human work and Cubic should be used when there is a person waiting for the transaction to complete. <BR /> <BR /> Let’s take a deeper look at the difference between the two using a specific example. Suppose we have a person doing their work with a web browser and a software update that is being delivered by SCCM. The difference here is clear. The software update should be using LEDBAT so that it does not interfere with the person and the web browser should be using Cubic. With this arrangement the software update will do its work leveraging unused bandwidth and when the person with the web browser wants to use the network the software update will relinquish the network resources. This allows the software updates to proceed without interfering with the person. Use your good judgement. Applications that need to proceed without interfering with people that are working need LEDBAT. Applications that people are using to do work need Cubic. <BR /> <BR /> [caption id="attachment_7585" align="aligncenter" width="641"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75288i890993698C72D739" /> <STRONG> Figure 1 -- Idealized Network Diagram </STRONG> [/caption] <BR /> <BR /> Let’s look at the idealized network diagram in the figure above. There is a TCP sender on the left and a TCP receiver on the right. The TCP sender sends packets into the network which is modeled by a single queue. Upon receiving a packet, the network devices immediately forward the packet towards its destination. If the device cannot forward the packet immediately (because it is busy forwarding a previously received packet) it will place the packet in a queue. In the figure there are four packets in the queue. If the TCP sender sends another packet at this time, then it will have to wait for the four packets in the queue to be sent. This is called queuing delay and it is what causes the &nbsp;behavior that irritates people. <BR /> <BR /> [caption id="attachment_7595" align="aligncenter" width="641"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75289i73ABE33853DD4C65" /> <STRONG> Figure 2 -- Cubic drives the queue to saturation </STRONG> [/caption] <BR /> <BR /> Cubic tries to optimize throughput by sending packets faster and faster until one of them is dropped then it slows down and repeats the behavior. Because the sender keeps increasing the sending rate eventually the queue will be full. If a packet arrives when the queue is full then that packet must be dropped. When a packet is dropped (besides retransmitting) Cubic slows the sending rate by half (draining the queue) and repeats the process. The queue repeatedly fills and drains from this process optimizing throughput. <BR /> <BR /> [caption id="attachment_7605" align="aligncenter" width="641"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75290iB5AFFC67AE5C895E" /> <STRONG> Figure 3 -- LEDBAT controls the queuing delay </STRONG> [/caption] <BR /> <BR /> LEDBAT (shown in the Figure above) tries to optimize throughput just like Cubic by sending packets faster and faster. However, LEDBAT also keeps track of the queuing delay (lagginess&nbsp;). When the lagginess increases too much LEDBAT slows down and drains the queue. This accomplishes two things. LEDBAT keeps the queuing delay lower and since Cubic drives the queue past LEDBAT’s delay threshold LEDBAT will always defer the network resources to Cubic. In other words, LEDBAT will use all of the network resources unless Cubic is using them. <BR /> <BR /> This makes the perfect combination. Your background tasks such as software updates, backup, etc. can cruise along doing their work while the network is not in use and when a person hops on the network to do their work the LEDBAT tasks will get out of the way. Let’s use our person with their Edge web browser using Cubic and our software updates using LEDBAT as an example again to see how this works. <BR /> <BR /> What makes this combination work so well is that humans and computers are opposite. The person using the web browser clicks on a few things and impatiently waits for the network to deliver them. This needs to be done as quickly as possible because even a few seconds can be painful and frustrating to a person. However, once the human has their content, they spend a great deal of time reading and looking at pictures. During this time, they are consuming their content and not really using the network at all. <BR /> <BR /> This is what makes the combination awesome! Computers don’t get frustrated and they react very quickly. So, we have our software update proceeding nicely at full speed and then along comes a human who is in need of their information immediately, so they hit their Edge web browser clickity, clickity, clack, click, click, clack! The LEDBAT controller operating the software update download notices the increase in queuing delay (remember the figures?) and gets out of the way right away. The person gets their stuff immediately and happily begins consuming the information. While they are doing that the LEDBAT controller notices the unused network resources and downloads some more data. The person decides that they need more stuff and click away at their web browser and so on. The perfect team! <BR /> <BR /> <BR /> <BR /> So, what are your action items here? If you are running a client just get the latest Windows 10 update and you will have Cubic by default. If you have Windows Server 2019 same thing. Cubic is already the default Congestion Provider. If you are running Windows Server 2016 Cubic is not default, but you can fix that by running Windows Update and this powershell: <BR /> <BR /> This is what the default templates look like: <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting | Select SettingName, CongestionProvider, AutomaticUseCustom <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CongestionProvider &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AutomaticUseCustom <BR /> -----------&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ------------------ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;------------------ <BR /> Automatic <BR /> InternetCustom.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Disabled <BR /> DatacenterCustom. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCTCP&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;Disabled <BR /> Compat&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NewReno&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Disabled <BR /> Datacenter &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCTCP&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;Disabled <BR /> Internet&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Disabled <BR /> </BLOCKQUOTE> <BR /> We can only change the Custom templates, so we need to make the server use the custom templates. Changing AutomaticUseCustom to Enabled will do this for us: <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Set-NetTCPSetting -SettingName InternetCustom -AutomaticUseCustom Enabled <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting | Select SettingName, CongestionProvider, AutomaticUseCustom <BR /> SettingName &nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CongestionProvider&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AutomaticUseCustom <BR /> -----------&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;------------------&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;------------------ <BR /> Automatic <BR /> InternetCustom&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Enabled <BR /> DatacenterCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCTCP&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;Enabled <BR /> Compat&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;NewReno&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Enabled <BR /> Datacenter&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DCTCP&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;Enabled <BR /> Internet&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Enabled <BR /> </BLOCKQUOTE> <BR /> Hey, they all changed even though I only changed the InternetCustom template! Yes, AutomaticUseCustom is an all or nothing setting. <BR /> <BLOCKQUOTE> <BR /> Now we need to change the templates Congestion Provider to Cubic! <BR /> <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CongestionProvider&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AutomaticUseCustom <BR /> -----------&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;------------------&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;------------------ <BR /> Automatic <BR /> InternetCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CUBIC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Enabled <BR /> DatacenterCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CUBIC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Enabled <BR /> Compat&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;NewReno&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Enabled <BR /> Datacenter&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DCTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Enabled <BR /> Internet&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Enabled <BR /> </BLOCKQUOTE> <BR /> And now we are a Cubic server just like WS 2019! <BR /> <BR /> If you want to use LEDBAT then see the instruction (“Try it out!” link) in my LEDBAT blog: <A href="#" target="_blank"> Top 10 Networking Features in Windows Server 2019: #9 LEDBAT – Latency Optimized Background Transport </A> <BR /> <BR /> <BR /> <BR /> Thanks for reading! </BODY></HTML> Thu, 14 Feb 2019 18:11:39 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-transport-converges-on-two-congestion-providers-cubic/ba-p/339819 Daniel Havey 2019-02-14T18:11:39Z Leap Seconds for the AppDev: What you should know https://gorovian.000webhostapp.com/?exam=t5/networking-blog/leap-seconds-for-the-appdev-what-you-should-know/ba-p/339813 <P><STRONG> First published on TECHNET on Oct 24, 2018 </STRONG> <BR />Author: Travis Luke <BR /><BR />Last week my esteemed colleague Dan Cuomo introduced Leap Seconds support for Windows 10 including what you need to know if you’re an IT Pro. <BR /><BR />If you’re an application developer, the things you need to know are a little bit different. I'm sure all of you were wondering how your application take advantage of the ‘60’ second?&nbsp; How can you accurately measure time and time durations during leap seconds?&nbsp; And how can frameworks and applications that calculate time stay in sync with the Operating System?&nbsp; So, in this article, I’ll attempt to explain all that and describe some of the details and considerations needed to support leap seconds in your application. <BR /><BR />Before we get into the details of what developers should consider, let’s have a brief history of measuring time and the birth of the Leap Second.&nbsp; As we all know (:)</img>), the Gregorian calendar has a set standard of measuring intervals for time. <BR /><BR /></P> <UL> <UL> <LI>1000 milliseconds per second.</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>60 seconds per minute</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>60 seconds per hour</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>60 minutes in every hour</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>24 hours in every day</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Months have a variable (but repeatable pattern) for the number of days in a month, ranging from 28 to 31.&nbsp; And years have 365 days.</LI> </UL> </UL> <P><BR /><BR />The big exception to that is leap years.&nbsp; ( <A href="#" target="_blank" rel="noopener"> Almost </A> ) every 4 years a day is added to February to create a 366-day year.&nbsp; What is convenient about all of this is it is very predictable.&nbsp; We can say with certainty how time will be counted in the coming years, decades, and centuries, down to the millisecond. <BR /><BR />However, Leap Seconds are not a predictable event.&nbsp; An International committee called IERS periodically decides to insert a leap second based on observations of the rotation of the earth.&nbsp; Every six months there is an announcement about whether a leap second will or will not be added or subtracted.&nbsp;This extra second occurs on either June 30 <SUP> th </SUP> or December 31 <SUP> st </SUP> .&nbsp; The timing of this event occurs at the same time all over the globe, at 23:59:59 UTC. <BR /><BR />If a second is added the official clock will switch move in 1000 ms increments from 23:59:59 UTC to 23:59:60 UTC to 00:00:00 UTC.&nbsp; If a second is subtracted (which has never happened so far) time would move in 1000 ms increments from 23:59:58 UTC to 00:00:00. <BR /><BR />When IERS publishes a leap second event this data will arrive to all Windows PCs through a few mechanisms.&nbsp; It may get this data when it is syncing its time through a <A href="#" target="_blank" rel="noopener"> NTP </A> server.&nbsp; By default, windows syncs with an NTP time source, such as time.windows.com every day.&nbsp; It may also receive this data through Windows Update.&nbsp;When this data arrives, it is stored in the Operating System.&nbsp; This allows Windows to operate on the knowledge of those events. <BR /><BR />Windows uses a structure called <A href="#" target="_blank" rel="noopener"> FILETIME </A> to record a timestamp. (If you are curious like me, you may wonder why it is called filetime. This is because it was originally used only in the Windows File System to represent the timestamp of a file. This structure is now used throughout the operating system for all timestamp related scenarios). The FILETIME structure represent the number of 100 nanosecond intervals since Jan 1, 1601.&nbsp; There are several APIs that are available to convert this value into a more readable form.&nbsp; For example <A href="#" target="_blank" rel="noopener"> FileTimeToSystemTime </A> will convert the FILETIME into a SYSTEMTIME structure representing the UTC time of that value.&nbsp; The <A href="#" target="_blank" rel="noopener"> SYSTEMTIME </A> structure provides a breaks down the year, month, day, hour, minute, and second values. <BR /><BR />Starting in Server 2019 and the Windows 10 October update <A href="#" target="_blank" rel="noopener"> time APIs </A> will now take into account all leap seconds the Operating System is aware of when it translates FILETIME to SystemTime. No change is made to FILETIME. It still represents the number of 100 ns intervals since the start of the epoch. What has changed is the interpretation of that number when it is converted to SYSTEMTIME and back. Here is a list of affected APIs: <BR /><BR /></P> <UL> <UL> <LI>GetSystemTime</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>GetLocalTime</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>FileTimeToSystemTime</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>FileTimeToLocalTime</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>SystemTimeToFileTime</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>SetSystemTime</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>SetLocalTime</LI> </UL> </UL> <P><BR /><BR />Previous to this release, SYSTEMTIME had valid values for wSecond between 0 and 59. &nbsp;SYSTEMTIME has now been updated to allow a value of 60, provided the year, month, and day represents day in which a leap second is valid. <BR /><BR /><BR /><BR />Here are number of Frequently Asked Questions about developing Leap Second Aware Applications: <BR /><BR /><STRONG> How can applications take advantage of the ‘60’ second? </STRONG> <BR /><BR />In order receive the 60 second in the SYSTEMTIME structure a process must explicitly opt-in.&nbsp; You can have your process do this by calling <A href="#" target="_blank" rel="noopener"> SetProcessInformation </A> with the ProcessLeapSecondInfo option. <BR />DWORD ErrorCode; <BR />BOOL Success; <BR />PROCESS_LEAP_SECOND_INFO LeapSecondInfo; <BR />ZeroMemory(&amp;LeapSecondInfo, sizeof(LeapSecondInfo)); <BR />Success = SetProcessInformation(GetCurrentProcess(), <BR />ProcessLeapSecondInfo, <BR />&amp;LeapSecondInfo, <BR />sizeof(LeapSecondInfo)); <BR />if (!Success) { <BR />ErrorCode = GetLastError(); <BR />fprintf(stderr, "Set Leap Second priority failed: %d\n", ErrorCode); <BR />goto cleanup; <BR />} <BR />By calling this you are telling the operating system that your application will accept a SYSTEMTIME structure with the values between 0 and 60.&nbsp; Applications are expected to handle the 60 value in a way that makes sense.&nbsp; For example, if your application is showing transactions in a list with a time stamp, it will display the timestamp with the 23:59:60 value. &nbsp;&nbsp;Or if your application is an analog clock it may play a special animation to indicate a leap second is occurring. <BR /><BR />Application developers are encouraged to test their application with the process opted in.&nbsp; We have provided a simple method to the opted in behavior without recompiling your code. Please check our previous blog entry on <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Leap-Seconds-for-the-IT-Pro-What-you-need-to-know/ba-p/339811" target="_blank" rel="noopener"> Leap Seconds for the IT Pro </A> which has a section named “Testing Applications” that provides a method to opt-in through the registry. <BR /><BR />You can also see examples of how to insert leap seconds yourself for testing purposes using w32tm.exe. <BR /><BR /><BR /><BR /><STRONG> How can application developers ensure their application is Leap Second compatible? </STRONG> <BR /><BR />There is a valid concern that if the SYSTEMTIME structure displayed a Seconds value of 60 that it would break applications that are not leap second aware.&nbsp; Imagine your application was an analog clock.&nbsp; It may be assuming that valid values of the seconds is between 0 and 59.&nbsp; If it is 60 it may crash as it attempts to calculate the angle to draw the second hand. <BR /><BR />To address this, by default all processes are in a “compatibly mode” unless they explicitly opt in to receive the ‘60’ second.&nbsp; In the compatibility mode the second value will be guaranteed to be between 0 and 59.&nbsp; In the second before a leap second is added, or at the ‘59’ second, the clock will slow down to half speed for two seconds.&nbsp; This will have the visual effect of the 59 second being twice as long.&nbsp; During this time the milliseconds values will also be slowed down by ½.&nbsp; When this 2000 milliseconds is complete the clock will resume incrementing at normal speed.&nbsp; This has the effect of giving applications the leap second and allowing for all timestamps that occur during the “slowdown” period to be sorted in the correct order they occurred. To reiterate, the above is the default behavior while in compatibility mode. <BR /><BR /><BR /><BR /><STRONG> How can you accurately measure time and time durations during leap seconds? </STRONG> <BR /><BR />One question that frequently comes up is how do you measure a time duration? &nbsp;Say you want to add 1 day to an existing time stamp. &nbsp;Does that mean adding 24 hours, or 1,440 minutes, or 86,400 to a given time stamp?&nbsp; Or does that mean adding 1 day, regardless of how many seconds that day has (including possible leap seconds).&nbsp; If what you want is to add 86,400 seconds you can follow the guidance <A href="#" target="_blank" rel="noopener"> here </A> .&nbsp; In this case you are taking the FILETIME structure, and moving it forward a specific number of milliseconds to achieve one normal day worth of time. <BR /><BR />On the other hand, if you want to increment one day regardless of the seconds in that day then there is another approach you must use. In this case you convert your FILETIME to a SYSTEMTIME structure using <A href="#" target="_blank" rel="noopener"> FileTimeToSystemTime </A> .&nbsp; Then add the number of days to the structure.&nbsp; Then covert it back to a FILETIME using the <A href="#" target="_blank" rel="noopener"> SystemTimeToFileTime </A> API.&nbsp; This will allow the operating system to apply the arithmetic to covert the SYSTEMTIME to a FILETIME, while factoring in any known leap seconds. <BR /><BR />Care must be taken when FILETIME values are passed between computers.&nbsp; If a FILETIME value is generated on an older windows PC or from a non-Windows PC then it may not be taking into account Leap Seconds.&nbsp; If that FILETIME was then converted to a SYSTEMTIME structure on a PC that does take into account leap seconds, then the intended time may be off.&nbsp; To correct for this a registry key has been provided which disables all leap second logic.&nbsp; If you set this registry key then all behavior involving SYSTEMTIME and leap seconds is reverted.&nbsp; If you are passing FILETIME values in a heterogeneous environment, you may consider setting this key.&nbsp;You can find more details about this in <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Leap-Seconds-for-the-IT-Pro-What-you-need-to-know/ba-p/339811" target="_blank" rel="noopener"> here </A> under the subject “Revert to Prior OS Behavior”. <BR /><BR /><BR /><BR /><STRONG> How can frameworks and applications that calculate time stay in sync with the Operating System? </STRONG> <BR /><BR />Some frameworks and applications may attempt to calculate time using their own arithmetic.&nbsp; For example, the .NET Framework has logic in the System.DateTime structure to represent time.&nbsp; If the implementation of the calculation of time is not handled by the operating system, then the framework may arrive at a different time then the Operating System.&nbsp; For example, imagine if you called DateTime.Now one month after a leap second occurred.&nbsp; The framework would call GetSystemTimeAsFileTime to get the FILETIME of the current moment.&nbsp; It would then store this value inside the structure.&nbsp; When a user wanted to know the date they may call the .ToString() function.&nbsp; If this framework was attempting to perform its own arithmetic to turn that time into year, month, day, hour, minute, second value, and didn’t take into account the leap second, then the time it returned would be one second faster than the time the operating system reported. For each leap second that was added to the system the framework would continue to drift forward in time.&nbsp; To correct this the .NET Framework updated the implementation to call the FileTimeToSystemTime API.&nbsp; This allows the operating system rather than the framework to account all leap seconds and perform the proper arithmetic. <BR /><BR />Applications that rely on 3 <SUP> rd </SUP> party frameworks should ensure their framework’s implementation on Windows is also calling into the correct APIs to calculate the correct time, or else the application will have the wrong time reported. <BR /><BR /><BR /><BR /><STRONG> Does the .NET framework support Leap Seconds? </STRONG> <BR /><BR />At the time of this writing the System.DateTime structure does not account for leap seconds. It effectively runs in compatibility mode as described in the above section. In other words, during the moment of a leap second the ‘59’ second will be twice as long. Stay tuned for updates as greater leap second support is added to the .NET Framework. <BR /><BR /><BR /><BR /><STRONG> How can I prevent Leap Seconds from occurring? </STRONG> <BR /><BR />We have had a lot of discussion about this. We are thinking of organizing a day of everybody who is against leap seconds to run west. This will hopefully have the effect of changing the rotation of the earth and eliminate the need of the leap second. <BR /><BR /><BR /><BR />We recommend all developers to make their applications leap second aware. We encourage you to try out the tools we provided to test your applications and choose the approaches that work for your scenarios. We are eager to hear back from the development community their experiences with Leap Seconds. <BR /><BR />Thanks for reading, <BR /><BR />Travis Luke</P> Sun, 31 Mar 2019 23:01:48 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/leap-seconds-for-the-appdev-what-you-should-know/ba-p/339813 Daniel Havey 2019-03-31T23:01:48Z Leap Seconds for the IT Pro: What you need to know https://gorovian.000webhostapp.com/?exam=t5/networking-blog/leap-seconds-for-the-it-pro-what-you-need-to-know/ba-p/339811 <P><STRONG> First published on TECHNET on Oct 17, 2018 </STRONG> <BR />Hi Everybody – Program Manager Dan Cuomo here to tell you, the IT Pro, everything you need to know about Leap Seconds on Windows. If you saw our recent blog series on the <A href="#" target="_blank" rel="noopener"> Top 10 Networking Features </A> , you may have already noticed an announcement about <A href="https://gorovian.000webhostapp.com/?exam=t5/Networking-Blog/Top-10-Networking-Features-in-Windows-Server-2019-10-Accurate/ba-p/339739" target="_blank" rel="noopener"> Leap Second support </A> included in Windows Server 2019 and Windows 10 October 2018 Update. </P> <P><STRONG> Note: </STRONG> If you’re an Application Developer, stay tuned for our future post Leap Seconds for the Application Developer: What you need to know</P> <P><BR />For most IT Professionals, you may not be concerned about Leap Seconds. However, if you’re a customer with time-sensitive applications or in a regulated industry requiring high accuracy time, a <SUB> measly little second </SUB> could hurl you into an auditing and compliance frenzy. Whether you call it a v-team or tiger-team nobody wants to have to write those status reports, <STRONG> A </STRONG> fter <STRONG> A </STRONG> ction <STRONG> R </STRONG> eports, or <STRONG> R </STRONG> oot <STRONG> C </STRONG> ause <STRONG> A </STRONG> nalysis (or whatever your organization calls them) to explain just what exactly went wrong. A leap second comes and goes quickly, but the effects could last some time. <BR /><BR />So in this article, we’ll attempt to explain everything the IT Pro needs to know so you can explain, test, and deploy Windows Server 2019 and Windows 10 October 2018 Update with confidence for your time-sensitive scenarios. </P> <P><STRONG> Note </STRONG> : Leap Seconds are only included in Windows Server 2019 and Windows 10 October 2018 Update and later releases so this content is not applicable to operating systems prior to this release.</P> <P><BR /><BR /></P> <H2>What are Leap Seconds</H2> <P><BR />Lets first understand what a leap second is. A leap second is an occasional 1-second adjustment to UTC. As the earth’s rotation slows (e.g. tidal forces, earthquakes, hurricanes, etc.) <A href="#" target="_blank" rel="noopener"> UTC </A> diverges from <A href="#" target="_blank" rel="noopener"> mean solar time </A> or astronomical time.&nbsp; Leap seconds are added to keep the difference between UTC and astronomical time to less than 0.9 seconds. Don’t worry, we don’t need to start colonizing new planets (yet ;)</img>).&nbsp; But still, wish we found out how that jump across galaxies worked out for the <A href="#" target="_blank" rel="noopener"> Stargate Universe </A> crew… </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 998px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75278i3EA35865D922EDC9/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />An organization called the <STRONG> I </STRONG> nternational <STRONG> E </STRONG> arth <STRONG> R </STRONG> otation and <STRONG> R </STRONG> eference <STRONG> S </STRONG> ystems <STRONG> S </STRONG> ervice (IERS) oversees the announcement of Leap Seconds. They release several <A href="#" target="_blank" rel="noopener"> bulletins </A> ; Bulletin C is released every 6 months to confirm whether there will be a leap second or not. </P> <P><STRONG> Note: </STRONG> At the time leap seconds were introduced in 1972, a necessary correction of ten seconds was made to UTC. There have since been 27 leap seconds added to UTC for a total of 37 one-second corrections. Leap seconds are added, on average, every 1.5 yrs ( <A href="#" target="_blank" rel="noopener"> NIST FAQ </A> ).</P> <P><BR /><BR /></P> <H2>Leap Seconds on Windows Overview</H2> <P><BR />Now let’s talk about some of the high-level principles needed to understand Leap Seconds on Windows. </P> <H3>UTC-Compliant Leap Seconds</H3> <P><BR />If you are in a regulated industry, you must not only implement leap seconds, but you must do so in a UTC-compliant manner. This means that the leap second must be added to the last minute of the UTC day. During this minute, the clock goes from 0 to 60 seconds (for a total of 61 seconds). </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 334px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75279iAA9E3DF8B6E98CD9/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Windows Server 2019 and Windows 10 October 2018 Update implements the leap second in a UTC-compliant manner enabling customers to meet the requirements in regulated industries. </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 364px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75280i838CDC2E596F692B/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Industry experts <A href="#" target="_blank" rel="noopener"> have gone on record </A> to denounce leap second “smearing” – an alternative approach that carves the leap second into smaller units and inserts them throughout the day. Leap second smearing is not UTC-compliant and as such, Windows does <STRONG> NOT </STRONG> implement leap second smearing. </P> <H2>Built for compatibility</H2> <P><BR />The majority of Windows users will not need Leap Second information; either their workloads do not depend on that high of accuracy or are not under industry regulations. If this description sounds like you, feel free to tweet a link to this blog, might I recommend... </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 715px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75281i4A908327820D17D0/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />...And feel free to stop reading. While the system (kernel) is tracking leap seconds, they will not affect your every day life as applications are never notified that a leap second is occurring unless an application has specifically “opted-in.”&nbsp; Applications are, by default, <STRONG> none the wiser unless action is taken </STRONG> . <BR /><BR />This is important both for customers who have heterogeneous operating system environments to interoperate seamlessly as they always have prior to this release as well as for application compatibility. Many applications expect seconds to be between 0 and 59. If the application isn’t expecting a 60, apps could fail, cats and dogs living together, mass hysteria! </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 500px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75282i34BF5621A2F43692/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR /><BR /></P> <H3>Previous Leap Seconds</H3> <P><BR />For these same reasons, we do not track prior leap seconds. Our goal is to enable customers needing high accuracy time moving forward. Regulations requiring high accuracy, UTC-compliant time, did not come into affect until relatively recently, and therefore prior leap seconds are not necessary to track. For reference the last leap second prior to the release of leap-second aware Windows was December 31 <SUP> st </SUP> 2016, that is, at the time of writing, we have not had a leap second since this date. Leap seconds after this date, will be tracked by Windows Server 2019 and Windows 10 October 2018 Update. </P> <H3>What happened to previous leap seconds</H3> <P><BR />There’s a logical question of how previous operating systems treated leap seconds. If previous operating systems didn’t track leap seconds, are they 37 seconds off from UTC? <BR /><BR />No, although previous operating systems did not track leap seconds, when they synchronized their time at the next interval, they recognized that they were one-second behind and time was moved forward to match the current UTC time. </P> <H3>A Tale of Two Timelines</H3> <P><BR />"It was the best of times, it was the worst of times…It was the epoch of belief, it was the epoch of incredulity." Since leap seconds are new in Windows 10 October 2018 Update and Windows Server 2019, prior operating systems will not know about this augmented time scale. As a result, the timelines under the hood of Windows will begin to diverge between these two operating systems as leap seconds occur. </P> <P>So when the next leap second rolls in, we’ll begin an alternate timeline for Windows :smiling_face_with_smiling_eyes:</img></P> <P><BR />Unless your application is leap second aware, it is unlikely that you will notice this delta. However if you were to view an event log from a leap-second aware system on a machine that is not aware of the leap seconds, the time displayed for the event will be off by the number of leap seconds known by the system (mmc.exe is opted-in by default). </P> <H3>Revert to Prior OS Behavior</H3> <P><BR />As a reminder, applications must opt-in to receiving leap second notifications so leap seconds will not affect any applications by default and is likely unnecessary to modify the default behavior. <BR /><BR />However, if you have a heterogenous time-sensitive environment you can revert to the prior operating system behavior and disable leap seconds across the board by adding the following registry key: </P> <P><STRONG> HKLM:\SYSTEM\CurrentControlSet\Control\LeapSecondInformation </STRONG></P> <P>&nbsp;</P> <P><STRONG> Type </STRONG> : "REG_DWORD"</P> <P>&nbsp;</P> <P><STRONG> Name </STRONG> : Enabled</P> <P>&nbsp;</P> <P><STRONG> Value </STRONG> : 0 Disables the system-wide setting</P> <P>&nbsp;</P> <P><STRONG> Value </STRONG> : 1 Enables the system-wide setting</P> <P><BR />Next, restart your system. </P> <H2>How Leap Seconds Propagate</H2> <P><BR />Every four years, we have a leap year - this is known and predictable. Leap seconds however, are different in that they are not on a regular cadence. Instead, leap seconds are announced by IERS only 6 months in advance. From there, GPS distributes the leap second notification to time servers and ultimately to Windows systems. So let’s talk about some of the mechanisms in-place to make sure that you get the leap second notification. </P> <H3>Time Server Distribution</H3> <P><BR />The Windows Time service includes a server provider that allows a Windows system to operate as a time server. For example, when you add a domain controller to your forest this domain controller can serve time to other clients on the network through this mechanism. This is not the only method of installing a time server; you can check to see if your system is operating as a time server by using the command (Enabled: 1): <BR /><BR /><STRONG> w32tm /query /configuration </STRONG> </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 691px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75283i73FDE02984723E84/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />The Windows Time server distributes the leap second notification to time clients. As GPS distributes time (and the leap second notification) to the Windows Time server, it will pass that notification onto clients; to be clear, your system doesn’t need to be a domain controller to do this. </P> <H3>Windows Update</H3> <P><BR />But what if your system is when the notification comes? Or more likely what if you re-image your system? You’ll want to make sure that new systems know about the upcoming leap second and if the new system is created after a leap second, you’ll want to make sure that this system is synchronized with the other machines on the network. <BR /><BR />To make sure this is possible, we’ll distribute leap second notifications through Windows Update as well. This provides a simple mechanism for reporting (nodes that have the latest updates have the leap second information as well). <BR /><BR /><STRONG> Best Practice: </STRONG> The simplest and most effective manner for distributing and verifying leap second information across your environment is through Windows Update.&nbsp; If you're on the latest updates, you'll have the notifications! </P> <H3>Hyper-V VMIC</H3> <P><BR />If you have Hyper-V virtual machines, the Hyper-V virtual machine integration components will also provide leap second notifications to those virtual machines.&nbsp; If the virtual machine is not one of the leap-second aware operating systems (or later) this will have no affect. </P> <H2>Verify that your system got the leap second</H2> <P><BR />In addition to verifying updates across your system, you can also use the following command to view the leap seconds known by a specific system. In the screenshot below, a positive ( <STRONG> + </STRONG> ) leap second will be inserted after 23:59:59 on 6/30/2019 </P> <P><STRONG> w32tm /leapseconds /getstatus /verbose </STRONG></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 565px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75284i3B70E3FCF3784E5F/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR /><BR /></P> <H2>Testing Applications</H2> <P><BR />Applications must be written to consume and process leap seconds – As you're read a number of times already, we assume that applications are not leap-second aware. You can search every application’s documentation to find out if it’s leap second aware, or if you’re an IT Pro in one of these regulated industries, we anticipate that you will want to test and verify your application or system images for leap seconds. <BR /><BR />If you want to manually test and opt-in an application, identify the process name, for example: </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 953px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75285iC1EB0A4370E5995E/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Next open the registry editor and navigate to </P> <P><STRONG> HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options </STRONG></P> <P><BR />Add a key which is the same name as the process you want to opt-in to leap seconds. In this example, we’ve opted-in the <STRONG> winword.exe </STRONG> process by creating a Registry Key (folder icon). <BR /><BR />Next create a <STRONG> REG_DWORD </STRONG> named <STRONG> GlobalFlag2 </STRONG> with a value of 1. </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 816px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75286i5517DAFCC7DAE3A3/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Now <STRONG> restart the process </STRONG> and insert a test leap second as shown in the "Insert Positive Leap Seconds" section of our <A href="#" target="_blank" rel="noopener"> IT Pro Validation Guide </A> , and test critical application functionality. <BR /><BR />If your application doesn’t support leap seconds, please contact the application owner and tell them to check our future post, Leap Seconds for the Application Developer: What you need to know. </P> <H2>Testing Systems</H2> <P><BR />Instead of testing an individual application one-by-one, you may want to test a holistic system. To do this, open the registry editor and navigate to: </P> <P><STRONG> HKLM:\SYSTEM\ControlSet001\Control\Session Manager </STRONG></P> <P><BR />Next create a REG_DWORD named <STRONG> GlobalFlag2 </STRONG> with a value of <STRONG> 1 </STRONG> as shown here. </P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75287iB3A2635DC57649F3/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Restart the system then insert leap seconds as before and test critical application functionality. Note any application or system events in the event log. </P> <H2>Summary</H2> <P><BR />Most IT Professionals may not need to be concerned about Leap Seconds. However, if you’re a customer in a regulated industries requiring high accuracy time or have time sensitive applications, you need to ensure your systems apply and maintain time accurately through a leap second. Windows Server 2019 and Windows 10 October 2018 Update brings support for, true UTC-compliant leap seconds. To make sure that these are properly implemented on your systems, you should verify your patch management strategy, application compatibility, and more. <BR /><BR />Please give this a shot, and of course let us know how it went! <BR /><BR />Dan "my leap seconds land on 60" Cuomo</P> Sun, 31 Mar 2019 17:24:27 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/leap-seconds-for-the-it-pro-what-you-need-to-know/ba-p/339811 Dan Cuomo 2019-03-31T17:24:27Z Notes from the Field: Microsoft SDN Software Load Balancers https://gorovian.000webhostapp.com/?exam=t5/networking-blog/notes-from-the-field-microsoft-sdn-software-load-balancers/ba-p/339800 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Oct 10, 2018 </STRONG> <BR /> Kyle Bisnett and Bill Curtis here. We are two Software Defined Network Blackbelts and Premier Field Engineers at Microsoft and specialize in Hybrid Cloud technologies, which includes Cloud Platform System, Azure Stack and WSSD/SDDC. Most importantly, we ensure it’s easy for our customers and partners to deploy and leverage Software Defined Networking (SDN), whether it’s within an enterprise or as part of a Partner Solution (WSSD). <BR /> <BR /> Recently, our customer came to us asking questions about our <A href="#" target="_blank"> SDN Load Balancers </A> (SLB) as they were looking into using fewer physical appliances and deployments of the venerable Microsoft Network Load Balancing (NLB) with an SDN solution. In this blog, we will cover some common questions we received from this customer and others in the field about SDN SLB. <BR /> <H2> Briefly, what is Microsoft Software Defined Networking? </H2> <BR /> If you have deployed Windows Server 2016 and/or Windows Server 2019, chances are you’ve heard about Software Defined Networking (SDN) that comes at no additional cost in our Datacenter SKU. Also, if you’ve looked at our prior blogs, you have seen mentions about SDN going <A href="#" target="_blank"> mainstream </A> . <BR /> <BR /> Microsoft SDN provides software-based network functions such as virtual networking with switching, routing, firewalling with micro-segmentation, third-party appliances, and of course load balancing – the subject of today’s post&nbsp;. These are all virtualized and highly optimized for availability and performance and, like Storage Spaces Direct, is a key component of the <A href="#" target="_blank"> Windows Server Software Defines(WSSD)/Software Defined Datacenter (SDDC). </A> <BR /> <H2> Why should I use Microsoft’s SDN Software Load Balancer? </H2> <BR /> ...There are plenty of other SDN Load Balancer solutions that have been around for longer, right? <BR /> <BR /> Microsoft SDN is an end-to-end solution. All the components work in harmony together, and you can leverage features that are a direct result of this synchronization, such as Direct Server Return (DSR), health-probing on the Hyper-V hosts, and NAT functionality. Keep in mind, the other benefit is from an administrative perspective as you no longer need worry about expensive support contracts, hardware upgrade cadences (these are just Windows VMs), and some of the odd items like Active/Passive. All SLB MUXs are always Active/Active whether it’s two or eight. <BR /> <H2> SDN in Server 2016\2019 is closely based on the SDN running in Microsoft Azure </H2> <BR /> Software Defined Networking is being utilized across 32 different global Azure datacenters. When you configure a Standard or Basic Load Balancer, Virtual Network (vNet), Site to Site VPN Connections and more in Microsoft Azure, you are using SDN architecture that has been ported over to SDN in Windows Server 2016\2019, and Azure Stack. Microsoft SDN is well-tested at scale and is very competitive with other SDN products in terms of performance and scalability. <BR /> <H2> Are the SLB MUXs highly available? </H2> <BR /> If so, how can I ensure it is checking my Guest VMs to ensure they are ‘up’ or ‘down’? <BR /> <BR /> SLB MUXs are fault tolerant and utilize Border Gateway Protocol (BGP), which is a dynamic routing protocol that advertise all MUXs within the pool in a /32 subnet form to the top-of-rack switch. When a keep-alive metric is missed, BGP automatically removes the individual load balancer from the routing table. This is helpful in host outage or in case of individual MUX monthly patching. <BR /> <BR /> So that’s great! We have fantastic fault tolerance for the MUX infrastructure, but how about our Guest VMs that leverage the SLB MUXs? <BR /> <BR /> Well, we have a feature that is most known in the load balancing community as Health Probing, and our implementation is state-of-the-art. In Windows Server 2016 and above, we support both TCP probe-to- port and HTTP probe-to-port and URL. <BR /> <BR /> Unlike traditional load balancer solutions where the probe originates on the appliance and is sent across the wire to the guest IP, SLB probes will originate on the host where that Guest VM IP is located and is sent directly from the SLB Host Agent running on the Hyper-V Host to the VM IP. This eliminates wire traffic and spreads the overhead of conducting health probes between the Hyper-V hosts within the SDN-enabled cluster. <BR /> <H2> How much performance can I expect from the load balancers? </H2> <BR /> Direct Server Return (DSR) is a fantastic feature. In the two scenarios below, you’ll see this in action. For external traffic, DSR can eliminate most of the outbound traffic going through a SLB MUX as it will send directly from the Hyper-V Host to the top-of-rack switch\router. For internal load balancing, it can eliminate most traffic being received at the load balancer infrastructure and will be strictly VM to VM traffic after the initial packet. Let’s look at these scenarios: <BR /> External Load Balancing <BR /> For a Public Virtual IP (VIP) load balancing scenario, the initial packet will arrive at our public VIP on the Top of Rack Switch/Router, which will then be routed to one of our SLB MUXs, and then onto the host, and to the individual tenant VM. Now, on the outbound path, egress packet avoids the MUX infrastructure all together since the Hyper-V host has performed NAT on the packet and routed directly to the Top of Rack Switch. This increases available bandwidth for tenant and infrastructure workloads by 50% when compared to other appliances and solutions. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75275i417C9BE66EC150D9" /> <BR /> <OL> <BR /> <LI> Internet traffic routed to a Public VIP comes in through the Top-of-Rack switch\router, and, then using ECMP, a SLB MUX VM is chosen in which to route the traffic. </LI> <BR /> <LI> The SLB MUX VM then finds what Dynamic IPs (DIPs – the actual IPs of the VMs) the Public VIP is associated with. One of the DIPs is chosen, the traffic is encapsulated into VXLAN packets, and is then sent to the Hyper-V Host which owns the VM with the chosen DIP. </LI> <BR /> <LI> The Hyper-V Host receives the packets, removes the VXLAN encapsulation, and routes it to the VM. </LI> <BR /> <LI> When the VM sends a response packet, it is intercepted by the Hyper-V Host’s virtual switch, the response packet is re-written with the Public VIP IP, and routed directly to the Top-of-Rack switch\router bypassing the SLB MUX VMs. This results in massive scalability as DSR eliminates the SLB MUX VM(s) from being a bottleneck for return traffic. </LI> <BR /> </OL> <BR /> Internal Load Balancing <BR /> During the internal load balancing scenario, the initial packet will flow to the internal VIP, the SLB MUX will find the DIPs (guest VMs), encapsulate the packet using VXLAN, and send to the host which removes the encapsulation and forwards to the DIP, i.e. Tenant VM. Now, the best part, all traffic after this initial internal load balancing scenario will avoid the MUX and perform VM to VM traffic until a health event occurs such as a probe failure, etc. This can eliminate a large percentage of internal load balancing traffic. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75276i733427EF576C2708" /> <BR /> <OL> <BR /> <LI> The first internal VIP request goes through the SLB MUX to pick a DIP. </LI> <BR /> <LI> The SLB MUX detects that the source and destination are on the same VM Network and then the MUX sends a redirect packet to the source host. </LI> <BR /> <LI> The source host then sends subsequent packets for that session direct to the destination. The SLB MUX is bypassed completely! </LI> <BR /> </OL> <BR /> <H2> How do I grant my business units access to a jump box within an isolated vNET?&nbsp; Could I also grant Internet Access to all of the VMs without using a Gateway Connection? </H2> <BR /> If you have ever created a virtual machine in Microsoft Azure, you will have a Public IP and a Private IP. The private IP is used for Intra vNet traffic in Azure or can be used for Express Route and/or Site to Site. The public IP, however, is a NAT interface that you can expose RDP 3389 on. SDN has the same functionality to both inbound and outbound NAT. Outbound NAT is especially useful to give all your VMs within a vNet, internet access, but you do not need a Gateway connection for each vNet! <BR /> Inbound NAT <BR /> Let’s walk through how inbound NAT occurs: NAT will not terminate within the load balancer but on the Hyper-V host itself. When the Public VIP is created and configured, along with an external port, the SLB MUXs will start advertising the VIP by updating the routes using BGP to the Top of Rack switch. When a packet is destined for the Public VIP, it will forward this to an available MUX which will look up the DIPs and encapsulate the packet, using VXLAN to be forwarded to the Hyper-V host. The Hyper-V host will remove the encapsulation and re-write the packet, so the destination is now the DIP and internal port that you wish to use. <BR /> <BR /> A great use of this feature that we see from the field is the “Our infrastructure team wishes to allow a business unit RDP access to multiple VMs inside of the ‘Finance’ vNet.” Within VMM, the infrastructure team can assign separate Public ports, I.e. 3340, 3341, etc. that still have the same back end port of 3389, but to different DIPs. This fulfills the requirement of RDP to a few jump boxes inside the vNet. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75277iB27335523DB9EBEF" /> <BR /> <H2> Can I use SDN Software Load Balancers on VMs that are not using Hyper-V Network Virtualization? </H2> <BR /> Yes! In some organizations, the extra configuration required for Hyper-V Network Virtualization (HNV) as well as the need for SDN RAS Gateways for HNV enabled networks to be configured so that VMs can communicate with the physical network can be overkill. Virtual Machines that are not using HNV VM Networks can still take advantage of SDN load balancing. <BR /> <BR /> Microsoft Network Load Balancer can also be used, but it does not come close to providing all the robust features and scalability that SDN SLB provides, as mentioned above. <BR /> <BR /> If the following criteria is met, SDN SLB can be used on non-HNV VMs: <BR /> <UL> <BR /> <LI> Top of Rack Switch is BGP capable </LI> <BR /> <LI> Network Controller is deployed </LI> <BR /> <LI> Hyper-V Hosts are managed by Network Controller </LI> <BR /> <LI> Software Load Balancer MUX VMs have been deployed and onboarded by NC </LI> <BR /> <LI> The VM Networks being used by the VMs that require load balancing are on a defined VLAN and are managed by Network Controller </LI> <BR /> </UL> <BR /> <H2> How do I get started evaluating SDN Software Load Balancers? </H2> <BR /> Deploying SDN has never been easier!&nbsp; As announced during our Top 10 Network Features series <A href="#" target="_blank"> SDN has gone mainstream </A> ! <BR /> <BR /> There are two methods for deploying SDN: <BR /> SDN Express <BR /> SDN Express now includes a GUI (see our <A href="#" target="_blank"> SDN Goes Mainstream post </A> )!&nbsp; You can also deploy via PowerShell for environments not utilizing System Center Virtual Machine Manage (SCVMM). Additional details on how to deploy SDN using SDN Express are located <A href="#" target="_blank"> here </A> and scripts and other resources are in the <A href="#" target="_blank"> Microsoft SDN repository on GitHub </A> . <BR /> System Center Virtual Machine Manager 2016 or higher <BR /> SDN can also be deployed and managed by SCVMM 2016 and higher. Instructions for how to deploy SDN in SCVMM are located <A href="#" target="_blank"> here </A> and scripts and other resources are in the <A href="#" target="_blank"> Microsoft SDN repository on GitHub </A> . <BR /> <H2> How can Microsoft help my enterprise become part of SDN? </H2> <BR /> That's a great question and we are sure glad that our customer asked. There are a few different options listed below: <BR /> Premier Advisory Call <BR /> Ask your Technical Account Manager (TAM) who is assigned to your account to get you in touch with the Microsoft SDN Blackbelt community. We can hold a remote advisory call to discuss prerequisites and ensure that it will meet the requirements of your business. This is also a great time for a Q &amp; A session! <BR /> <BR /> <A href="#" target="_blank"> Premier WorkshopPLUS: Windows Server: Software Defined Networking </A> <BR /> <BR /> This workshop is a full 4-day workshop that walks through planning, architecture, implementation, and operation of an SDN-enabled hybrid cloud. It includes labs that are hosted on our Learn on Demand platform, simply bring-your-own device and you gain access to all the content and labs. Also, coming towards end of this year, our Unified Support customers will have access to all the Blended Learning Unit (BLUs) video recordings we completed. It’s sort of like a Bill and Kyle SDN on-demand channel! <BR /> SDN Blackbelt Community <BR /> The SDN Blackbelt community is also here to assist remotely. We can certainly have an advisory call as mentioned and that should be your first step. However, if you have a quick question or need assistance, send us a quick note at SDNBlackbelt@microsoft.com and one of us will get back to you. <BR /> <H2> Summary </H2> <BR /> We hope you found this blog to be useful and the scenarios beneficial. There are some fantastic features gained from implementing SDN including the battle-tested and performant Software Load Balancing included in your datacenter SKU. Stay tuned for more <STRONG> Notes from the Field </STRONG> and check the tag below for the full series.&nbsp; We plan to post future blogs that will discuss many other components of SDN! <BR /> <BR /> Stay tuned and see you next time! <BR /> <BR /> Kyle Bisnett and Bill Curtis </BODY></HTML> Thu, 14 Feb 2019 18:09:36 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/notes-from-the-field-microsoft-sdn-software-load-balancers/ba-p/339800 Dan Cuomo 2019-02-14T18:09:36Z Support for LEDBAT: Public Service Announcement https://gorovian.000webhostapp.com/?exam=t5/networking-blog/support-for-ledbat-public-service-announcement/ba-p/339796 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Oct 05, 2018 </STRONG> <BR /> Don't forget to <A href="#" target="_blank"> #LEDBAT </A> and <A href="#" target="_blank"> @Win10Transports </A> <BR /> <BR /> There is buzz on the IT Blogs &amp; Boards that LEDBAT isn’t working as advertised on Windows Server 2016 and up – this is easily explained and is the result of a misconfiguration that is also easily remedied. <BR /> <BR /> The symptoms of the misconfiguration are that LEDBAT gets stuck in a slow transfer mode and will not recover unless you restart the connection. In other words, it does not leverage the unused bandwidth that is available on the network. If your LEDBAT connections are experiencing really low throughput even though there is bandwidth available this is probably the reason. <BR /> <BR /> The problem has to do with <A href="#" target="_blank"> TCP templates </A> . In order to work properly, LEDBAT has to be configured using the <EM> InternetCustom </EM> template. In the misconfiguration LEDBAT is configured using the <EM> DatacenterCustom </EM> template. The good news is that there is a simple way to check your configuration as well as an easy fix. <BR /> <BR /> There are two powershell commands used to configure LEDBAT. Set-NetTCPSettings and New-NetTransportFilter. The NetTCPSetting is used to configure the InternetCustom template for LEDBAT and the NetTransportFilter is used to guide LEDBAT connections into the InternetCustom template. <BR /> <BR /> NetTransportFilters use IP address and port numbers to guide connections to a template. SCCM uses ports 80 and 443 so let’s use those for an example. Go ahead and try it out. Open an elevated powershell window and type Get-NetTransportFilter. <BR /> <BR /> <STRONG> *** There is an “SCCM 1806 hotfix rollup” that will fix this issue for new configurations (but if you already configured it, after installing the 1806 hotfix rollup just make sure you disable and then enable LEDBAT in SCCM).&nbsp; However, if you are manually configuring keep reading ;). </STRONG> <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Get-NetTransportFilter <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Automatic <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 0 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : DatacenterCustom &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;-- Bad configuration, should be InternetCustom </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 443 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 443 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : DatacenterCustom &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;-- </STRONG> <STRONG> Bad configuration, should be InternetCustom </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 80 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 80 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> </BLOCKQUOTE> <BR /> The first thing we see is that the server is misconfigured for SCCM (port 80 and port 443). Do you see where the output says <STRONG> SettingName: <EM> DatacenterCustom </EM> </STRONG> ? Those should say <STRONG> SettingName: <EM> InternetCustom </EM> . </STRONG> This LEDBAT is probably unable to leverage unused bandwidth because of this bad configuration. <BR /> <BLOCKQUOTE> <BR /> *** Don’t worry about the automatic template and certainly don’t delete it! If you have read my tutorial on <A href="#" target="_blank"> TCP Templates </A> then you already know that this template is used to switch between Datacenter and Internet. <BR /> </BLOCKQUOTE> <BR /> Cool, now we are getting somewhere! Let’s take a look at those templates next. Go ahead and try it: <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting | Select Settingname, CongestionProvider <BR /> <BR /> Settingname&nbsp;&nbsp;&nbsp;&nbsp; CongestionProvider <BR /> -----------&nbsp;&nbsp;&nbsp;&nbsp; ------------------ <BR /> Automatic <BR /> <STRONG> InternetCustom&nbsp;&nbsp; CTCP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;-- </STRONG> <STRONG> Bad configuration, should be LEDBAT </STRONG> <BR /> <STRONG> DatacenterCustom LEDBAT&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;-- </STRONG> <STRONG> Bad configuration, should be CTCP (WS2016) or Cubic (WS2019) </STRONG> <BR /> Compat&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NewReno <BR /> Datacenter&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCTCP <BR /> Internet&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CTCP <BR /> </BLOCKQUOTE> <BR /> Once again, the server is misconfigured. <EM> DatacenterCustom </EM> template is configured for LEDBAT and <EM> InternetCustom </EM> template is configured for CTCP (the old default). <BR /> <BR /> Now all we have to do is fix it! First let’s remove the bad NetTransportFilters: <BR /> <BLOCKQUOTE> <BR /> ### Remove DatacenterCustom filters <BR /> PS C:\Users\dahavey&gt; Remove-NetTransportFilter -SettingName DatacenterCustom <BR /> <BR /> Confirm <BR /> Are you sure you want to perform this action? <BR /> Performing operation "Remove" on Target "NetTransportFilter -SettingName DatacenterCustom -Protocol TCP -DestinationPrefix * <BR /> -LocalPortStart 443 -LocalPortEnd 443 -RemotePortStart 0 -RemotePortEnd 65535" <BR /> [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): y <BR /> <BR /> Confirm <BR /> Are you sure you want to perform this action? <BR /> Performing operation "Remove" on Target "NetTransportFilter -SettingName DatacenterCustom -Protocol TCP -DestinationPrefix * <BR /> -LocalPortStart 80 -LocalPortEnd 80 -RemotePortStart 0 -RemotePortEnd 65535" <BR /> [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): y <BR /> <BR /> <BR /> </BLOCKQUOTE> <BR /> Let’s have a look and see how it worked! <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Get-NetTransportFilter <BR /> <BR /> SettingName&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;: Automatic <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 0 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> </BLOCKQUOTE> <BR /> Good work! The bad configuration is gone. Now let’s replace it with a good configuration. Here we go: <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; New-NetTransportFilter -SettingName InternetCustom -Protocol TCP -LocalPortStart 443 -LocalPortEnd 443 -RemotePortStart 0 -RemotePortEnd 65535 <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;-- </STRONG> <STRONG> Good configuration </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 443 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 443 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> <BR /> <BR /> PS C:\Users\dahavey&gt; New-NetTransportFilter -SettingName InternetCustom -Protocol TCP -LocalPortStart 80 -LocalPortEnd 80 -RemotePortStart 0 -RemotePortEnd 65535 <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;-- </STRONG> <STRONG> Good configuration </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 80 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 80 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> </BLOCKQUOTE> <BR /> Looking good! We have the NetTransportFilters correctly configured. Let’s just verify that: <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Get-NetTransportFilter <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Automatic&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;-- </STRONG> <STRONG> Don’t worry about this configuration </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 0 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;-- </STRONG> <STRONG> Good configuration </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 443 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 443 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> <BR /> <STRONG> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;-- </STRONG> <STRONG> Good configuration </STRONG> <BR /> Protocol&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : TCP <BR /> LocalPortStart&nbsp;&nbsp; : 80 <BR /> LocalPortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 80 <BR /> RemotePortStart&nbsp;&nbsp; : 0 <BR /> RemotePortEnd&nbsp;&nbsp;&nbsp;&nbsp; : 65535 <BR /> DestinationPrefix : * <BR /> </BLOCKQUOTE> <BR /> Beautiful! Our NetTransportFilters are looking good! Now let’s take a look at those templates. <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Set-NetTCPSetting -SettingName InternetCustom -CongestionProvider LEDBAT <BR /> PS C:\Users\dahavey&gt; Set-NetTCPSetting -SettingName DatacenterCustom -CongestionProvider Cubic <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting -SettingName DatacenterCustom, InternetCustom | Select Settingname, CongestionProvider <BR /> <BR /> Settingname&nbsp;&nbsp;&nbsp;&nbsp; CongestionProvider <BR /> -----------&nbsp;&nbsp;&nbsp;&nbsp; ------------------ <BR /> <STRONG> InternetCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LEDBAT </STRONG> <BR /> <STRONG> DatacenterCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CTCP &lt;-- Or Cubic if you are using WS2019 </STRONG> <BR /> </BLOCKQUOTE> <BR /> Now we are correctly configured for LEDBAT on SCCM! Happy LEDBATing ;)</img>! <BR /> <BR /> <BR /> <BR /> <STRONG> **** We backported LEDBAT (and Cubic) to WS 2016 for you, but, you will have to either use Windows Update or install the two KBs manually to get the backport.&nbsp; (thanks for calling that out Eric!) </STRONG> <BR /> <STRONG> <A href="#" target="_blank"> https://www.catalog.update.microsoft.com/Search.aspx?q=KB4132216 </A> </STRONG> <BR /> <STRONG> <A href="#" target="_blank"> https://www.catalog.update.microsoft.com/Search.aspx?q=KB4284833 </A> </STRONG> <BR /> <BR /> </BODY></HTML> Thu, 14 Feb 2019 18:09:12 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/support-for-ledbat-public-service-announcement/ba-p/339796 Dan Cuomo 2019-02-14T18:09:12Z TCP Templates for Windows Server 2019 – How to tune your Windows Server Transports (Advanced users only 😉) https://gorovian.000webhostapp.com/?exam=t5/networking-blog/tcp-templates-for-windows-server-2019-8211-how-to-tune-your/ba-p/339795 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Oct 03, 2018 </STRONG> <BR /> Don't forget to <A href="#" target="_blank"> #LEDBAT </A> and <A href="#" target="_blank"> @Win10Transports </A> <BR /> <BR /> Windows TCP parameters can be &nbsp;configured using templates. However, Windows TCP template tuning is one of the deepest darkest mysteries throughout the land. There is only one solution to this problem. Simplicitas ad redigendum (according to Bing) this is Latin for “Reduce to simplicity”. Good thing that TCP templates are simpler than Latin! <BR /> <BR /> Ready? Set! Let’s get started! First, we need to understand the Powershell cmdlets &nbsp;that we will use. There are two of them: <STRONG> <A href="#" target="_blank"> Get-NetTCPSetting </A> </STRONG> and <STRONG> <A href="#" target="_blank"> Set-NetTCPsetting </A> </STRONG> . Let’s start with <STRONG> Get-NetTCPSetting </STRONG> . Go ahead. Open a powershell window, type the cmdlet and pipe it through the Select command as shown in the example. You should see something like this: <BR /> <BLOCKQUOTE> <BR /> PS C:\WINDOWS\system32&gt; Get-NetTCPSetting | Select SettingName <BR /> SettingName <BR /> ----------- <BR /> <STRONG> Automatic </STRONG> <BR /> InternetCustom <BR /> DatacenterCustom <BR /> <STRONG> Compat </STRONG> <BR /> Datacenter <BR /> Internet <BR /> </BLOCKQUOTE> <BR /> Why are those two templates in strikethrough font? Because those are two templates that you don’t have to worry about! The Automatic template is used for (automatically) switching between Internet and Datacenter templates. The Compat template is only for legacy applications and is not recommended for use with modern apps. Now we are down to four templates and this is getting closer to simplicity ;)</img>! <BR /> <BR /> [caption id="attachment_7255" align="aligncenter" width="973"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75274i41CB995706D35C46" /> Figure 1 -- TCP Template SettingNames[/caption] <BR /> <BR /> Referring to Figure 1 we see that there are really only 2 templates that can be customizable or not. The Internet template is used for connections with an RTT of more than 10 ms and the Datacenter template is used for connections with an RTT of 10 ms or less. Remember that Automatic template that I said you don’t need to worry about? Well you still don’t need to worry about it. But, just for information sake the Automatic template is taking the initial RTT as measured by the TCP connection handshake and applying the appropriate template to the TCP connection. <BR /> <BR /> What’s the difference between the two? The Datacenter template is designed for low-latency LAN environments and the Internet template is designed for higher latency WAN environments. Now let’s have a look at all the settings that you can tune! <BR /> <BLOCKQUOTE> <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting -SettingName InternetCustom <BR /> <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom <BR /> MinRto(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 300 <BR /> InitialCongestionWindow(MSS)&nbsp;&nbsp; : 10 <BR /> CongestionProvider&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : CUBIC <BR /> CwndRestart&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : False <BR /> DelayedAckTimeout(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 40 <BR /> DelayedAckFrequency&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2 <BR /> MemoryPressureProtection&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> AutoTuningLevelLocal&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Normal <BR /> AutoTuningLevelGroupPolicy&nbsp;&nbsp;&nbsp;&nbsp; : NotConfigured <BR /> AutoTuningLevelEffective&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Local <BR /> EcnCapability&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> Timestamps&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> InitialRto(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 3000 <BR /> ScalingHeuristics&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> DynamicPortRangeStartPort&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 49152 <BR /> DynamicPortRangeNumberOfPorts&nbsp;&nbsp; : 16384 <BR /> AutomaticUseCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> NonSackRttResiliency&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> ForceWS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> MaxSynRetransmissions&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2 <BR /> AutoReusePortRangeStartPort&nbsp;&nbsp;&nbsp;&nbsp; : 0 <BR /> AutoReusePortRangeNumberOfPorts : 0 <BR /> </BLOCKQUOTE> <BR /> WoW! Look at all those settings you can tune! That is enough to make an uber geek giggle with joy! Use <A href="#" target="_blank"> Set-NetTCPSetting </A> to change things. Like this: <BR /> <BLOCKQUOTE> <BR /> ### Change the congestion provider to LEDBAT <BR /> PS C:\Users\dahavey&gt; Set-NetTCPSetting -SettingName InternetCustom -CongestionProvider LEDBAT <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting -SettingName InternetCustom <BR /> <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom <BR /> MinRto(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 300 <BR /> InitialCongestionWindow(MSS)&nbsp;&nbsp; : 10 <BR /> <STRONG> CongestionProvider&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;: LEDBAT&nbsp;&nbsp; ### </STRONG> <STRONG> &lt; </STRONG> <STRONG> ------- Changed </STRONG> <BR /> CwndRestart&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : False <BR /> DelayedAckTimeout(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 40 <BR /> DelayedAckFrequency&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2 <BR /> MemoryPressureProtection&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> AutoTuningLevelLocal&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Normal <BR /> AutoTuningLevelGroupPolicy&nbsp;&nbsp;&nbsp;&nbsp; : NotConfigured <BR /> AutoTuningLevelEffective&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Local <BR /> EcnCapability&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> Timestamps&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> InitialRto(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 3000 <BR /> ScalingHeuristics&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> DynamicPortRangeStartPort&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 49152 <BR /> DynamicPortRangeNumberOfPorts&nbsp;&nbsp; : 16384 <BR /> AutomaticUseCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> NonSackRttResiliency&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> ForceWS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> MaxSynRetransmissions&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2 <BR /> AutoReusePortRangeStartPort&nbsp;&nbsp;&nbsp;&nbsp; : 0 <BR /> AutoReusePortRangeNumberOfPorts : 0 <BR /> </BLOCKQUOTE> <BR /> Fair warning! If you do not know what a setting means you it’s probably best to leave it alone.&nbsp; If you get into trouble and want to reset to default: <BR /> <BLOCKQUOTE> <BR /> ### Reset tcp parameters to default <BR /> PS C:\Users\dahavey&gt; netsh int tcp reset <BR /> Reset of all TCP parameters OK! <BR /> Ok. <BR /> <BR /> PS C:\Users\dahavey&gt; Get-NetTCPSetting -SettingName InternetCustom <BR /> <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : InternetCustom <BR /> MinRto(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;: 300 <BR /> InitialCongestionWindow(MSS)&nbsp;&nbsp; : 10 <BR /> <STRONG> CongestionProvider&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : CUBIC&nbsp;&nbsp; ### </STRONG> <STRONG> &lt; </STRONG> <STRONG> ------- Reset to default </STRONG> <BR /> CwndRestart&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : False <BR /> DelayedAckTimeout(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 40 <BR /> DelayedAckFrequency&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2 <BR /> MemoryPressureProtection&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> AutoTuningLevelLocal&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Normal <BR /> AutoTuningLevelGroupPolicy&nbsp;&nbsp;&nbsp;&nbsp; : NotConfigured <BR /> AutoTuningLevelEffective&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Local <BR /> EcnCapability&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> Timestamps&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> InitialRto(ms)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;: 3000 <BR /> ScalingHeuristics&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> DynamicPortRangeStartPort&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 49152 <BR /> DynamicPortRangeNumberOfPorts&nbsp;&nbsp; : 16384 <BR /> AutomaticUseCustom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> NonSackRttResiliency&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Disabled <BR /> ForceWS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Enabled <BR /> MaxSynRetransmissions&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 2 <BR /> AutoReusePortRangeStartPort&nbsp;&nbsp;&nbsp;&nbsp; : 0 <BR /> AutoReusePortRangeNumberOfPorts : 0 <BR /> </BLOCKQUOTE> <BR /> Please see the links to <A href="#" target="_blank"> Set/Get-NetTCPSettings </A> for descriptions of the individual settings and what they do. Have fun and happy TCP tuning! </BODY></HTML> Thu, 14 Feb 2019 18:09:09 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/tcp-templates-for-windows-server-2019-8211-how-to-tune-your/ba-p/339795 Dan Cuomo 2019-02-14T18:09:09Z Top 10 Networking Features in Windows Server 2019: Wrapping up! https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-wrapping-up/ba-p/339793 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Sep 26, 2018 </STRONG> <BR /> The past ten weeks have been energizing for us a team, as we’ve had the opportunity to share the networking investments and innovation with Windows Server 2019, and relish the excitement many of you have shared about the product. I hope you enjoyed our blog series leading up to Ignite as much as we enjoyed writing them! <BR /> <BR /> Here's the Top 10 Networking Features in Windows Server 2019 roll call!&nbsp; To read the full list, please see <A href="#" target="_blank"> this link </A> . <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75273i5134510705585A97" /> <BR /> <BR /> A special shout-out to our fantastic MVPs, TAP customers, and Partners who have tirelessly worked with us along the way to ensure readiness of this technology suite for our valuable customers. <BR /> <BR /> If you're at Ignite in Orlando, Fl. this week, please stop by booth AIC125 and bring your feedback and questions! <BR /> <BR /> I look forward to our ongoing collaboration on networking enhancements to enable the journey to the hybrid cloud, across apps and infrastructure. <BR /> <BR /> Ravi Rao (on behalf of the Core Networking team at Microsoft) </BODY></HTML> Thu, 14 Feb 2019 18:08:58 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-wrapping-up/ba-p/339793 Dan Cuomo 2019-02-14T18:08:58Z Top 10 Networking Features in Windows Server 2019: #1 Container Networking with Kubernetes https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-1-container/ba-p/339791 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Sep 19, 2018 </STRONG> <BR /> <STRONG> Share On: <A href="#" target="_blank"> Twitter </A> </STRONG> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. This concludes our <BR /> Top 10 List. We hope to see you at Ignite next week! <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> <BR /> In today’s increasingly competitive and fast-paced technology market, enterprises are constantly discovering amazing new ways to innovate and evolve. One such area with expanding interest in recent years is application modernization using <A href="#" target="_blank"> containers </A> and <A href="#" target="_blank"> container orchestration </A> . The numbers speak for themselves -- a <A href="#" target="_blank"> recent press release </A> (06/18) by Allied Market Research concluded that: <BR /> <P> <EM> <STRONG> The global application container market was valued at $698 million in 2016, and is projected to reach $8.2 billion by 2025. </STRONG> </EM> </P> <BR /> As applications are <A href="#" target="_blank"> lifted-and-shifted </A> from VMs to containers, IT Pros and Dev Ops teams require the same network management agility of Software-Defined Datacenter ( <A href="#" target="_blank"> SDDC </A> ). Kubernetes, the de facto container orchestration tool, addresses this gap under the umbrella of a standardized &amp; open-sourced framework. <BR /> <BR /> In Kubernetes version 1.9 with Windows Server, version 1709 we first announced <A href="#" target="_blank"> beta for Windows Server containers </A> . Now, with Windows Server 2019, we greatly improved usability of Kubernetes on Windows by <EM> enhancing platform networking resiliency and support of container networking plugins </EM> . <EM> </EM> <BR /> <BR /> Additionally, customers deploying workloads on Kubernetes demand network security to protect both Linux and Windows services using embedded tooling. The Windows Networking team has been working closely with Tigera, who is an industry-recognized leader in this space, and is pleased to announce upcoming availability of <EM> Tigera Calico for Windows </EM> . Both companies are working jointly with TAP customers to deploy Calico on Windows in POC environments, with current focus on network policy enforcement. Network management using dynamic routing (BGP) and IPAM is also on Tigera’s roadmap, with forthcoming Calico CNI support on Windows. <BR /> <H2> Kubernetes + Windows Server 2019 </H2> <BR /> The Windows Networking team (together with the Kubernetes community) has done tremendous work on both the platform and open-source front to enable a smooth interoperability between Windows and other first-class citizens belonging to the Kubernetes project. <BR /> <BR /> Windows Server 2019 supports all the Kubernetes networking building blocks (“primitives”), such that you’re able to deploy mixed-OS Kubernetes clusters in the environment of your choice. Whether you’re looking for maximum control in your own on-premises datacenter, or conveniently getting it all provisioned on Azure infrastructure – all the networking pieces are ready for composing your own cluster <EM> now </EM> . <BR /> <BR /> Here is a timeline summarizing the groundbreaking achievements that enabled Windows to pursue its very own "Kubistential" awakening: <BR /> <BR /> [caption id="attachment_7165" align="alignnone" width="1916"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75269i3DABB1AFA629DE1F" /> Figure 1: Kubernetes improvements since Windows Server 2016[/caption] <BR /> <BR /> The major headline from the graphic above, is that Kubernetes for Windows is projected to GA with Kubernetes 1.13, including official support on Windows Server 2019. Considering that Kubernetes is the <A href="#" target="_blank"> most actively discussed GitHub project </A> <EM> in the world </EM> today, this means you have a really compelling new force gravitating you towards Kubernetes on Windows: <EM> <STRONG> both </STRONG> open-source community <STRONG> and </STRONG> platform-level support from Microsoft enterprise </EM> . <BR /> <BR /> Equally exciting to us, is that customers and users are also beginning to see these incremental improvements: <BR /> <P> <EM> <STRONG> "My windows node successfully joined the cluster and I’m able to schedule a pod on it.&nbsp; Definitely a victory." </STRONG> </EM> </P> <BR /> <P> <EM> – Nikhil Shampur, ESRI. </EM> </P> <BR /> <P> <EM> <STRONG> "To me, it’s great to see that Kubernetes on Windows is now working so smoothly!" </STRONG> </EM> </P> <BR /> <P> <EM> - Ulrich Rabenstein, SAP, Developer </EM> </P> <BR /> Let's make things more concrete through the lens of a (simplified) deployment example to demonstrate how Kubernetes features (all supported on Windows Server 2019) address the needs of an enterprise. <BR /> <H2> Deploying Kubernetes on Windows Server 2019: A reference example </H2> <BR /> Consider a Windows-based, .NET application consisting of an ordering service, location service, and identity service. Here are some basic, yet plausible example needs: <BR /> <OL> <BR /> <LI> I can safeguard confidential data by defining network security policies to control access to my workloads. </LI> <BR /> <LI> I need a highly available system that is always accessible (zero downtime). </LI> <BR /> <LI> My applications can scale with demand during high-traffic periods (Cyber Monday for example). </LI> <BR /> <LI> My Windows services can communicate with my Linux services, and vice-versa. </LI> <BR /> </OL> <BR /> [caption id="attachment_6705" align="alignnone" width="972"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75270i1C9D0DBC3FC22B72" /> Figure 2: Monolithic application[/caption] <BR /> <BR /> In theory, we could take this application and simply run it in a container, with each of these services communicating via shared networking and compute resources. However, in pursuit of technical benefits such as high cohesion and low coupling (see <A href="#" target="_blank"> microservices architecture </A> ), work could also be done to refactor the application, and split it up into more manageable components. <BR /> <BR /> [caption id="attachment_6715" align="alignnone" width="1101"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75271i536AB6CF3DA07F09" /> Figure 3: Microservices application[/caption] <BR /> <BR /> In doing so, it quickly becomes apparent that the most effective way to redesign the architecture and standardize this distributed system, is through a container orchestrator. Why? Well, Kubernetes already provides native features that satisfy these requirements, it enhances portability in case of future changes, and there is no need to reinvent the wheel. Here's the Kubernetes solution: <BR /> <OL> <BR /> <LI> I can safeguard confidential data by defining network security policies to control access to my workloads. <BR /> <UL> <BR /> <LI> <STRONG> Solution: </STRONG> Use the <A href="#" target="_blank"> network policy feature </A> that gives you a granular way to restrict traffic and isolate containers running your workloads. <BR /> <UL> <BR /> <LI> (See an example of this on Windows Server 2019 <A href="#" target="_blank"> here </A> ). </LI> <BR /> </UL> <BR /> </LI> <BR /> </UL> <BR /> </LI> <BR /> <LI> I need a highly available system that is always accessible (zero downtime). <BR /> <UL> <BR /> <LI> <STRONG> </STRONG> <STRONG> Solution: </STRONG> Use the <A href="#" target="_blank"> deployment feature </A> for health monitoring + automatic replication of your containers in case of failure. <BR /> <UL> <BR /> <LI> (See an example of this on Windows Server 2019 <A href="#" target="_blank"> here </A> ). </LI> <BR /> </UL> <BR /> </LI> <BR /> </UL> <BR /> </LI> <BR /> <LI> My applications can scale with demand during high-traffic periods (Cyber Monday for example). <BR /> <UL> <BR /> <LI> <STRONG> Solution: </STRONG> Use the <A href="#" target="_blank"> service feature </A> together with <A href="#" target="_blank"> deployments </A> to define a set of load-balanced pods that can easily be scaled to handle incoming network traffic. <BR /> <UL> <BR /> <LI> <STRONG> </STRONG> (See an example of this on Windows Server 2019 <A href="#" target="_blank"> here </A> ). </LI> <BR /> </UL> <BR /> </LI> <BR /> <LI> * <STRONG> Tip*: </STRONG> Stay tuned for a more automated version of this solution called <A href="#" target="_blank"> HPA </A> (horizontal pod autoscaling), also <A href="#" target="_blank"> coming soon </A> with Windows GA. </LI> <BR /> </UL> <BR /> </LI> <BR /> <LI> My Windows services can communicate with my Linux services, and vice-versa. <BR /> <UL> <BR /> <LI> <STRONG> Solution: </STRONG> See <A href="#" target="_blank"> connecting application services </A> about Kubernetes service discovery that allows services on any OS to communicate with each other via IP or name. <BR /> <UL> <BR /> <LI> (See an example of this on Windows Server 2019 <A href="#" target="_blank"> here </A> ). </LI> <BR /> </UL> <BR /> </LI> <BR /> </UL> <BR /> </LI> <BR /> </OL> <BR /> [caption id="attachment_6735" align="alignleft" width="1414"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75272iDA6FC62D3DDAD083" /> Figure 4: Containerized application on Kubernetes[/caption] <BR /> <BR /> Notice how most of these requirements are not really unique to our simplified example, but applicable to many other domains. Lower maintenance outages thanks to a fault-tolerant solution designed with failure in mind, or a more responsive, light-weight infrastructure are universal needs. <EM> </EM> <BR /> <BR /> Here is also a video that we took to demonstrate the aforementioned features in action: <BR /> <BR /> <IFRAME frameborder="0" height="504" src="https://www.youtube.com/embed/j2B7cLdTXMw" width="896"> </IFRAME> <BR /> <UL> <BR /> <LI> <A href="#" target="_blank"> 3:51 - Load balancing across a set of pods </A> </LI> <BR /> <LI> <A href="#" target="_blank"> 6:01 - Connecting services across Windows + Linux </A> </LI> <BR /> <LI> <A href="#" target="_blank"> 8:33 - Network policy enforcement on Windows </A> </LI> <BR /> </UL> <BR /> <H2> Try it out today! </H2> <BR /> Ready to get started?&nbsp; Great, let's take a look at the different avenues to get started today! <BR /> Option 1: Do-it-yourself deployment (on-premise) <BR /> Description <BR /> Perhaps the most challenging option to deploy (albeit giving you most control), is to just develop and deploy Kubernetes yourself on your own datacenter. This has the advantage that it reduces reliance on third parties and keeps proprietary data on your own infrastructure. For this, Windows supports Flannel CNI (in host-gateway mode) for route management, as well as Calico CNI for network policy enforcement. For maximum networking control, you can also just program static routes yourself and leverage the “wincni” container networking plugin. <BR /> Requires <BR /> <UL> <BR /> <LI> You have a computer or VM (requires virtualization and MAC spoofing) running Windows Server 2019. </LI> <BR /> <LI> You have a computer or VM running a Linux OS which supports Kubernetes (needed for master node). </LI> <BR /> </UL> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? Download the latest Insider build and try it out: <BR /> <A href="#" target="_blank"> Flannel (host-gw), recommended for starting out! </A> <A href="#" target="_blank"> Wincni + manual route mangement </A> <A href="#" target="_blank"> Calico (beta) for network policy enforcement </A> <BR /> <BR /> <BR /> <BR /> In the future <BR /> <UL> <BR /> <LI> Simplified &amp; scalable network configuration thanks to network management and IPAM provided by Calico CNI! </LI> <BR /> <LI> Overlay networking on Flannel! </LI> <BR /> <LI> DNS support for multiple namespaces! </LI> <BR /> </UL> <BR /> <BR /> Option 2: Deploy a Kubernetes cluster on Azure (acs-engine) <BR /> Description <BR /> Acs-engine is an open-source project from Azure that enables you to generate ARM (Azure Resource Manager) templates describing the size, shape, configuration, and version of your Kubernetes cluster. It can then use your template to generate and provision a cluster attached to a private Azure virtual network (Azure vNet) matching your desired description. This solution utilizes <A href="#" target="_blank"> Azure CNI </A> which was specifically developed for integration with the Azure vNet. <BR /> Requires <BR /> <UL> <BR /> <LI> <A href="#" target="_blank"> Azure subscription </A> with service principal profile </LI> <BR /> <LI> <A href="#" target="_blank"> JSON configuration file </A> with cluster description </LI> <BR /> <LI> <A href="#" target="_blank"> Azure CLI </A> or <A href="#" target="_blank"> Azure Powershell </A> </LI> <BR /> <LI> <A href="#" target="_blank"> Acs-engine CLI </A> </LI> <BR /> </UL> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? Download the latest Insider build and: <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> In the future <BR /> <UL> <BR /> <LI> AKS (Azure Kubernetes Service) with Windows Server containers! </LI> <BR /> </UL> <BR /> <H2> What's Next? </H2> <BR /> In addition to the options outlined above, we’ve partnered with RedHat to bring Windows Server containers to the <A href="#" target="_blank"> RedHat OpenShift Container Platform </A> (currently under private developer preview). This enterprise-grade container platform tailors particularly well to convenience-oriented IT Pros that want to forego complex deployment procedures (who doesn’t?!) while managing mixed-OS workloads through a familiar single pane of glass. Here are the most recent announcements from primary sources: <BR /> <UL> <BR /> <LI> <A href="#" target="_blank"> RedHat OpenShift and Microsoft Windows Containers </A> </LI> <BR /> <LI> <A href="#" target="_blank"> RedHat and Microsoft co-develop the first Red Hat OpenShift jointly managed service on a public cloud </A> </LI> <BR /> <LI> <A href="#" target="_blank"> OpenShift on Azure: The easiest, fully managed OpenShift in the cloud </A> </LI> <BR /> </UL> <BR /> <H2> Key Takeaways </H2> <BR /> We covered motivations behind Kubernetes, what it is, as well as how-to deploy it. We also gave an overview of both the platform and open-source improvements since Windows Server 2016. Finally, we gave a brief teaser on what’s coming next in the Kubernetes world from a Windows standpoint. <BR /> <BR /> Even though Kubernetes on Windows will GA soon, the road doesn’t end there. The Windows networking team is dedicated to continue working together with the open-source community to bring more Kubernetes networking goodness and CNI support to Windows. If you are curious about some of the technical platform work that enabled Kubernetes to run on Windows Server 2019 today, I’ll point you to these blog posts that already offer excellent insight: <BR /> <UL> <BR /> <LI> <A href="#" target="_blank"> Windows Networking at Parity with Linux for Kubernetes </A> </LI> <BR /> <LI> <A href="#" target="_blank"> Windows Networking for Kubernetes </A> </LI> <BR /> </UL> <BR /> <EM> Right now </EM> , much of the groundbreaking work is happening on upstream open-source bits, where anyone can contribute! If you want to stay up to date on any announcements, or want to make your voice heard, please join the dedicated Windows Kubernetes community at the <A href="#" target="_blank"> #sig-windows </A> meetups! <BR /> <BR /> Thanks for reading! <BR /> <BR /> David Schott </BODY></HTML> Thu, 14 Feb 2019 18:08:47 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-1-container/ba-p/339791 David Schott 2019-02-14T18:08:47Z Top 10 Networking Features in Windows Server 2019: #2 Propelling broadcast video with DPDK on Windows https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-2-propelling/ba-p/339785 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Sep 12, 2018 </STRONG> <BR /> <STRONG> Share On: <A href="#" target="_blank"> Twitter </A> </STRONG> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <A href="#" target="_blank"> HERE </A> to see the other blogs in this series. <BR /> <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> As the world moves from HD to 4K and other high-resolution media formats (e.g. 8K), media broadcasters are pioneering a transition to an IP-based infrastructure. Designing for the future, this transition requires high bandwidth and low latency networking re-architecture, not to mention state of the art GPU drivers. <BR /> <BR /> We recently announced the availability of <A href="#" target="_blank"> Data Plane Development Kit (DPDK) </A> libraries on Windows to provide user mode applications fast packet processing capabilities, bypassing the host networking stack. <BR /> <P> We are pleased to announce a partnership with Cisco and Intel to accelerate this transition in the media industry, by bringing Windows DPDK to Cisco’s media software package called virtual Media Interface (vMI)! Now, <STRONG> Windows Server with DPDK's express data path and wealth of GPU drivers becomes the platform of choice for delivering next gen media formats </STRONG> and other user-mode applications! </P> <BR /> In this article, we explore the applications, the journey till date and dive deeper into how the video broadcasting transition can leverage DPDK’s fast packet processing capabilities. <BR /> <P> <STRONG> DPDK libraries now available on Windows </STRONG> ! </P> <BR /> <BR /> <H3> Why did we bring DPDK to Windows? </H3> <BR /> In addition to the video broadcasting industry, multiple new workloads require an insatiable amount of network bandwidth with low latency performance. Whether it is real-time gaming, analytics and logic running on the edge, or augmented / virtual reality network performance has never been more critical. The Linux Foundation <A href="#" target="_blank"> DPDK </A> project is designed to meet the needs of these applications by providing fast packet processing capabilities to applications running in user-mode with direct access to NIC resources, thereby bypassing the host networking stack. <BR /> <BR /> We also <A href="#" target="_blank"> demoed DPDK running on Windows </A> with Intel XL710 NICs reaching 70+ million packets per second! Additional Poll Mode Drivers (PMD) for Chelsio and Cavium NICs are planned for Windows. <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75265i1881B9BAEA0C8529" /> </P> <BR /> <P> <A href="#" target="_blank"> Image courtesy </A> </P> <BR /> <BR /> <H3> Professional Media Transition </H3> <BR /> Applications and physical appliances are increasingly being virtualized and moving away from domain-specific solutions with specialized cabling and network fabrics to IP/Ethernet backbones using network function virtualization (NFV). <BR /> <BR /> One such example is the video broadcasting industry undergoing a massive transformation, moving from their existing Serial Digital Interface ( <A href="#" target="_blank"> SDI </A> ) based workflows to Internet Protocol(IP) based workflows. We briefly touched upon this earlier. <BR /> <BR /> Why is this relevant, you ask? A Cisco Video Networking Index (VNI) <A href="#" target="_blank"> study </A> estimates that: <BR /> <P> <EM> <STRONG> IP video traffic will be a staggering 82% of all consumer internet traffic by 2021. </STRONG> </EM> </P> <BR /> A <A href="#" target="_blank"> study </A> by <EM> Strategy Analytics </EM> estimates that: <BR /> <P> <EM> <STRONG> Nearly 50% of the homes in the US will have a 4K television by 2020. </STRONG> </EM> </P> <BR /> Broadcast and professional media vendors are forced to scale their applications and infrastructure to meet the exponentially increasing bandwidth demands as the world moves from high definition (HD) to 4K/UHD and 8K in the future. <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75266iD1ED6DBE239C34F4" /> <BR /> <BR /> This transition driven by <A href="#" target="_blank"> 4K streaming </A> requirements is being fast-tracked as IP over ethernet spans large distances, can be encrypted and is easier to manage relative to SDI. Professional media appliances and applications are challenged as the data rates involved in carrying <A href="#" target="_blank"> SMPTE </A> formatted, IP based audio and video streams exceed the capacity of today’s protocol stacks necessitating a high throughput and low latency data path. As our friends at Cisco highlight – <BR /> <BR /> <EM> <STRONG> While a HD-based video stream consumes 1.5Gb/s, a single HDR/HFR 4K stream can consume 12Gb/s. </STRONG> </EM> <BR /> <BR /> To meet such unprecedented packet processing demands, Windows developers need direct access to the Network Interface Controller hardware without having to go through the host OS networking stack. <A href="#" target="_blank"> Data Plane development kit (DPDK) </A> , managed as an open-source project with a BSD license, was until recently available exclusively on the Linux platform. <BR /> <BR /> We recently announced the availability &nbsp;of the <A href="#" target="_blank"> DPDK core libraries and a generic UIO driver for Windows </A> Platform. <BR /> <BR /> As introduced earlier, ongoing collaborations with Cisco and Intel are underway integrating Cisco’s open sourced vMI (Virtual media Interface) software package with DPDK on Windows to take advantage of the efficient path to the NICs. The challenges that come with the data rates in an IP based network are alleviated with DPDK’s fast user-mode packet processing. Professional media vendors have extensive Windows investments, with the wealth of GPU drivers making a compelling case to also actively seek Windows based solutions. <BR /> <BR /> Prototyping efforts have begun with our friends at Cisco and Intel, taking a step closer to making this a reality. <A href="#" target="_blank"> </A> <BR /> <BR /> <MARK> <A href="#" target="_blank"> <STRONG> Head over to the blog article from Cisco </STRONG> </A> </MARK> to learn more about vMI's integration with Windows DPDK and PCI-Passthrough solution. <BR /> <P> <EM> <B> Cisco, Intel and Microsoft Accelerate the Transition from SDI to IP with an Open Source Toolkit for Media Software Vendors </B> </EM> </P> <BR /> <BR /> <H3> Journey till date </H3> <BR /> The Windows data plane has been constantly evolving over many years with technologies such as RSS spreading and software offloads for the native host. In addition, accelerations into the Hyper-V virtual switch and guest VMs have been possible with technologies like SR-IOV to bypass the host. My friend Dan Cuomo dives deeper into our efforts to accelerate the host and the guest in this <A href="#" target="_blank"> article </A> . <BR /> <BR /> Now, adding a third dimension, Windows is bringing DPDK to accelerate user-mode applications. <A href="#" target="_blank"> </A> <A href="#" target="_blank"> </A> <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75267iC268AD68AEDC96D2" /> <BR /> <H3> DPDK architecture </H3> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75268i180E6D63C5E9C808" /> <BR /> <BR /> <BR /> <BR /> DPDK is a set of libraries and optimized network drivers providing fast packet processing abilities to user-mode applications in real world scenarios delivering the lowest latency and highest packets per second. <BR /> <BR /> This UIO driver provides user-mode applications direct access to the memory mapped to the PCIe bus corresponding to the NIC hardware. The user-mode application links to the DPDK libraries and the UIO driver to access hardware resources. Post-initialization, the Poll mode driver will send/receive packets directly to and from the NIC by passing the kernel network processing. <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <H3> Resources Available </H3> <BR /> DPDK core libraries and generic UIO driver are now available on the Windows Platform. <BR /> <BR /> <A href="#" target="_blank"> <STRONG> Click Here to visit the Windows DPDK draft repository </STRONG> </A> ! <BR /> <BR /> The repository is synched with DPDK v18.02 release and supports 31 out of the 42 core libraries on Windows. These DPDK libraries and sample applications have been tested and validated on Windows Server 2016 and Windows Server 2019 preview builds. <BR /> <BR /> As of today, Poll mode driver for the Intel® Ethernet Converged Network Adapter XL710 40 GbE is available. <BR /> <BR /> Microsoft is working with the Windows NIC eco-system partners to bring DPDK Poll-Mode Drivers (PMD) to Windows to run DPDK applications on a variety of NICs. Intel, Cavium, and Chelsio have either made a PMD available for Windows or plan to do so in the near future. <BR /> <H3> Summary </H3> <BR /> We are deeply invested in making Windows Server with DPDK the preferred platform for applications that can benefit from the fast packet processing capabilities. <BR /> <BR /> Come back for more announcements on integrating Cisco’s vMI with DPDK as we work towards an open source solution in the near future. <BR /> We will also bring you demonstrations and case studies on how Windows DPDK is solving real world challenges. <BR /> <BR /> Windows remains committed to accelerating the application and the data path for our awesome developers. <BR /> <BR /> Thanks to contributions from our counterparts at Cisco and Intel for joining us in this ride…Many more miles to go in the Windows DPDK train…. <BR /> <BR /> Thanks for reading! <BR /> <BR /> Harini Ramakrishnan </BODY></HTML> Thu, 14 Feb 2019 18:08:13 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-2-propelling/ba-p/339785 Harini Ramakrishnan 2019-02-14T18:08:13Z Top 10 Networking Features in Windows Server 2019: #3 Azure Network Adapter https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-3-azure/ba-p/339780 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Sep 05, 2018 </STRONG> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> More and more on-premises workloads require connectivity to Azure resources.&nbsp; Connecting these on-premises workloads to their Azure resources traditionally requires an Express Route, Site-to-Site VPN, or Point-to-Site VPN connection. &nbsp;Each of these options require multiple steps and expertise in both networking and certificate management, and in some cases, infrastructure setup and maintenance. <BR /> <BR /> Now, Windows Admin Center enables a one-click experience to configure a point-to-site VPN connection between an on-premises Windows Server and an Azure Virtual Network.&nbsp; This automates the configuration for the Azure Virtual Network gateway as well as the on-premises VPN client. <BR /> <P> <STRONG> Windows Admin Center and the Azure Network Adapter makes connecting your on-premises servers to Azure a breeze! </STRONG> </P> <BR /> <BR /> <H2> Windows Admin Center </H2> <BR /> This feature relies on the <A href="#" target="_blank"> Windows Admin Center </A> which is an evolution of Windows Server in-box management tools; it’s a single pane of glass that consolidates all aspects of local and remote server management. It comes at no additional cost beyond Windows and is ready to use in production. <BR /> <BR /> Once Windows Admin Center is configured, you are ready to start. <BR /> <H2> Azure Network Adapter </H2> <BR /> The Azure Network Adapter is a new part of the Network extension inside Windows Admin Center which allows you to easily setup a Point-to-Site VPN connection to Azure. <BR /> <BLOCKQUOTE> <BR /> Note: Point-to-Site connections do not require a VPN device or a public-facing IP address. <BR /> For more information about Point-to-Site VPN, see <A href="#" target="_blank"> About Point-to-Site VPN </A> . <BR /> </BLOCKQUOTE> <BR /> Now let’s walk through the experience of adding an Azure Network Adapter to your on-premises Windows Server.&nbsp; You will be able to find the button <STRONG> +Add Azure Network Adapter </STRONG> on the <STRONG> Network extension </STRONG> in Windows Admin Center. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75261iAAB3CA5E92E48B0D" /> <BR /> <BR /> Once you click <STRONG> +Add Azure Network Adapter </STRONG> the <STRONG> Add Azure Network Adapter </STRONG> wizard will appear on the right pane. <BR /> <BR /> When you select any existing Azure Virtual Network, you will find all the values are already automatically filled-in and the <STRONG> Create </STRONG> button is ready for you to click. You can modify the default options selected by the wizard, or just click the <STRONG> Create </STRONG> button to accept the defaults and trigger the Point-to-site VPN connection to Azure. <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75262i343E4C232D9466EE" /> </P> <BR /> <BR /> <BR /> That’s it! After a few minutes you will see the newly created point-to-site VPN connection available in the inventory page. Here is a short animation to show you the steps! <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75263iA3CF4FD3DA20F9DA" /> <BR /> <BLOCKQUOTE> <BR /> Note: The creation could take much longer (~25 minutes) if the Azure Virtual Network <BR /> gateway needs to be created. <BR /> </BLOCKQUOTE> <BR /> <H2> Use and Validate Azure Network Adapter </H2> <BR /> Once your Point-to-site VPN is “Connected” your server now has a connection to the Azure Virtual Network. &nbsp;The server will be able to communicate to any Azure resources in the Virtual Network. <BR /> <BR /> Here’s a simple example of a ICMP Ping validation between one on-premises server and an Azure VM connected through the Azure Network Adapter. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75264i613F19DDA789D851" /> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? Try out Azure Network Adapter in the <A href="#" target="_blank"> Windows Admin Center Version 1809 </A> ! <BR /> Note: Windows Admin Center Version 1809 will be released in September. <BR /> <BR /> <BR /> <BR /> Previously creating hybrid cloud connectivity required expertise in networking, certificate management, and even infrastructure setup and maintenance. &nbsp;Now with the Azure Network Adapter in Windows Admin Center (version 1809), hybrid connectivity can be configured with the click of a button!&nbsp; The Azure Network Adapter automates the configuration of the Azure Virtual Network gateway and VPN client installation for you! <BR /> <BR /> Thanks for reading, <BR /> <BR /> Schumann Ge </BODY></HTML> Thu, 14 Feb 2019 18:07:41 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-3-azure/ba-p/339780 nnamuhcs 2019-02-14T18:07:41Z Top 10 Networking Features in Windows Server 2019: #4 Security with SDN https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-4-security/ba-p/339774 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Aug 29, 2018 </STRONG> <BR /> Share <STRONG> On: <A href="#" target="_blank"> </A> <A href="#" target="_blank"> Twitter </A> Share on: <A href="#" target="_blank"> LinkedIn </A> </STRONG> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> In this modern era of cloud computing, more and more customers are looking to move their workloads to public, private or hybrid clouds. Security is one of their main inhibitors in moving to cloud. How secure are their workloads in the cloud? Is their data safe from theft and tampering? Will it all work with IPv6? <BR /> <BR /> Windows Server 2019 SDN delivers many features to increase customer confidence in running workloads either on-premises or as a service provider in the cloud. These security enhancements are integrated into the comprehensive SDN platform that our customers have already been using since Windows Server 2016. <BR /> <BR /> For more information on general platform and management features, refer to SDN management blog ( <A href="#" target="_blank"> link </A> ) and the hybrid SDN gateway performance blog ( <A href="#" target="_blank"> link </A> ). <BR /> <H2> Encrypted Subnets </H2> <BR /> How many of the legacy applications on your network are using encryption?&nbsp; How many of them are using an encryption method that is still considered secure?&nbsp; Chances are you have some apps that are vulnerable to data theft and tampering. <BR /> <BR /> You could find every app, analyze the encryption and update it, or you could encrypt at the network level with SDN.&nbsp; With SDN network subnet encryption in Windows Server 2019, any packet that leaves a VM is automatically encrypted as it passes to other destinations on the same back-end network.&nbsp; If a vulnerability is found, then the fabric can be updated quickly and all applications automatically gain the necessary level of security. <BR /> <BR /> This is enabled on any of the subnets in a virtual network by specifying an encryption certificate to use and setting "Encryption" to true. <BR /> <P> " <STRONG> As organizations look to enable protection through software defined controls and eliminate complexities, configurations leveraging virtual network encryption greatly enhance security in a simplified manner </STRONG> " </P> <BR /> <P> - <EM> Rand Morimoto, President, Convergent Computing </EM> </P> <BR /> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> <H2> Firewall Logging </H2> <BR /> The ability to microsegment allows you to create isolation boundaries, but how do you know they're working? How can you tell if you're under attack? If a breach has occurred, how can you perform the post-mortem analysis to determine how far it went? <BR /> <BR /> Firewall logging is critical for the ability to do all of the above. <BR /> <BR /> In Windows Server 2019, SDN enables the Hyper-V host to generate Firewall logs that are consistent in format with Azure Network Watcher.&nbsp; This enables the ecosystem of tools that has sprung up around Network Watcher to be easily adapted to work with the Windows Server SDN implementation. <BR /> <BR /> After applying a one-time configuration to the network controller, you simply enable logging on individual Access Control List rules and network flows that match that rule are automatically logged. <BR /> <P> <STRONG> "Windows Server 2019's SDN settings have an extremely helpful firewall-auditing component that can be enabled to log all network communications between SDN connections" </STRONG> </P> <BR /> <P> - <EM> Rand Morimoto, President, Convergent Computing </EM> </P> <BR /> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> <H2> Fabric ACLs </H2> <BR /> Windows Server 2016 provides the ability to lock down the security of your virtual networks by automatically applying ACLs to VMs connected to virtual subnets.&nbsp; Windows Server 2019 expands this capability to the fabric as well, allowing you to restrict access to your infrastructure machines in a way that is more easily managed and automatic, by adding ACLs to the logical subnets.&nbsp; This means that any SDN managed VM connected to a VLAN based network will automatically get the necessary ACLs applied. <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> <H2> Virtual Network Peering </H2> <BR /> The primary security boundary for SDN is the isolation that's provided by the virtual network itself, but sometimes it becomes necessary to breach this boundary so that two virtual networks are able to communicate with each other.&nbsp; This may be the case if you've deployed a Database in one virtual network, but want it to be accessed by other applications that have been deployed in their own separate virtual networks.&nbsp; Virtual Network peering enables just that.&nbsp; It combines the virtual routers in associated virtual network so they can communicate with each other, without having to traverse through a gateway. This enables high throughput, low latency communication between the virtual networks. <BR /> <P> " <STRONG> This is really about making the scenario simpler to deploy / manage and removing the perf overhead.&nbsp; As it happens we have a bunch of scenario’s where this feature will be useful, even in its current form.&nbsp; As you know we run our two primary DC’s as active / active deployments and one of our big bug-ears has been providing this type of scenario, while still facilitating multiple entry points.&nbsp; I can see multiple current workloads scenario’s where this will improve performance, rather than using our current approach of L3 GW’s over the MPLS inter-link </STRONG> " </P> <BR /> <P> - <EM> Philip Moss, Chief Product Officer, Acuutech </EM> </P> <BR /> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> <H2> IPv6 support </H2> <BR /> While you may not want to use IPv6, at some point you may have to and because of that we've added support for IPv6 to SDN.&nbsp; While not a security feature per-se, with Windows Server 2019, SDN includes the ability to use IPv6 for virtual network address spaces, virtual IPs and for logical networks.&nbsp; All of the security features of SDN now work with IPv6 addresses and subnets, including Access Control Lists and User Defined Routing. <BR /> <BR /> To use this feature, download&nbsp;the latest Insider build and use IPv6 subnets on your virtual subnets in the same way that you would use IPv4, and assign IPv6 addresses to your virtual machines. <BR /> <H2> Summary </H2> <BR /> As you can see, we have made a ton of investments in SDN to safeguard the security of your workloads with Windows Server 2019. <BR /> <OL> <BR /> <LI> You can encrypt data in transit with virtual network encryption to prevent data theft and tampering </LI> <BR /> <LI> You can log traffic on the hosts for troubleshooting, auditing or simply post mortem analysis </LI> <BR /> <LI> You can now apply security ACLs for your physical fabric networks </LI> <BR /> <LI> You can enable secure, high performant communication between virtual networks </LI> <BR /> <LI> You can use IPv6 addressing for your virtual networks </LI> <BR /> </OL> <BR /> All these enhancements will bolster customer confidence when they run their workloads in the hybrid cloud. They can rest assured that their workloads are safe and secure with Windows Server 2019. <BR /> <BR /> Thanks for reading, <BR /> <BR /> Greg Cusanza and Anirban Paul </BODY></HTML> Thu, 14 Feb 2019 18:06:49 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-4-security/ba-p/339774 GregCusanza 2019-02-14T18:06:49Z Top 10 Networking Features in Windows Server 2019: #5 Network Performance Improvements for Virtual Workloads https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-5-network/ba-p/339773 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Aug 22, 2018 </STRONG> <BR /> <STRONG> Share On: <A href="#" target="_blank"> Twitter </A> Share on: <A href="#" target="_blank"> LinkedIn </A> </STRONG> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> The Software Defined Data-Center (SDDC) spans technologies like Hyper-V, Storage Spaces Direct (S2D), and Software Defined Networking.&nbsp; Whether you have compute workloads like File, SQL, and VDI, you&nbsp;run an S2D cluster, or perhaps you're using your SDN environment to bring hybrid cloud to a reality, no doubt we crave network performance – we have a “need for speed” and no matter how much you have you can always use more. <BR /> <BR /> In Windows Server 2016, <A href="#" target="_blank"> we demonstrated </A> 40 Gbps into a VM with Virtual Machine Multi-Queue (VMMQ).&nbsp;&nbsp;However, high-speed network throughput came at the additional cost of complex planning, baselining, tuning, and monitoring to alleviate CPU overhead from network processing.&nbsp; Otherwise, your users would let you know very quickly when the expected performance level of your solution degrades.&nbsp;&nbsp;In Windows Server 2019, virtual workloads will reach and maintain 40 Gbps while lowering CPU utilization and eliminate the painful configuration and tuning cost previously imposed on you, the IT Pro. <BR /> <BR /> To do this, we’ve implemented two new features: <BR /> <UL> <BR /> <LI> Receive Segment Coalescing in the vSwitch </LI> <BR /> <LI> Dynamic Virtual Machine Multi-Queue (d.VMMQ) <STRONG> </STRONG> </LI> <BR /> </UL> <BR /> These features maximize the network throughput to virtual machines without requiring you to constantly tune or over-provision your host. This lowers the Operations &amp; Maintenance cost while increasing the available density of your hosts. The efforts outlined here cover our progress <STRONG> in accelerating </STRONG> the <STRONG> host </STRONG> and <STRONG> guest </STRONG> ; in a future article a colleague of mine (Harini Ramakrishnan) will discuss our efforts to accelerate the app in a future post. <BR /> <BR /> <STRONG> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75255i594A99BEBB579A0A" /> </STRONG> <BR /> <H2> Receive Segment Coalescing in the vSwitch </H2> <BR /> Number 1 on our playlist is an “oldie but goodie.” Windows Server 2019 brings a remix for Receive Segment Coalescing (RSC) leading to more efficient host processing and throughput gains for virtual workloads. As the name implies, this feature benefits any traffic running through the virtual switch including traditional Hyper-V compute workloads, some Storage Spaces Direct patterns, or Software Defined Networking implementations (for example, see Anirban's post last week regarding GRE gateway improvements <A href="#" target="_blank"> #6 - High Performance SDN Gateways </A> ). <BR /> <BR /> Prior to this release, RSC was a hardware offload (in the NIC). Unfortunately, this optimization was disabled the moment you attached a virtual switch. As a result, virtual workloads were not able take advantage of this feature. In Windows Server 2019, RSC (in the vSwitch) works with virtual workloads and is enabled by default!&nbsp; No action required your part! <BR /> <BR /> Here’s a quick throughput performance example from some of our early testing. &nbsp;In the task manager window on the left, you see a virtual NIC on top of a 40 Gbps physical NIC <STRONG> without </STRONG> RSC in the vSwitch. As you can see, the system requires an average of 28% CPU utilization to process 23.9 Gbps. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75256i030FE6776162685D" /> <BR /> <BR /> <BR /> <BR /> In the task manager window on the right, the same virtual NIC is now benefiting from RSC in the vSwitch. The <STRONG> CPU processing </STRONG> <STRONG> has decreased to 23% </STRONG> despite the receive <STRONG> throughput increasing to </STRONG> <STRONG> 37.9 Gbps! </STRONG> <BR /> <BR /> Here's the performance summary: <BR /> <TABLE> <TBODY><TR> <TD> </TD> <TD> <STRONG> Average CPU Utilization </STRONG> </TD> <TD> <STRONG> Average Throughput </STRONG> </TD> </TR> <TR> <TD> <STRONG> Without </STRONG> RSC in the vSwitch </TD> <TD> 28% </TD> <TD> 23.9 Gbps </TD> </TR> <TR> <TD> <STRONG> With </STRONG> RSC in the vSwitch </TD> <TD> 23% </TD> <TD> 37.9 Gbps </TD> </TR> <TR> <TD> ---Totals </TD> <TD> <STRONG> 17.86% Decrease </STRONG> in CPU </TD> <TD> <STRONG> 58.58% Increase </STRONG> in Throughput </TD> </TR> </TBODY></TABLE> <BR /> <H3> Under the Hood </H3> <BR /> RSC in the vSwitch combines TCP segments that are a part of the same TCP-stream into larger segments destined for a Hyper-V Guest. Processing coalesced (fewer) packets is far more efficient than the processing required for segmented packets. This leads to large performance gains to Hyper-V virtual machines. <BR /> <BR /> Performance gains are seen in both high and low throughput environments; high-throughput environments benefit from more efficient CPU processing (lower CPU utilization on the host) while low throughput environments may even see throughput gains in addition to the processing efficiencies. Take a look at RSC in action: <BR /> <BR /> <IFRAME frameborder="0" height="504" src="https://aka.ms/RSC-Video-Embed" width="896"> </IFRAME> <BR /> <H3> Get Started! </H3> <BR /> If you’re a Windows Server 2019 Insider and using Hyper-V, Storage Spaces Direct, Software Defined Networking (including the <A href="#" target="_blank"> High Performance Gateways </A> Anirban talked about last week!), you’re likely already consuming this feature! <STRONG> This feature is enabled by default </STRONG> ! But of course, if you’d like to compare the results yourself, check out our validation guide below. <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> <H2> Dynamic Virtual Machine Multi-Queue (d.VMMQ) </H2> <BR /> With the advent of 10 Gbps NICs (and higher), the processing required for the network traffic alone exceeded what could be accomplished by a single CPU. Virtual Machine Queue and its successor Virtual Machine Multi-Queue allowed traffic destined for a vmNIC to be processed by one or more different processor cores. <BR /> <BR /> Unfortunately, this required complex planning, baselining, tuning, and monitoring; often more effort than the typical IT Pro intended to expend.&nbsp; Even then, problems arose. If you were to introduce a heterogeneous hardware footprint in your datacenter, the optimal configuration could be varied or if tuning was needed, virtual machines may not be able to maintain a consistent level of performance. <BR /> <BR /> To combat these problems, Windows Server 2019 dynamically tunes the host for maximum CPU efficiency and consistent virtual machine throughput. D.VMMQ requires no setup once a supporting driver in-place and will autotune the existing workload to ensure optimal throughput is maintained for each virtual machine. This reduces the OPEX cost imposed by previous versions of this technology. <BR /> <H3> How it Works </H3> <BR /> There are two key outcomes from this technology: <BR /> <UL> <BR /> <LI> When network <STRONG> throughput is low </STRONG> : The system coalesces traffic received on a vmNIC to as few CPUs as possible </LI> <BR /> </UL> <BR /> <P> Here’s a VM receiving around 5.3 Gbps. </P> <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75257i7912EBD0A24228C9" /> </P> <BR /> <BR /> <P> The system can coalesce all packets onto one CPU for processing efficiency. </P> <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75258i0FAF252AB92A9CAE" /> </P> <BR /> <BR /> <UL> <BR /> <LI> When network <STRONG> throughput is high </STRONG> : The system automatically expands traffic received to as many CPUs as needed </LI> <BR /> </UL> <BR /> <P> The VMs traffic has grown to about 21 Gbps, which is more than a single CPU can handle. </P> <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75259iB86C4F13CE3AF4BA" /> </P> <BR /> <BR /> <P> The system expands the traffic across additional CPUs as necessary (and available) – In this case 5 - to maintain the demand for traffic. </P> <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75260i634A6EEBD598DE88" /> </P> <BR /> Here's a quick video on Dynamic VMMQ in a low-throughput scenario.&nbsp; You'll see the dynamic scheduling algorithm coalesce all the network throughput onto one core.&nbsp; Then, once network traffic has completed, the queues will return to their "ready" state allowing them to expand very quickly if a burst of traffic occurs. <BR /> <BR /> <IFRAME frameborder="0" height="504" src="https://aka.ms/DVMMQ-Video-Embed" width="896"> </IFRAME> <BR /> <H3> Get Started! </H3> <BR /> This feature requires a driver update for your NICs to a Dynamic VMMQ capable driver (referred to by some vendors as RSSv2). Drivers for Dynamic VMMQ will not be included inbox as this is an advanced feature, so please contact your IHV or OEM for the latest drivers. <BR /> <BR /> If you are purchasing new hardware, you should pay special attention to the available NICs and verify that they have received the <STRONG> <A href="#" target="_blank"> SDDC Premium Logo </A> </STRONG> through our certification program (click on a specific NIC and look for <STRONG> SDDC Premium </STRONG> ). If not, Dynamic VMMQ is not supported on these devices and you will default the traditional Static mechanism. <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <BR /> <BR /> <H2> Summary </H2> <BR /> Regardless of workload, your virtual machines <EM> need </EM> the highest possible throughput. Not only can Windows Server 2019 reach outstanding network performance, it eliminates the costly planning, baselining, and tuning required by previous Windows versions. You may still get a late-night call to troubleshoot a poorly performing virtual machine, but it won’t be because of the network throughput! <BR /> <BR /> Thanks for reading and see you at Ignite! <BR /> <BR /> Dan “Auto-tuning” Cuomo </BODY></HTML> Thu, 14 Feb 2019 18:06:44 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-5-network/ba-p/339773 Dan Cuomo 2019-02-14T18:06:44Z Top 10 Networking Features in Windows Server 2019: #6 High Performance SDN Gateways https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-6-high/ba-p/339766 <P><STRONG> First published on TECHNET on Aug 15, 2018 </STRONG> <BR /><STRONG> Share On: </STRONG> <A href="#" target="_blank" rel="noopener"> Twitter </A> <STRONG> Share On: </STRONG> <A href="#" target="_blank" rel="noopener"> LinkedIn </A> <BR /><BR />This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR />-- Click <STRONG> <A href="#" target="_blank" rel="noopener"> HERE </A> </STRONG> to see the other blogs in this series. <BR /><BR />Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR />Don't forget to tune in next week for the next feature in our Top 10 list! <BR />Organizations today deploy their applications across multiple clouds including on-premises private clouds,&nbsp;service provider clouds, and public clouds such as Azure. In such scenarios, enabling secure,&nbsp;high-performance connectivity across&nbsp;workloads in different clouds is essential.&nbsp;Windows Server 2019&nbsp;brings huge SDN gateway performance improvements&nbsp;for&nbsp;these hybrid connectivity scenarios, with <STRONG> network throughput multiplying by&nbsp;up to 6x!!! </STRONG> <BR /><BR />If you have deployed Software Defined Networking (SDN) with Windows Server 2016, you must be aware that, amongst other things, it provides connectivity between your cloud resources and enterprise resources through SDN gateways. In this article, we will talk about the following capabilities of SDN gateways: <BR /><BR /></P> <UL> <UL> <LI><STRONG> IPsec tunnels </STRONG> provide secure connectivity over the Internet between your hybrid workloads</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI><STRONG> GRE tunnels </STRONG> provide connectivity between your workloads hosted in SDN virtual networks and physical resources in the datacenter/high speed MPLS networks. More details about GRE connectivity scenarios <A href="#" target="_blank" rel="noopener"> here </A> .</LI> </UL> </UL> <P><BR /><BR />In Windows Server 2016, one of the customer concerns was the inability of SDN gateway to meet the throughput requirements of modern networks. The network throughput of IPsec and GRE tunnels was limited, with the single connection throughput for IPsec connectivity being about 300 Mbps and for GRE connectivity being about 2.5 Gbps. <BR /><BR />We have improved significantly in Windows Server 2019, with the <STRONG> numbers soaring to 1.8 Gbps and 15 Gbps for IPsec and GRE connections </STRONG> , respectively. All this, with <STRONG> huge reductions in the CPU cycles/per byte </STRONG> , thereby providing <STRONG> ultra-high-performance throughput with much less CPU utilization </STRONG> .</P> <P>&nbsp;</P> <P><STRONG>NOTE</STRONG>: All the improvements are pertaining to SDN multi-tenant gateways. You will not see the performance benefits if you deploy standalone RRAS server in single tenant mode.</P> <P>&nbsp;</P> <H3>Let's talk numbers.</H3> <P><BR />We have done extensive performance testing for the SDN gateways in our test labs. In the tests, we have compared gateway network performance&nbsp;with Windows Server 2019 in&nbsp;SDN&nbsp;scenarios&nbsp;and&nbsp;non-SDN&nbsp;scenarios. The results are shown below: <BR />GRE&nbsp;Performance Numbers <BR />Network throughput for&nbsp;GRE tunnels in Windows Server 2019 <EM> without SDN </EM> varies from 2 to 5 Gbps, <STRONG> with SDN it leapfrogs to the range of 3 to 15 Gbps!!! </STRONG></P> <BLOCKQUOTE><BR />Note that the network throughput in Windows Server 2016 is much less than network throughput in <BR />Windows Server 2019 without SDN. With Windows Server 2019 SDN, the comparison is even more stark.</BLOCKQUOTE> <P><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 829px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75251i796D738FA03F5C77/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR /><BR /><BR />The CPU cycles/byte without SDN varies from 50 to 75, while it barely <STRONG> crosses 10 with SDN!!! </STRONG> <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 829px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75252iA424FFCCDB7DE4F6/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR /><BR />IPsec&nbsp;Performance Numbers <BR />For IPsec tunnels, the <STRONG> Windows Server 2019 SDN network throughput is about 1.8 Gbps for 1 tunnel and about 5 Gbps for 8 tunnels </STRONG> . Compare this to Windows Server 2016 where the network throughput of a single tunnel was 300 Mbps and the aggregate IPsec network throughput for a gateway VM was 1.8 Gbps. <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 835px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75253i74009EBC8EA0F04B/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR /><BR /><BR />The CPU cycles/byte&nbsp;without SDN varies from 50 to 90, <STRONG> while it is well within 50 with SDN!!! </STRONG> <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 850px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75254i7F82C9D78B2675FF/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR /><BR />With GRE, the aggregate SDN gateway network throughput scales to 15 Gbps and with <BR />IPsec, it can scale to 5 Gbps!!! <BR />Test Setup <BR />The test setup simulates connectivity between the SDN gateway and on-prem gateway in a private lab environment. The on-prem gateway is configured with Windows Routing and Remote Access (RAS) to act as a VPN Site-to-Site endpoint. Following are the setup details on the SDN gateway host and the SDN gateway VM: <BR /><BR /><STRONG> Gateway HOST </STRONG> <BR /><BR /></P> <OL> <OL> <LI>There are two NUMA nodes on the host machine with 8 cores per NUMA node. RAM on the gateway host is 40 GB. The gateway VM has full access to one NUMA node. And it is different from the NUMA node used by the host.</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Hyper threading is disabled</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Receive side buffer and send side buffer on physical network adapters is set to 4096</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Receive side scaling (RSS) is enabled on the host physical network adapters. Min and max processors are set to be from the NUMA node which the host is affinitized to. MaxProcessors is set to 8 (number of cores per NUMA node).</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Jumbo packets are set on the physical network adapters with value of 4088 bytes</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Receive Side Scaling is enabled in the vSwitch.</LI> </OL> </OL> <P><BR /><BR /><STRONG> Gateway VM </STRONG> <BR /><BR /></P> <OL> <OL> <LI>The gateway VM is allocated 8 GB of memory</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>For the Internal and External network adapters, the Send Side Buffer is configured with 32 MB of RAM and Receive Side Buffer is configured with 16 MB of RAM</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Forwarding Optimization is enabled for the Internal and External network adapters.</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Jumbo packets are enabled on the Internal and External network adapters with value of 4088 bytes</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>VMMQ is enabled on the internal port of the VM</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>VMQ and VRSS&nbsp;is enabled on the external network adapter of the VM</LI> </OL> </OL> <P><BR /><BR /></P> <H3>See it in action</H3> <P><BR />The short demo below showcases the improved performance throughput with Windows Server 2019. This demo uses a performance tool called <A href="#" target="_blank" rel="noopener"> ctsTraffic </A> to measure the network throughput of a single IPsec connection through the SDN VPN gateway. Traffic is being sent from a customer workload machine in the SDN network to an on-prem enterprise resource across a simulated Internet. As you can see, with Windows Server 2016, the network throughput of a single IPsec connection is only about 300 Mbps, while with Windows Server 2019, the network throughput scales to about 1.8 Gbps. <BR /><BR /><IFRAME src="https://youtu.be/CC-4oBDgx00" width="525" height="325"> </IFRAME> <BR /><BR /><BR /></P> <H3>Try it out</H3> <P><BR />For GRE connections, you should automatically see the improved performance once you deploy/upgrade to Windows Server 2019 builds on the gateway VMs. No manual steps are involved. <BR /><BR />For IPsec connections, by default, when you&nbsp;create&nbsp;the connection&nbsp;for your virtual networks you will get the Windows Server 2016 data path and performance numbers. To enable the Windows Server 2019 data path, you will need to do the following: <BR /><BR /></P> <OL> <OL> <LI>On an SDN gateway VM, go to Services console (services.msc).</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Find the service named “Azure Gateway Service”, and set the startup type of this service to “Automatic”</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Restart the gateway VM. Note that the active connections on this gateway will be failed over to a redundant gateway VM</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>Repeat the previous steps for rest of the &nbsp;gateway VMs</LI> </OL> </OL> <P><BR /><BR />NOTE: For best performance results, ensure that the <STRONG> cipherTransformationConstant </STRONG> and <BR /><STRONG> authenticationTransformConstant </STRONG> in quickMode settings of the IPsec connection uses the “ <STRONG> GCMAES256 </STRONG> ” cipher suite. <BR />One more thing: To get maximum performance, the gateway host hardware must support AES-NI and PCLMULQDQ CPU instruction sets. These are available on any Westmere (32nm) and later Intel CPU except on models where AES-NI has been disabled. You can look at&nbsp;your hardware vendor&nbsp;documentation to see if&nbsp;the CPU&nbsp;supports AES-NI and PCLMULQDQ CPU instruction sets. <BR /><BR /><BR /><BR />Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank" rel="noopener"> Try it out! </A> <BR /><BR /><BR /><BR /></P> <H3>We value your feedback</H3> <P><BR />The most important part of a frequent release cycle is to hear what’s working and what needs to be improved, so your feedback is extremely valued. <BR /><BR />Contact us if you have any questions or having any issues for your deployment or validation. We also encourage you to visit send us email - sdninsider@microsoft.com to collaborate, share and learn from other customers like you. <BR /><BR />Thanks for reading, <BR /><BR />Anirban Paul</P> Wed, 22 Jul 2020 14:46:50 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-6-high/ba-p/339766 AnirbanPaul 2020-07-22T14:46:50Z Top 10 Networking Features in Windows Server 2019: #7 SDN Goes Mainstream https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-7-sdn-goes/ba-p/339761 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Aug 08, 2018 </STRONG> <BR /> <EM> <STRONG> Authors: Greg Cusanza, Schumann Ge </STRONG> </EM> <STRONG> <BR /> </STRONG> <BR /> <STRONG> Share On: </STRONG> <A href="#" target="_blank"> Twitter </A> <STRONG> Share On: </STRONG> <A href="#" target="_blank"> LinkedIn </A> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> If you’ve ever deployed Software Defined Networking (SDN), you know it provides great power but is historically difficult to deploy. Now, with Windows Server 2019, it's easy to deploy and manage through a new deployment UI and Windows Admin Center extension that will enable anyone to harness the power of SDN.&nbsp; Even better, these improvements also apply to SDN in Windows Server 2016! <BR /> <BR /> For those of you new to SDN in Windows Server, you might be surprised to learn that it's included as a key component of the Software Defined Data Center (SDDC), providing software-based network functions such as virtual networking with switching, routing, firewalling with micro-segmentation, third-party appliances, and load balancing, all virtualized and highly optimized for availability and performance. <BR /> <BR /> This enables you to create a datacenter or branch office with lower costs, with increased security, and greater agility to tailor the network to meet the needs of your applications. It’s all included in Windows Server Datacenter edition, so any SDDC deployment from two node hyper-converged systems up to multi-rack data-center deployments benefit from these capabilities... <BR /> <H3> <EM> <STRONG> At no extra cost! </STRONG> </EM> </H3> <BR /> <H2> Getting Started </H2> <BR /> In this post we'll cover the new steps for setting up your physical network for SDN, deploying SDN with SDN Express and then using the Windows Admin Center to manage the new deployment. <BR /> <H3> Setting up the physical network </H3> <BR /> Every SDN system must connect and interface with a physical network, this is not unique to Microsoft’s SDN implementation. We publish a recommended topology for how your physical network should be configured to help bridge the gap between server admin and network admin. <BR /> <BR /> Updates to our best practice topology streamline it down to requiring only two networks to be configured: <BR /> <OL> <BR /> <LI> A <STRONG> management network </STRONG> for infrastructure communication </LI> <BR /> <LI> A <STRONG> provider network </STRONG> for the virtualized workload traffic. </LI> <BR /> </OL> <BR /> This separation allows the two networks to be fully isolated, even down to two separate sets of physical networking if you choose, creating a complete boundary between infrastructure and tenant. <BR /> <BR /> To learn more, refer to the <A href="#" target="_blank"> Plan a Software Defined Network Infrastructure </A> topic on docs.microsoft.com. <BR /> <H3> Deploying SDN with ease </H3> <BR /> Once you have the physical network configured you are ready for deployment. <BR /> <EM> Note: If you're using SCVMM for management you must use VMM to perform the deployment, and from then on will use VMM’s integrated and managed SDN environment. When deploying through SCVMM you can use it's UI, or the automation available through the <A href="#" target="_blank"> VMM SDN Express scripts </A> on github. </EM> <BR /> <BR /> <BR /> SDN Express is a UI, a PowerShell script and set of modules available on Github to get you up and running quickly.&nbsp; The new guided UI is able to perform parameter validation so may common errors are detected at input time.&nbsp; This gives you an immediate opportunity to correct mistakes before deployment begins. <BR /> <BR /> This image animates a walkthrough of the SDN Express wizard.&nbsp; Fields that show up in red are invalid, initially because they are required, and turn gray as valid data is entered: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75243iBBA06CEEB9239660" /> </P> <BR /> You can have the wizard kick off the deployment or use it to create a configuration file for further customization or repeated deployments. <BR /> <BR /> <BR /> If you open the config file, you’ll find that it’s been greatly streamlined as compared with previous versions of SDN Express, with about a 75% reduction in number of parameters. <BR /> <H3> SDN Express PowerShell module </H3> <BR /> If you want further control over deployment, you can use the new SDN Express PowerShell module. This gives you the ability to customize your initial deployment further, or scale out an existing deployment by adding Hyper-V hosts, Load Balancers or Gateways.&nbsp; Here’s an example of adding a new Hyper-V server; this snippet takes a freshly deployed Hyper-V host and fully enables it for SDN: <BR /> import-module .\SDNExpressModule.psm1 <BR /> <BR /> $restname = 'SDN.CONTOSO.COM' <BR /> $rootcerts = get-childitem "cert:\localmachine\root" <BR /> $hostcert = $rootcerts | ? {$_.Subject -eq "CN=$restname"} <BR /> <BR /> Add-SDNExpressHost -ComputerName 'Host5' -RestName $restname -HostPASubnetPrefix '10.0.0.0/24' -NCHostCert $hostcert -Credential (get-credential) <BR /> The full SDN Express deployment and configuration takes about 45 minutes run, but it includes creation of the SDN infrastructure VMs, enabling the necessary roles and performing the configuration on each node. When SDN Express finishes your SDN environment is ready to manage with the preview of the SDN extension for the Windows Admin Center! <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? <A href="#" target="_blank"> Try out the new SDN Express now! </A> <BR /> <BR /> <BR /> <BR /> <BR /> <H3> Try out the new SDN Express </H3> <BR /> You can get SDN express today from the <A href="#" target="_blank"> Microsoft SDN repository on Github </A> . Just download the SDN repository, navigate to SDNExpress/scripts and run the SDNExpress.ps1 file from a Windows Server 2016 or 2019 computer. It will guide you the rest of the way. <BR /> <H2> SDN in Windows Admin Center </H2> <BR /> <H3> About Windows Admin Center </H3> <BR /> If you haven’t tried out Windows Admin Center yet, you’re really missing out. Windows Admin Center is an evolution of Windows Server in-box management tools; it’s a single pane of glass that consolidates all aspects of local and remote server management. Windows Admin Center is a locally deployed, browser-based app for managing servers, clusters, hyper-converged infrastructure, and Windows 10 PCs. It comes at no additional cost beyond Windows and is ready to use in production. In the first 6 months since announcement, more than 25,000 customers deployments have been managed by Windows Admin Center. <BR /> <H3> Using Windows Admin Center for SDN management </H3> <BR /> SDN has been integrated with the Hyper-Converged Cluster experience in Windows Admin Center. By adding a Network Controller to your Hyper-Converged cluster, you can manage your SDN resources and infrastructure through a single application. <BR /> <STRONG> Important: A hyper-converged cluster is required to use the SDN extension for the Windows Admin Center. </STRONG> <BR /> Here's how easy it is to add a hyper-converged cluster with a Network Controller to the Windows Admin Center: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75244i9EDED76D0944163E" /> </P> <BR /> <BR /> <H3> Virtual network management </H3> <BR /> Once you’ve added SDN to your hyper-converged environment, you can create, modify and configure virtual networks and their subnets. You can also view the VMs connected to the virtual network subnets. <BR /> <BR /> This is a view of the virtual networks list: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75245iEDF7ABF6E26E83E1" /> </P> <BR /> This is only the beginning! We are working to bring full end-to-end virtual network management to the SDN extension for Windows Admin Center. <BR /> <H3> Connecting a virtual machine </H3> <BR /> The next step is connecting a new virtual machine to your brand new virtual network. During this process the Windows Admin Center detects that the virtual switch you are connecting to has SDN enabled and provides you with the ability to select the SDN networks that are available. <BR /> <BR /> Here you can see the virtual network adapter settings which detect that SDN is in place and show the appropriate options for selecting a virtual network and subnet: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75246i1C71AF8B47E0BE63" /> </P> <BR /> <BR /> <H3> SDN infrastructure management </H3> <BR /> Being able to manage your SDN infrastructure is a critical aspect of operating SDN infrastructure. Due to the resiliency built into Windows Server SDN individual component failures will not impact your workloads, so you need an easy way to see when something is unhealthy. <BR /> <BR /> With the SDN Monitoring extension you can monitor the state of the SDN services and infrastructure in real-time. You can view detailed information about the health of your Network Controller, Software Load Balancers, Virtual Gateways, and hosts.&nbsp; You can also monitor consumption of your Virtual Gateway Pools, Public IP Pools, and Private IP Pools. <BR /> <BR /> This is the summary view. It is the best place to look to understand the overall health of your system: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75247i209313964CD7F1CC" /> </P> <BR /> Drilling into the Network Controller panel you can get additional information about the health of individual Network Controller services and Hyper-V hosts: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75248i5C6057B661F4720F" /> </P> <BR /> On the Load Balancer panel you can also see the health of individual components of the Software Load Balancer as well as the utilization of your load balancer virtual ip (VIP) pools so you will know if you are about to run out of IP addresses: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75249i1F64227080E3D7F1" /> </P> <BR /> And finally, the gateway panel shows you the status of each gateway pool and the health of the individual gateway VMs that make up the pool. <STRONG> At Risk </STRONG> indicates that one of your gateway VMs is unhealthy, but due to the redundancy built into the pool, it is not yet affecting any of the workload traffic.&nbsp; Unless corrected, one additional failure will likely impact the ability to host workloads: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75250iA7225C5C56E69977" /> </P> <BR /> <BR /> <BR /> As mentioned previously, there is much more on the way! Please stay tuned for more features as we build out the complete set of SDN features in the Window Admin Center. <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? Try out SDN with the Windows Admin Center now! <BR /> <BR /> Registered Insiders may download the Windows Admin Center Preview containing what you see <BR /> here directly from the <A href="#" target="_blank"> Windows Server Insider Preview download page </A> , under the Additional <BR /> Downloads dropdown. <BR /> <BR /> If you have not yet registered as an Insider, see <A href="#" target="_blank"> Getting Started with Windows Server </A> on <BR /> the Windows Insiders for Business portal. <BR /> <BR /> <BR /> <BR /> <BR /> <H2> Stay tuned to this blog series for more on SDN in Windows Server 2019! </H2> <BR /> This post highlights the efforts we are taking to make SDN easier to deploy through SDN Express and manage through the Windows Admin Center, but there are also a number of features that we've added to SDN in Windows Server 2019 that we'll highlight in some of the later blog posts in this series, so stay tuned! <BR /> <BR /> Thanks! <BR /> <BR /> Greg Cusanza and Schumann Ge </BODY></HTML> Thu, 14 Feb 2019 18:05:21 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-7-sdn-goes/ba-p/339761 GregCusanza 2019-02-14T18:05:21Z Top 10 Networking Features in Windows Server 2019: #8 A Faster, Safer Internet https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-8-a-faster/ba-p/339749 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Aug 01, 2018 </STRONG> <BR /> <EM> <STRONG> Authors: Gabriel Montenegro, Daniel Havey </STRONG> </EM> <BR /> <STRONG> Share On: <A href="#" target="_blank"> Twitter </A> Share On: <A href="#" target="_blank"> LinkedIn </A> </STRONG> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> The Internet is part of our daily lives both at work and at home, in the enterprise and in the cloud.&nbsp; We are committed to making your Internet experience faster and safer, and in this blog, we discuss how the features in Windows Server 2019 brings those goals closer to reality.&nbsp; To do this we: <BR /> <UL> <BR /> <LI> Improved coalescing of connections to deliver an uninterrupted and properly encrypted browsing experience. </LI> <BR /> <LI> Upgraded HTTP/2’s server-side cipher suite negotiation for automatic mitigation of connection failures and ease of deployment. </LI> <BR /> <LI> Changed our default TCP congestion provider to Cubic to give you more throughput! </LI> <BR /> </UL> <BR /> <H2> HTTP/2 for a faster and safer Web </H2> <BR /> We originally added <A href="#" target="_blank"> support for HTTP/2 </A> ( <A href="#" target="_blank"> RFC 7540 </A> ) in Windows Server 2016 (and Windows 10) in the native HTTP server (in particular, http.sys, the kernel component for the HTTP server and IIS).&nbsp; Now, Windows Server 2019 delivers performance and security benefits to your web site deployments with HTTP/2. <BR /> <BR /> For a clear illustration of HTTP/2 performance gains over HTTP/1.1, checkout this demo: <A href="#" target="_blank"> https://http2.akamai.com/demo </A> or&nbsp;play the video below: <BR /> <BR /> <IFRAME height="325" src="https://www.youtube.com/watch?v=SZlGNNqsxTo" width="525"> </IFRAME> <BR /> <H3> HTTP/2 Refresher </H3> <BR /> HTTP/2&nbsp;increments the&nbsp;HTTP protocol version for the first time in well over a decade – version 1.1 ( <A href="#" target="_blank"> RFC 2616 </A> ) was published in 1999! &nbsp;As befits a <EM> major </EM> protocol version, the new version was not bumped from 1.1 to 1.2, but from 1.1. to 2. &nbsp;HTTP/2 brings some radical improvements for web site performance based on features such as: <BR /> Multiplexing <BR /> HTTP is the best-known and most deployed protocol in the internet – It is the basis of the web.&nbsp; Nevertheless, by itself it cannot accomplish anything! &nbsp;To exchange data, HTTP depends on the services of the “transport layer.” <BR /> <BR /> HTTP/2 and HTTP/1.1 have a radically different use of this underlying transport layer: <BR /> <UL> <BR /> <LI> <STRONG> HTTP/1.1 </STRONG> : Each request required a dedicated TCP (and TLS, when using <EM> HTTPS </EM> ) connection potentially imposing several round trips to establish that connection. </LI> <BR /> <LI> <STRONG> HTTP/2 </STRONG> : HTTP/2 shares a single TCP connection across many requests to the same web site.&nbsp; This is called multiplexing. </LI> <BR /> </UL> <BR /> With HTTP/2’s multiplexing capability, only the first request incurs the roundtrips required to establish the connection.&nbsp; Subsequent associated requests (more information under <STRONG> Coalescing </STRONG> section below) require no connection establishment and immediately send HTTP data (A.K.A. <EM> 0-RTT </EM> ).&nbsp; The picture below contrasts how HTTP/1.1 and HTTP/2 use the transport layer. <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75240i0E1A9B66827E0949" /> </P> <BR /> <BR /> Header Compression <BR /> HTTP exchanges typically employ many HTTP headers – Sometimes these headers represent much more data than the actual payload. &nbsp;Recognizing this problem, HTTP/2 uses HPACK, a compression scheme built explicitly for HTTP Header compression. This drastically reduces the amount of data that needs to be exchanged between client and server which may also save on round-trip times. <BR /> <H3> HTTP/2 Improvements in Windows Server 2019 </H3> <BR /> Connection Coalescing <BR /> Windows Server 2019 extends the benefits of HTTP/2 to domains designed for HTTP/1.1 by applying <EM> connection coalescing </EM> to mitigate sharding.&nbsp; In HTTP/1.1 <EM> sharding </EM> is when a given domain is made to appear as different domains to force more independent TCP connections. This is an artificial method of creating parallelism that is no longer required in HTTP/2, but sharding and websites designed for HTTP/1.1 will remain for a long time. <BR /> <BR /> To mitigate sharding, Windows Server 2019 also enables <EM> connection coalescing </EM> on both Edge and the HTTP server. With coalescing, domains like <EM> a.bing.com </EM> and <EM> b.bing.com </EM> will end up sharing a single TCP connection if their certificate matches.&nbsp; Without coalescing, sites like <EM> a.bing.com </EM> and <EM> b.bing.com </EM> would require separate TCP connections. <BR /> Security Improvements <BR /> Windows Server 2019 automatically fixes potential connection failures with HTTP/2!&nbsp; To understand why connection failures may arise, let’s remember that HTTP/2 requires at least version 1.2 of TLS with modern and secure cipher suites while <A href="#" target="_blank"> blacklisting others </A> . &nbsp;Unfortunately, this security requirement can lead to brittle HTTP/2 negotiations. If so, users of your web site may be unable to connect until the web server administrator fixes the SSL cipher suite ordering. <BR /> <BR /> With Windows Server 2019, this is resolved without any intervention from the administrator, so users don’t encounter connection failures. <BR /> <BR /> Here are some details of what we did: <BR /> <UL> <BR /> <LI> These failure modes can arise if the default SSL cipher suite ordering in Windows Server 2016 is changed incorrectly: if any of the cipher suites blacklisted by HTTP/2 appears before those allowed by HTTP/2, Firefox and Chrome abort the connection (as allowed, but not recommended by HTTP/2). Chrome shows ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY, and Firefox, NS_ERROR_NET_INADEQUATE_SECURITY. </LI> <BR /> <LI> Even though <A href="#" target="_blank"> correct ordering of the SSL cipher suites </A> (as assured by the default ordering in Windows) avoids this problem, in Windows Server 2019 we have improved the robustness of the cipher suite negotiation mechanism to be impervious to the ordering of the SSL cipher suites. Of course, the list must still include cipher suites allowed by HTTP/2, but they no longer need to necessarily appear at the beginning of the list before any blacklisted ones. </LI> <BR /> </UL> <BR /> This reduces the operational complexity of HTTP/2 deployment, enabling customers to more readily reap its <A href="#" target="_blank"> benefits </A> including the higher-grade cipher suites required by HTTP/2. <BR /> <H2> Windows TCP goes Cubic! </H2> <BR /> Many of you already are aware of our march towards lower latency, higher throughput transports with pluggable congestion control providers. These regulate TCP senders, so they apply equally to both client and server.&nbsp; Our congestion control providers are: New-Reno, Compound TCP, Cubic and <A href="#" target="_blank"> LEDBAT </A> .&nbsp; So, what is the big news?&nbsp; Cubic is now the default congestion control provider going forward.&nbsp; How do you know?&nbsp; Open up a PowerShell window and run this command: <BR /> <BLOCKQUOTE> <BR /> PS C:\WINDOWS\system32&gt; Get-NetTCPSetting | Select SettingName, CongestionProvider <BR /> SettingName&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CongestionProvider <BR /> -----------&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ------------------ <BR /> Automatic <BR /> InternetCustom&nbsp;&nbsp; CUBIC <BR /> DatacenterCustom CUBIC <BR /> Compat&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NewReno <BR /> Datacenter&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CUBIC <BR /> Internet&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CUBIC <BR /> </BLOCKQUOTE> <BR /> See?&nbsp; Cubic, cubic everywhere (except the compatibility template)! <BR /> <BR /> So, why should you care that the new default is Cubic?&nbsp; Because Cubic is faster and fairer for the end-user.&nbsp; Cubic is especially well suited for high bandwidth, high latency links where Standard TCP tends to perform poorly. If you are an admin and have a need to send a significant amount of data over a high bandwidth, high latency (long distance) link, you will like the benefits that Cubic brings. <BR /> <P> <EM> <STRONG> “We’ve got a ton of data that shows CUBIC moves our mean throughput, in MB/s on coast-to-coast transfers at ~70ms RTT a full 40MB/s relative to CTCP. It’s an exciting shift in throughput that we feel is going to grant us the recovery head room to reach our goals.” </STRONG> </EM> </P> <BR /> <P> <EM> <STRONG> – </STRONG> Engineer at a very large Microsoft service </EM> </P> <BR /> Why is Cubic faster on high speed long distance network links?&nbsp; It’s all in the sending window curves.&nbsp; Figure 1 (below) shows how the data sending rate (throughput) changes over time.&nbsp; The blue line is standard TCP (New Reno) and the red line is Cubic. <BR /> <BR /> The left side of the graph shows the connection start-up (slowstart) phase.&nbsp; TCP start-up is not affected by congestion control algorithms so this part of the curves are identical. &nbsp;After start-up phase is completed the connections goes into congestion avoidance phase.&nbsp; Notice that the Cubic curve spends more of its time near the network saturation point than standard TCP.&nbsp; This is why Cubic is faster because its congestion window curve is exponential rather than linear like standard TCP. <BR /> <BR /> [caption id="attachment_5835" align="aligncenter" width="672"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75241i320457603D596959" /> <EM> <STRONG> Figure 1 — New Reno vs Cubic TCP Congestion Window Curves </STRONG> </EM> [/caption] <BR /> <BR /> <BR /> <BR /> Okay, enough theory.&nbsp; Let's see some data!&nbsp; In Figure 2, we see an experiment where we sent data across the continental United States in 250 MB chunks overnight.&nbsp; We binned and sorted the data into a Pareto chart (without the line graph for clarity).&nbsp; On the left we see Cubic and on the right we see Compound (the default congestion provider prior to Windows server 2019).&nbsp; Cubic is consistently reaching higher throughputs than Compound. <BR /> <BR /> Bottom line: <STRONG> Cubic gets more throughput. </STRONG> <BR /> <BR /> [caption id="attachment_6285" align="aligncenter" width="1248"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75242i0EDE3391ADE366D5" /> <EM> <STRONG> Figure 2 -- Cubic Throughput vs Compound TCP Throughput </STRONG> </EM> [/caption] <BR /> <H3> <I> Conclusion </I> </H3> <BR /> At&nbsp;the Windows Core Networking team,&nbsp;we are excited to&nbsp;deliver Windows Server 2019's&nbsp;improvements in&nbsp;(1) the HTTP server's ease of deployment and performance&nbsp;of HTTP/2, and&nbsp;(2) TCP congestion control. The world just got faster and safer for consumers and server administrators and all you need to do is run Windows. <BR /> <BR /> Thanks for reading, <BR /> <BR /> Daniel Havey and Gabriel Montenegro </BODY></HTML> Thu, 14 Feb 2019 18:04:18 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-8-a-faster/ba-p/339749 Daniel Havey 2019-02-14T18:04:18Z Top 10 Networking Features in Windows Server 2019: #9 LEDBAT – Latency Optimized Background Transport https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-9-ledbat-8211/ba-p/339745 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Jul 25, 2018 </STRONG> <BR /> <STRONG> Share On: <A href="#" target="_blank"> Twitter </A> Share on: <A href="#" target="_blank"> LinkedIn </A> </STRONG> <BR /> <BR /> This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR /> -- Click <STRONG> <A href="#" target="_blank"> HERE </A> </STRONG> to see the other blogs in this series. <BR /> <BR /> Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR /> Don't forget to tune in next week for the next feature in our Top 10 list! <BR /> Keeping a network secure is a never-ending job for IT Pros, and doing so requires regularly updating systems to protect against the latest threat vectors.&nbsp; This is one of the most common tasks that an IT Pro must perform.&nbsp; Unfortunately, it can result in dissatisfaction for end-users as the network bandwidth used for the update can compete with interactive tasks that the end-user requires to be productive. <BR /> <BR /> Have you ever had a support call that started like this? <BR /> <P> “…I can’t seem to save my presentation to SharePoint” <BR /> “…my Skype session sounds like I’ve entered the Matrix!” </P> <BR /> With Windows Server 2019, we bring a latency optimized, network congestion control provider called LEDBAT, which stands for Low Extra Delay Background Transfer. LEDBAT is designed to automatically yield bandwidth to users and applications, while consuming the entire bandwidth available when the network is not in use. It’s a scavenger protocol – it scavenges whatever network bandwidth is available on the network, and uses it. In other words, you can transfer SCCM Packages or Microsoft Updates without interfering with your user’s sanity. <BR /> <STRONG> Important: </STRONG> LEDBAT can optimize any TCP sender-side workload. It is not limited to updates! <BR /> If you remember our Anniversary edition post: <A href="#" target="_blank"> Announcing: New Transport Advancements in the Anniversary Update for Windows 10 and Windows Server 2016 </A> , LEDBAT was configured through an undocumented socket option.&nbsp; As of Windows Server 2019, LEDBAT is now a fully supported feature. <BR /> <BR /> Here’s what our some of our Microsoft MVPs had to say about their experience with LEDBAT: <BR /> <P> <STRONG> "LEDBAT will play a key part of how Enterprises deal with infrastructure being more and more component based in the future, being the workhorse that keeps the Enterprise up to date without interrupting or impacting critical business traffic." </STRONG> </P> <BR /> <P> <EM> – Andreas Hammarskjöld (Co-founder of <STRONG> <A href="#" target="_blank"> 2Pint Software </A> / </STRONG> Übergeek) </EM> </P> <BR /> <P> <STRONG> "This issue listed in <A href="#" target="_blank"> KB4163525 </A> caused extremely high network bandwidth consumption due to clients running full SUP scans. LEDBAT could have minimized this so that network saturation did not occur. We need LEDBAT!&nbsp; Sign me up as soon as it is ready for WS 2016!" </STRONG> </P> <BR /> <P> <I> - <STRONG> </STRONG> Mike Terrill &nbsp;(Enterprise Experiences &amp; Management MVP and OS Engineer at a Global Financial Company) </I> </P> <BR /> <BR /> <H2> Challenges with Existing Approaches </H2> <BR /> Some protocols like BITS (Background Intelligent Transfer Service) use an Adaptive Bit Rate (ABR) to adjust bandwidth of lower priority traffic. ABRs usually require multiple adjustments prior to reaching an optimized level of bandwidth that does not interfere with other current workloads.&nbsp; However, each adjustment can require up to 2-seconds (which is not insignificant in our instant gratification world!). In addition, these two second increments add-up, negatively affecting the user’s experience over the long run! As a result, BITS has switched to using LEDBAT for upload traffic. <BR /> <BR /> Another existing approach is to use throttling, or specifying the maximum amount of bandwidth that can be used for a specific purpose. For instance, many of our customers have an SCCM distribution point that throttles the downloads of packages to 50% of the available bandwidth to its clients. In this scenario, you’ll only ever use 50% of the bandwidth even if 100% is available – You’ve set a maximum amount that cannot be exceeded under any circumstance. As a result, your client downloads could take 2x as long! Even worse, user traffic may require more than 50% of the overall bandwidth – in such scenarios, the bandwidth set aside for background transfers would interfere with the user experience.&nbsp; You can see this effect in picture below: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75235i825A1BBD171C6B08" /> </P> <BR /> In contrast, LEDBAT leverages unused network resources, and does not need a bandwidth caps for background transfers typically required by other solutions: <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75236i33B15C6290983190" /> <BR /> <H2> Latency as the Key Metric </H2> <BR /> My favorite quote from, <EM> <A href="#" target="_blank"> Primer on Latency and Bandwidth Networking 101, chapter 1 </A> </EM> is: <EM> “To succeed, network latency has to be carefully managed and be an explicit design criteria at all stages of development.” </EM> <BR /> <BR /> One of the things we realized over the years is that latency is the key metric to optimize when it comes to having a great user experience. Whether it is a website that needs to load, a Skype call that needs to connect (and stay connected with high quality), or watching the recent world cup – latency is critical to keep low. An increase in latency generally indicates increased usage of the network, and such&nbsp;increases in latency usually result in a poor user experience with their productive tasks. <BR /> <BR /> Consequently, LEDBAT carefully tracks latency and automatically yields the network to other traffic as the latencies start to go beyond a threshold. It operates on the sending-side of network communication, implementing <A href="#" target="_blank"> RFC 6817 </A> . <STRONG> We open-sourced these modifications and noted this at a recent IETF to ensure the community could benefit from our learnings. </STRONG> <BR /> <H2> LEDBAT in Action </H2> <BR /> On the left side of the image below we see a time series graph calibrated over latency without LEDBAT.&nbsp; Before I started sending data (this is the user/application traffic), the latency was hovering around 10ms, which generally translates to a nice and smooth experience for users.&nbsp; At about 10 seconds into the experiment, I started a data flow not optimized using LEDBAT (say, someone else initiated a large file download) and BOOM!&nbsp; The latency goes straight to the moon!&nbsp; Over three thousand milliseconds! <BR /> <BR /> As noted earlier, this significantly impacts the user experience, and I’m guessing as an IT Pro, you do not want to take that support call. With that said, if your company uses VOIP, that frustrated user might not be able to get through to tech support anyway! :smiling_face_with_smiling_eyes:</img> <BR /> <BR /> [caption id="attachment_6105" align="aligncenter" width="953"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75237iB20D4AF47AF3B0D0" /> <EM> <STRONG> LEDBAT minimizes latency and user frustration </STRONG> </EM> [/caption] <BR /> <BR /> In contrast, we see the same experiment with LEDBAT on the right side of the image above.&nbsp; Just as before, I started the data flow (this time a LEDBAT optimized flow) at about 10 seconds into the experiment and the latency did indeed go up, but, only a little averaging about 100ms.&nbsp; That’s less than the time that it takes for you to blink your eye, and is generally not perceptible for many actions!&nbsp; As a result, user experience typically will not be impacted by these updates, which translates to happier end-users. In other words, as an IT Pro, you will be able to distribute updates to keep your organization secure, and without significantly impacting a user’s experience. <BR /> <BR /> <STRONG> Here's a video that further illustrates the effect of the latency: </STRONG> <BR /> <P> <IFRAME frameborder="0" height="315" src="https://www.youtube.com/embed/2jvxqNdqDJE?rel=0" width="560"> </IFRAME> </P> <BR /> The image below tracks network throughput over time. The height of the bars indicates network utilization, and the color of the bar indicates the type of traffic. LEDBAT is displayed in blue and non-LEDBAT (this is the user productivity traffic) is displayed in orange. Observe that till the 13 second marker, there is no competing user traffic – consequently, LEDBAT utilizes a significant portion of the network. At 13 seconds, I start the Not-LEDBAT data flow. Observe how LEDBAT promptly backs off giving the non-LEDBAT data flows the needed bandwidth. Then at the 25 second marker, I stop the non-LEDBAT data flow (this would be equivalent to the user stops watching a video or otherwise) and LEDBAT comes right back ramping up to good utilization automatically. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75238i192A2757F9B213CF" /> <BR /> <BR /> <STRONG> Here's a demo illustrating the user productivity shown above: </STRONG> <BR /> <H1> <IFRAME frameborder="0" height="315" src="https://www.youtube.com/embed/6fBGs7t3kRM?rel=0" width="560"> </IFRAME> </H1> <BR /> Think about what this means to a system update.&nbsp; The system updates are in blue using LEDBAT.&nbsp; When a user (in orange) starts using the network, LEDBAT quickly and automatically gets out of the way.&nbsp; Subsequently, when the user is not using the network, LEDBAT automatically ramps back up to full utilization. <STRONG> No throttles, no tuning, no scheduling, no hassles for the IT Pro.&nbsp; Doesn’t that sound nice? </STRONG> <BR /> <BR /> <BR /> <BR /> Ready to give it a shot!? Download the latest <A href="#" target="_blank"> Insider </A> build and <A href="#" target="_blank"> Try it out! </A> <BR /> <BR /> <STRONG> *** </STRONG> There was a bug in the validation guide. The guide incorrectly referred to the DatacenterCustom template. <BR /> Please use the InternetCustom template instead. The guide has been fixed. <BR /> <BR /> <BR /> <BR /> <H2> LEDBAT with SCCM </H2> <BR /> LEDBAT can also be enabled on a SCCM distribution point running Windows Server 2019.&nbsp; Because LEDBAT operates on the sending side, any client <STRONG> regardless of the operating system </STRONG> , will enjoy the benefits that it brings.&nbsp; To enable this in SCCM, check the following option: <BR /> <P> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75239i8D05D71617F1BB57" /> </P> <BR /> Here's more information on how you can <A href="#" target="_blank"> enable SCCM distribution points to use network congestion control </A> <BR /> <BR /> Well that is the end of this blogpost.&nbsp; I hope you have enjoyed reading and watching the videos.&nbsp; Please remember that LEDBAT can be used for any TCP-based workload that sends large amounts of data.&nbsp; Don’t forget to check out the validation guide. We would love to hear your feedback in the comments section below! <BR /> <BR /> Thanks again for reading, <BR /> <BR /> Daniel <EM> "low latency" </EM> Havey <BR /> <BR /> Here's a quick summary of the resources included in this article: <BR /> <UL> <BR /> <LI> LEDBAT Validation Guide: <A href="#" target="_blank"> Try it out! </A> </LI> <BR /> <LI> How to Enable SCCM to leverage LEDBAT: <A href="#" target="_blank"> Enable SCCM distribution points to use network congestion control </A> </LI> <BR /> <LI> Anniversary edition post: <A href="#" target="_blank"> Transport Advancements in the Anniversary Update for Windows 10 and Windows Server 2016 </A> <A href="#" target="_blank"> </A> </LI> <BR /> <LI> 2 Pint Software -- <A href="#" target="_blank"> Bandwidth Management in Windows using Microsoft LEDBAT++ </A> </LI> <BR /> <LI> Johan Arwidmark, Deployment Research -- <A href="#" target="_blank"> Setup Low Extra Delay Background Transport (LEDBAT) for ConfigMgr </A> </LI> <BR /> </UL> <BR /> [1] <A href="#" target="_blank"> Request for Comments: 6817 --&nbsp;Low Extra Delay Background Transport (LEDBAT) </A> </BODY></HTML> Thu, 14 Feb 2019 18:03:51 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-9-ledbat-8211/ba-p/339745 Daniel Havey 2019-02-14T18:03:51Z Top 10 Networking Features in Windows Server 2019: #10 Accurate Network Time https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-10-accurate/ba-p/339739 <P><STRONG> First published on TECHNET on Jul 18, 2018 </STRONG> <BR /><BR />This blog is part of a series for the Top 10 Networking Features in Windows Server 2019! <BR />-- Due to the move of blog locations, if you find a broken link, please check <A href="#" target="_self">https://aka.ms/W32Time</A> for the referenced content.<BR /><BR />Look for the <STRONG> Try it out </STRONG> sections then give us some feedback in the comments! <BR />Don't forget to tune in next week for the next feature in our Top 10 list! <BR />Windows Server 2019 provides regulatory compliance with highly accurate time that is traceable and UTC-compliant, including support of leap seconds <EM> . </EM> In this article, we’ll talk about the technical advances we made between Windows Server 2016 and Windows Server 2019 including true UTC-compliant leap second support, a new time protocol called Precision Time Protocol, and end-to-end traceability.&nbsp; But before we talk about the technical details, let’s talk about why this matters to you. <BR /><BR />In the past, the requirement for time accuracy on Windows was limited to domain-based scenarios that required all devices to be synchronized within 5 minutes.&nbsp; Now worldwide government regulations (for example, US: <A href="#" target="_blank" rel="noopener"> FINRA </A> , EU: <A href="#" target="_blank" rel="noopener"> ESMA/MiFIDII </A> ) are demanding much higher accuracy time – as stringent as 100µs (microseconds).&nbsp; Self-proclaimed accuracy is not enough.&nbsp; You must also be able to prove or “trace” your time back to an authoritative time source – More on this later.&nbsp; ESMA justifies the accuracy and traceability requirements in this way: <EM> “...It is also essential for conducting cross-venue monitoring of orders and detecting instances of market abuse and allows for a clearer comparison between the transaction and the market conditions prevailing at the time of their execution.” </EM> <BR /><BR />As a result, we first brought 1 ms (millisecond) time accuracy to Windows Server 2016 meeting some of the regulatory requirements – This is supported in-market today.&nbsp; However, our work was not done, and so Windows Server 2019 makes improvements to comply with these regulations and allow Windows to be the preferred choice for workloads with time dependencies. &nbsp;Now, let’s talk a little bit about the features you’ll find in Windows Server 2019 and current <A href="#" target="_blank" rel="noopener"> Insider builds </A> .</P> <BLOCKQUOTE><BR /><STRONG> Important </STRONG> ! While many of our efforts directly address concerns from regulated industries, <BR />this technology applies to any industry, application, or cloud-service with a time dependency.</BLOCKQUOTE> <P><BR />There's a lot of content in this article (because we did a lot!) - here's a quick summary of the information you'll see in this article <BR /><BR /></P> <UL> <UL> <LI>Compliant Leap Second Support</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Accuracy Improvements (Precision Time Protocol, Software Time-stamping, Clock Source Stability)</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Traceability (including system logging, performance counters, and our work with partners)</LI> </UL> </UL> <P><BR /><BR /></P> <H2><I> </I> Leap Second Support</H2> <P><BR />A leap-second is an occasional 1-second adjustment to UTC.&nbsp; Now you may be thinking, “why on earth would anybody need to adjust UTC?”&nbsp; As the earth’s rotation slows, <A href="#" target="_blank" rel="noopener"> UTC </A> (an atomic timescale) diverges from <A href="#" target="_blank" rel="noopener"> mean solar time </A> or astronomical time.&nbsp; Once UTC has diverged by at most .9 seconds, a <A href="#" target="_blank" rel="noopener"> Leap Second </A> is inserted to keep UTC in-sync with mean solar time.&nbsp; Since the practice of inserting leap seconds began in 1972, a leap second has typically occurred every 18 months (for more information, please see the <A href="#" target="_blank" rel="noopener"> Leap Second FAQ </A> ). <BR /><BR />In the US, the maximum end-to-end divergence from UTC(NIST) is 50ms – It’s even more strict in the EU.&nbsp; This requires that Windows Server 2019 be able to maintain accuracy during a Leap Second.</P> <BLOCKQUOTE><BR /><STRONG> Note </STRONG> : It’s not enough to apply leap-seconds; it matter how you apply them. <BR />Leap-second smearing has been condemned by the Time authorities at NIST and other national labs <BR />around the world.&nbsp; As such, Microsoft <STRONG> will not </STRONG> include a smearing option in Windows Server 2019. <BR /><BR />Keep reading to understand the difference between the Microsoft approach, <BR />and the non-compliant practice of leap second smearing.</BLOCKQUOTE> <P><BR />To most, this seems like a such a simple idea – just add 1 more <SUB> tiny, </SUB> <SUB> little, </SUB> <SUB> insignificant </SUB> <SUB> second </SUB> to the day.&nbsp; As IT Pros, we remember all those Y2K shenanigans that had us (rightfully) a little...well..worried... <BR /><BR />So how does a leap second actually work?&nbsp; Normally, computers keep seconds from 0 through 59 for a total of 60 seconds.&nbsp; When a leap second occurs, an extra second is added to the last minute of the UTC day and the clock goes from 0 through 60 for a total of 61 seconds. <BR /><BR />On the clock it looks like this (in my time zone, the last minute of the UTC day is actually 4:59 PM local time):</P> <TABLE> <TBODY> <TR> <TD><STRONG> Without a Leap Second </STRONG></TD> <TD><STRONG> With a Leap Second </STRONG></TD> </TR> <TR> <TD>16:59:58</TD> <TD>16:59:58</TD> </TR> <TR> <TD>16:59:59</TD> <TD>16:59:59</TD> </TR> <TR> <TD><STRONG> 17:00:00 </STRONG></TD> <TD><STRONG> 16:59:60 </STRONG></TD> </TR> <TR> <TD>17:00:01</TD> <TD>17:00:00</TD> </TR> <TR> <TD>17:00:02</TD> <TD>17:00:01</TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 364px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75226i5B9C1A80D8B228BA/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR /><BR /></P> <BLOCKQUOTE><BR /><STRONG> Important: </STRONG> Some of the "gurus" out there (I’m looking at you Neil deGrasse Tyson) might <BR />rightfully say “technically, there can be both positive or negative leap seconds.&nbsp; A positive <BR />leap second adds one second and a negative leap second removes one second from the day.” <BR /><BR />Rest assured Neil, while a negative leap second has never actually occurred, if it does, you <BR />can still celebrate your leap seconds with very tiny bottles of champagne – We’ll support both :smiling_face_with_smiling_eyes:</img> <BR /><A href="#" target="_blank" rel="noopener">https://twitter.com/neiltyson/status/615269855835631616</A> <BR />Here's how it looks with a negative leap second <BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /><BR /> <TABLE> <TBODY> <TR> <TD><STRONG> Without a Leap Second </STRONG></TD> <TD><STRONG> With a Leap Second </STRONG></TD> </TR> <TR> <TD>16:59:57</TD> <TD>16:59:57</TD> </TR> <TR> <TD>16:59:58</TD> <TD>16:59:58</TD> </TR> <TR> <TD><STRONG> 16:59:59 </STRONG></TD> <TD><STRONG> 17:00:00 </STRONG></TD> </TR> <TR> <TD>17:00:00</TD> <TD>17:00:01</TD> </TR> <TR> <TD>17:00:01</TD> <TD>17:00:02</TD> </TR> </TBODY> </TABLE> <BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 364px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75227i05E7DE9BD8B1888F/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR /></BLOCKQUOTE> <P>&nbsp;</P> <H3>The problem with Leap Second smearing</H3> <P><BR />As noted above, we will not include a leap second smearing option.&nbsp; Leap Second smearing (where you carve the extra second up into smaller units and add them throughout the day) has "an error of order ±0.5 s with respect to the definition of UTC" (see below).&nbsp; As noted previously, this will not meet the accuracy requirements in these regulated industries and as outlined below, there is no standard method for applying smearing frequency adjustments which can lead to a disagreement in time stamps.&nbsp; As such, smearing does not meet customer regulatory requirements. <BR /><BR />In their 2018 paper, " <A href="#" target="_blank" rel="noopener"> <STRONG> Metrological and legal traceability of time signals </STRONG> </A> ", presented at the Precise Time and Time Interval Meeting, industry leaders from NIST and USNO outlined these two primary problems with Leap Second smearing:</P> <P><EM> Some corporations, in an attempt to minimize the impact on their systems and eliminate the discontinuity, have implemented “smears”, that slow down their clocks for a period around the time of the leap second insertion. </EM></P> <P>&nbsp;</P> <P><EM> This method has the </EM> <EM> advantage that the time stamps are monotonically increasing even in the vicinity of the leap second, <STRONG> but it has an error of order </STRONG> </EM> <STRONG> <EM> ± </EM> </STRONG> <STRONG> <EM> 0.5 s with respect to the definition of UTC </EM> </STRONG> <EM> . </EM></P> <P>&nbsp;</P> <P><EM> In addition, there is no standard </EM> <EM> method for applying this frequency adjustment, so that different implementations may disagree among themselves in addition to the time error with respect to UTC. </EM></P> <P><BR />I’m sure there will be many implementation and application compatibility questions stemming from this article; please stay tuned for more detailed information.&nbsp; &nbsp;In the meantime, please note that regular day-to-day operations, you won’t need to change anything.&nbsp; Check the “Leap Seconds for the Dev” validation guide for examples and stay tuned for further guidance. <BR /><BR /><BR /><BR />Ready to give it a shot!? &nbsp; Download the latest Insider build and Try it out! <BR /><A href="#" target="_blank" rel="noopener"> Leap Seconds for the IT Pro </A> <A href="#" target="_blank" rel="noopener"> Leap Seconds for the Dev </A> <BR /><BR /><BR /><BR /></P> <H2>Accuracy Improvements</H2> <P><BR />We’re also improving our inherent accuracy in the platform.&nbsp; First, why is it so hard to get the time right!?&nbsp; While the answer may not be immediately apparent, there are a lot of pieces working against time-sensitive systems, <STRONG> <EM> some </EM> </STRONG> of which I’ve listed below: <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 998px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75228i4D9F14A2EF35A24A/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR />Here’s some of the work we did to address each of the challenges listed above: <BR />Precision Time Protocol: <BR />In Windows Server 2019, Windows will include a new time synchronization protocol called Precision Time Protocol (PTP).&nbsp; You may be asking yourself what’s wrong with NTP?&nbsp; It’s served us well for so many years! <BR /><BR />Think back to the last thunderstorm you saw&nbsp; – Did you see lightning and hear thunder at the same time?&nbsp; Unless you’re very close to the storm, you’ll likely detect an audible delay after you’ve seen the lightning.&nbsp; How much of an audible delay are you experiencing?&nbsp; The delay is not based strictly on the speed of sound and your distance from the storm.&nbsp; It's also affected by buildings or other influences that introduce additional acoustic delay.&nbsp; If you want to know just how close you are to the storm, you'd have to consider all the influences.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 997px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75229i50ED19A95BA0CE29/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Likewise, there is delay (latency) introduced in the timing packets being passed from the time server across the network.&nbsp; If that delay is not accounted for, or if it is not symmetric (equal in both directions – to and from the client), then it becomes increasingly difficult for the client to properly apply the time stamp sent from the time server.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75230iB04899D0ABD20AF3/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />Network Time Protocol (NTP) has long been the primary time synchronization method for Windows but unfortunately, NTP does not have a solution to this problem; NTP assumes that the round-trip delay introduced by the network is symmetric. <BR /><BR />Enter Precision Time Protocol ( <A href="#" target="_blank" rel="noopener"> IEEE 1588v2 </A> ). &nbsp;PTP enables network devices to add the latency introduced by each network device into the timing measurements thereby providing a far more accurate time sample to the endpoint (Windows Server 2019 or Windows 10, host or virtual machine). <BR /><BR />Precision Time Protocol is not for everyone; due to the network configuration requirements, NTP will continue as the default protocol.&nbsp; However, for customers with the highest of accuracy requirements, you can drive towards even higher accuracy systems using our inbox PTP Client in Windows Server 2019. <BR /><BR /><BR /><BR />Ready to give it a shot!? Download the latest Insider build and <A href="#" target="_blank" rel="noopener"> Try it out! </A> <BR /><BR /><BR /><BR />Software Timestamping: <BR />When a timing packet is received over the network from a time server it must be processed by the OS’ networking stack prior to being consumed in the time service.&nbsp; Each component in the networking stack introduces a variable amount of latency that affects the accuracy of the timing measurement.&nbsp;&nbsp;This may sound insignificant, but this can add 30µs and in extreme scenarios closer to 200µs.&nbsp; You may remember from earlier in this article, some systems are targeting sub-100µs accuracy! <BR /><BR />In addition, there may be many other services on the system all looking for data from the network.&nbsp; As a simple example, imagine a SQL Server with remote databases, or file servers with SAN/NAS storage that also require time accuracy.&nbsp; Packets for these workloads would all compete with the Windows Time service packets attempting to traverse the networking stack introducing additional delay.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 891px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75231iCD063D95BAA8D86E/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR />To address this problem, we timestamp packets before and after the "Windows Networking Components" shown above. Now we can improve time accuracy by accounting for software delays! <BR /><BR /><BR /><BR />Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank" rel="noopener"> Try it out! </A> <BR /><BR /><BR /><BR />Clock Source Stability <BR />Our final accuracy-based improvement actually affects the stability of the clock.&nbsp; It’s not enough to have an accurate clock occasionally; you must maintain that accuracy over long periods of time.&nbsp; It’s important to understand that a host system receives time “samples” from its time server, however it does <STRONG> not </STRONG> immediately apply these samples to the clock. <BR /><BR />You can imagine that if a time sample is subject to variable network delay (among other unpredictable network challenges) and we immediately stepped the clock to match every time sample, the clock would likely be incorrect fairly often - it could even move backwards - a problem that would certainly make for a rainy day in the life of an IT Pro... <BR /><BR />Instead we take multiple time samples, eliminate the outliers, and <STRONG> discipline the clock </STRONG> with the goal of bringing the system closer and closer to synchronization with the time server. <BR /><BR />Disciplining the clock entails making adjustments to gradually converge on the correct time.&nbsp; Ultimately there is a natural limit to how small of a change we can make but the key is that smaller is better.&nbsp; Just how granular can we get?&nbsp; This is a complicated question but is based on the frequency of the QPC clock.</P> <BLOCKQUOTE><BR />For a more in-depth look at this subject including QPC, please reference <A href="#" target="_blank" rel="noopener"> this </A> article.</BLOCKQUOTE> <P><BR />Previous versions of Windows allowed for a QPC granularity (the smallest change we could make to the system clock) of 6.4 µs/second (microseconds / second). &nbsp;In Windows Server 2019, the QPC granularity drops to 100 nanoseconds / second!&nbsp; This is akin to the difference in clarity between 480p and 4K television.&nbsp; There is much finer granularity in the 4K picture! <BR /><BR />So why does all this matter?&nbsp; Well accuracy as measured over time is reflective of your stability; not only can we hit the bulls-eye, we can hit the bulls-eye over and over again. &nbsp;In a 3.5-day measurement, our partners at <STRONG> <A href="#" target="_blank" rel="noopener"> Sync-N-Scale </A> </STRONG> measured, and NIST corroborated, Windows Server 2019 pre-release bits.&nbsp; In the picture below, notice the MIN Time Offset reports 41µs (microseconds) RMS diverged from UTC(NIST)!</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 999px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75232iC975D0140ED535E7/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P>&nbsp;</P> <P><I> <EM> <STRONG> Note </STRONG> : The <STRONG> AVG method </STRONG> involves comparing the system under test to UTC(NIST) every 10 seconds, then averaging these measurements for 10 minutes (60 readings). UTC(NIST) is available with 0.0001 ms resolution. The difference between the two 10-minute averages is the difference between the time broadcast by the server and UTC(NIST). </EM> </I></P> <P>&nbsp;</P> <P><I> <EM> The <STRONG> MIN method </STRONG> involves comparing each NTP server to UTC(NIST) every 10 seconds for a 10 minute interval (60 measurements). However, only one of the 60 measurements is saved, the one with the shortest round trip delay. This method is based on the assumption that NTP measurements with the shortest round trip delays provide the best estimate of the true time difference. </EM> </I></P> <P><BR />This leads me to our last topic, Traceability.</P> <H2>Traceability</H2> <P><BR />Self-proclaimed accuracy is not enough – you must be able to prove, or trace, your accuracy to a known reference time source.&nbsp; In the US, this would be UTC(NIST).&nbsp; Traceability is a multi-faceted aspect of the regulations.&nbsp; FINRA for example, states: <BR /><BR /><EM> Members must document and maintain their clock synchronization procedures. Among other requirements, members must keep a log of the times when they synchronize their clocks and the results of the synchronization process. </EM> <BR />System Logging <BR />The first step in meeting these requirements is auditing changes and synchronization of the local system.&nbsp; To do this, Windows Server 2019 will include additional logging capabilities that can be used to audit the actions taken by the Windows Time service.&nbsp; We’ve documented the full list of events <A href="#" target="_blank" rel="noopener"> here </A> .&nbsp; These logs can be used to answer the questions above, such as: <BR /><BR /></P> <OL> <OL> <LI>What is the chosen time server and synchronization frequency</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>When was the last synchronization and results of that synchronization</LI> </OL> </OL> <P>&nbsp;</P> <OL> <OL> <LI>What actions were taken after the synchronization (did we discipline the clock?)</LI> </OL> </OL> <P><BR /><BR />These logs are contained in a standard event log channel called Time-Service (more details in the link provided) and can be queried and forwarded by your SIEM of choice. <BR />Performance Counters <BR />We also have performance counters that allow you to observe and troubleshoot a number of critical time-related areas. &nbsp;In the picture below, you can see two of the included counters, the Computed Time Offset (in microseconds) and the NTP Roundtrip Delay (also in microseconds). <BR /><BR />The <STRONG> Computed Time offset </STRONG> is the absolute time offset between the system clock and the chosen time source, as computed by W32Time Service - This number should be as small as possible indicating how close your clock is synchronized with the reference clock.&nbsp; The <STRONG> NTP Roundtrip Delay </STRONG> is the time elapsed on the NTP client between transmitting a request to the NTP server and receiving a valid response from the server - The higher this number, the harder it will be to maintain an accurate clock.&nbsp; There are other counters and we encourage you to explore and provide some feedback!</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 998px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75233i5DF29EFCC8087BC0/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR /><BR />SCOM Management Pack <BR />If your monitoring system includes SCOM, you could also leverage a SCOM management pack that allows you to monitor and alert when a specified NTP Offset threshold is exceeded for a particular node. <BR /><BR /><BR /><BR />Ready to give it a shot!? &nbsp; Download the latest Insider build and <A href="#" target="_blank" rel="noopener"> Try it out! </A> <BR /><BR /><BR /><BR />Completing the Unbroken Chain <BR />Dr. Judah Levine of NIST defines traceability as requiring an <A href="#" target="_blank" rel="noopener"> unbroken chain of measurements </A> .&nbsp; While Windows can provide information about its local system, traceability requires timing information from the entire chain of time sources as well - This is more than what Windows alone can provide.&nbsp; Windows Server 2019 can participate in a fully traceable environment through our partners like <A href="#" target="_blank" rel="noopener"> Sync-N-Scale </A> and <A href="#" target="_blank" rel="noopener"> Spectracom </A> , .&nbsp; Shown here is the partner solution from <A href="#" target="_blank" rel="noopener"> Spectracom </A> :</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 588px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75234i9775EEAE4FAE8E2D/image-size/large?v=v2&amp;px=999" role="button" /></span></P> <P><BR /><BR /></P> <H2>Summary</H2> <P><BR />Previous time accuracy requirements were lax by today’s standards.&nbsp; Now regulated industries have much more stringent accuracy requirements but accuracy alone is not enough – Your systems must also be traceable. <BR /><BR />Windows Server 2019 meets the current accuracy and regulatory requirements required for time-sensitive workloads through a variety of improvements including compliant and accurate time during a leap second, a new time synchronization method in Precision Time Protocol, inherent platform improvements for stability, and lastly (but equally important), system-wide and end-to-end traceability.&nbsp; You can use Windows Server 2019 for time-sensitive workloads, whether you’re in a regulated industry, application, or cloud service. <BR /><BR />I’m sure there will be additional questions about some of these features as we near Windows Server 2019 launch at Ignite; please stay tuned as we’ll update our public documentation and provide additional blogs on this site as necessary.&nbsp; Please give our validation guides (shown in the <STRONG> Try it Out </STRONG> links above!)&nbsp;a shot!&nbsp; And most importantly, let us know what you think in the comments! <BR /><BR />For the Windows Core Networking Team, <BR /><BR />Dan “Sometimes my seconds Leap” Cuomo <BR /><BR /><BR /><BR /><STRONG> Here's a list of all the Try it Out! sections in this blog </STRONG></P> <P>Leap Seconds for the IT Pro - <A href="#" target="_blank" rel="noopener"> Try it out! </A></P> <P>&nbsp;</P> <P>Leap Seconds for the Dev - <A href="#" target="_blank" rel="noopener"> Try it out! </A></P> <P>&nbsp;</P> <P>Precision Time Protocol - <A href="#" target="_blank" rel="noopener"> Try it out! </A></P> <P>&nbsp;</P> <P>Software Timestamping - <A href="#" target="_blank" rel="noopener"> Try it out! </A></P> <P>&nbsp;</P> <P>High Accuracy Validation Guide - <A href="#" target="_blank" rel="noopener"> Try it out! </A></P> Mon, 16 Sep 2019 03:49:20 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/top-10-networking-features-in-windows-server-2019-10-accurate/ba-p/339739 Dan Cuomo 2019-09-16T03:49:20Z Announcing: Transport Features and Performance Advancements in Fall Creators Update for Windows 10 and Windows Server 2016 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/announcing-transport-features-and-performance-advancements-in/ba-p/339726 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Jul 10, 2018 </STRONG> <BR /> <STRONG> <B> Authors: </B> Praveen Balasubramanian,&nbsp;Daniel Havey </STRONG> <BR /> <BR /> Windows core networking team has been introducing innovations in transport and blogging about it since the <A href="#" target="_blank"> Anniversary Edition </A> .&nbsp; We do this because we are committed to providing continuous performance improvements, better security, reliability, battery life and diagnostics as well as making Windows a better member of the Internet community.&nbsp; Here is a summary for Fall Creators Update: <BR /> <OL> <BR /> <LI> LEDBAT for background connections IETF RFC 6817 [2] </LI> <BR /> <LI> TCP Cubic [3], [4] </LI> <BR /> <LI> TCP Fast Open (TFO) for zero RTT TCP connection setup. IETF RFC 7413 [1] </LI> <BR /> <LI> Software Receive Side Coalescing </LI> <BR /> </OL> <BR /> <STRONG> LEDBAT </STRONG> for Windows is an inbox experimental non-interference technology designed to keep heavy workflows (such as system updates) from interfering with normal network usage.&nbsp; LedBat has a great deal of capabilities not found in other non-interference technologies because it is a kernel level transport flow control. <BR /> <OL> <BR /> <LI> LedBat can sense and adapt to network data flows from <EM> any other systems anywhere on the network. </EM> </LI> <BR /> <LI> LedBat <EM> is topologically agnostic </EM> .&nbsp; It will find and measure the loading characteristics of the bottleneck link in any network equipment. </LI> <BR /> <LI> LedBat is <EM> faster and more efficient </EM> than technologies based on non-transport layer flow control techniques. </LI> <BR /> </OL> <BR /> In addition to our original testing of LedBat on the WAN (Figure 1 lower left) we are in the process of testing LedBat on the LAN.&nbsp; The test topology is shown at the top of Figure 1.&nbsp; A system connected to Microsoft corpnet provides the test workloads.&nbsp; There are 10 bulk traffic loads transported with LedBats shown in blue and one regular workload transported with Cubic TCP and shown in orange.&nbsp; Some preliminary results are shown in the lower left-hand corner of Figure 1.&nbsp; The 10 buld data flows using LedBats start first and use all of the available bandwidth.&nbsp; Every 10 seconds a regular workflow is started using TCP Cubic.&nbsp; The results speak for themselves.&nbsp; All 10 LedBats are out of the way immediately one the regular workflow starts and when it is done the 10 LedBats return to full utilization of the link.&nbsp; We encourage data transport professionals and Creators to join us in experimentation.&nbsp; For more details join the Windows 10 <EM> discussion group at: <A href="https://gorovian.000webhostapp.com/?exam=Win10talk@microsoft.com" target="_blank"> mailto:Win10talk@microsoft.com </A> </EM> <BR /> <BR /> <STRONG> <EM> <B> Figure 1 -- LEDBAT Testbed Topologies in Redmond </B> </EM> </STRONG> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75223iB15F38208C51369B" /> <BR /> <STRONG> <EM> <B> Figure 2 -- Windows Pluggable&nbsp; CC Algorithms </B> </EM> </STRONG> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75224i3C2CB20624C33CA2" /> <BR /> <BR /> <B> TCP Cubic </B> is the latest addition to Windows pluggable Congestion Control (CC) algorithm modules bringing the total to 4.&nbsp; Figure 2 is a chart to help describe the passive aggressive range of CC alg options.&nbsp; All CC algs attempt to take their min max fair share of the network bandwidth.&nbsp; However, some algs are more aggressive than others about it.&nbsp; More aggressive algs such as Cubic are placed in the upper left quadrant because they tend to grab a little more of their bandwidth share and create a little extra latency with their aggressive behavior.&nbsp; &nbsp;Less aggressive algs such as New Reno are towards the lower right quadrant.&nbsp; New Reno creates less latency, but, will lose a little bandwidth in competition with more aggressive algs.&nbsp; Notice that LedBat looks a little different from the other algs and is placed separately from them.&nbsp; This is because LedBat is a specialized CC algorithm that is designed to “not compete” with other algs.&nbsp; LedBat’s distribution is bi-modal being in the upper right quadrant (high throughput/low latency) except when in competition with an “aggressive” algorithm.&nbsp; For more details join the Windows 10 discussion group at: <A> mailto:Win10talk@microsoft.com </A> <STRONG> <EM> </EM> </STRONG> <BR /> <BR /> <B> TCP Fast Open </B> is a latency reducing technology that achieves zero RTT TCP connections using secure TFO Cookies in the TCP options field.&nbsp; The problem is that middleboxes sometimes do not understand the TFO Cookie option even though it is well documented in <A href="#" target="_blank"> RFC 7413 </A> .&nbsp; These misguided middleboxes mangle TCP connections tampering with TCP connection semantics and distorting or destroying the TCP connection.&nbsp; Because of this questionable behavior no browser has been able to deploy TFO "on by default".&nbsp; Until now!&nbsp; Using Windows TCP fallback algorithm the Edge browser has successfully deployed TCP Fast Open and Edge users are able to enjoy zero RTT TCP connections. For more details join the Windows 10 discussion group at: <A> mailto:Win10talk@microsoft.com </A> <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75225iF3DCD1057675D9B0" /> <BR /> <STRONG> <EM> <B> Figure 3 -- TCP Fast Open on the Datapath with Windows TCP Fallback </B> </EM> </STRONG> <BR /> <BR /> <B> TCP and UDP data path upgrades </B> including software Receive Side Coalescing (RSC) have nearly doubled throughput for both UDP and TCP send/receive data paths.&nbsp; This data was collected using microbenchmarks and inhouse (Redmond) testbeds. <BR /> <BR /> We invite you to join us in this journey of Windows Networking development and experimentation by following the Windows 10 discussion group at: <A href="https://gorovian.000webhostapp.com/?exam=Win10talk@microsoft.com" target="_blank"> mailto:Win10talk@microsoft.com </A> <STRONG> <EM> </EM> </STRONG> <BR /> <BR /> <STRONG> Works Cited: </STRONG> <BR /> [1] Y. Cheng et al, "RFC: 7413: TCP Fast Open," December 2014. [Online]. Available: <A href="#" target="_blank">https://tools.ietf.org/html/rfc7413</A> <BR /> [2] S. Shalunov et al, "RFC 6817 Low Extra Delay Background Transport (LEDBAT)," December 2012. [Online]. Available: <A href="#" target="_blank">https://tools.ietf.org/html/rfc6817</A> <BR /> [3] S. Ha et al, "CUBIC: A New TCP-friendly High-speed TCP Variant"," July 2008 </BODY></HTML> Thu, 14 Feb 2019 18:02:03 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/announcing-transport-features-and-performance-advancements-in/ba-p/339726 Daniel Havey 2019-02-14T18:02:03Z Introducing the NetAdapter Driver model for the next generation of networks and applications https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-the-netadapter-driver-model-for-the-next-generation/ba-p/339722 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on May 23, 2018 </STRONG> <BR /> As we move towards a fully connected world, inundated with intelligent devices &nbsp;and massively distributed computing infrastructure, networks that can sustain high bandwidth have never been more relevant. <A href="#" target="_blank"> Initial requirements </A> for a 5G network project peak data rates in the order of 10s of gigabits per second. The gaming and the video streaming applications continue to push the frontier seeking a higher throughput and lower latency data path. In addition, the new breed of developers necessitates a simpler driver model that offers agility and greater reliability. <BR /> <BR /> The Windows core networking team has been hard at work building a new, simpler network driver model.&nbsp;Introducing <STRONG> NetAdapter Class Extension </STRONG> using Windows Driver Framework(WDF) and an <STRONG> u </STRONG> <B> pdated data path, </B> for the next generation of networking on Windows. <BR /> <BR /> <STRONG> NetAdapter Class Extension(NetAdapterCx </STRONG> ) <BR /> <BR /> <STRONG> NetAdapter class extension(NetAdapterCx) </STRONG> module to the <A href="#" target="_blank"> Windows driver Framework </A> can be used to write a <STRONG> driver for the network interface card </STRONG> . NetAdapter brings with it a <STRONG> simpler, easy to use driver model </STRONG> that offloads complexities to WDF and offers improved reliability. Initial focus for this model is on <STRONG> consumer devices with mobile broadband </STRONG> network adapters paving the way for adoption in ethernet and the rest of the ecosystem in the next couple of years. <BR /> <BR /> <STRONG> NOTE </STRONG> : WDF allows developers to implement simple and robust drivers. It has been the model of choice for most Windows driver developers because it abstracts away a lot of complexities such as interacting with the PnP, power and power policy subsystems. <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75219iF012ADD4627D2733" /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> Deriving from <A href="#" target="_blank"> PacketDirect </A> , an experimental Windows Server data plane technology, <STRONG> NetAdapterCx </STRONG> brings an <STRONG> updated data path </STRONG> , that sits below the TCP/IP stack, with improved performance over current NDIS stack by reducing latency and cycles/packet. The new data path is built based on the <STRONG> polling-based IO </STRONG> model vs. the interrupt driven model, allowing the OS to optimize performance. <BR /> <BR /> These improvements not only result in accelerated data paths and better drivers but deliver easier to build drivers. Windows developers can now focus on solving network domain specific problems while leveraging the framework for common device tasks. <BR /> <BR /> But wait, what happened to NDIS? <STRONG> NDIS is not going away anytime soon </STRONG> ! NetAdapter combines the productivity of WDF and the networking performance of NDIS. <BR /> <BR /> <STRONG> Why build NetAdapter? </STRONG> <BR /> <UL> <BR /> <LI> <B> Stay Consistent with WDF: </B> Popular Demand! Yes, we heard YOU! In the past, WDF and NDIS each had advantages but did not interoperate well which meant that only a small subset of WDF features were accessible from the NDIS miniport driver&nbsp;. Whereas now, extending WDF, new OS kernel features are readily available to the NetAdapterCx based driver. This allows the developers to focus most of their effort on enabling their hardware to work on Windows rather than deal with OS complexities. </LI> <BR /> </UL> <BR /> <UL> <BR /> <LI> <STRONG> A simpler driver model: </STRONG> WDF brings a familiar set of abstractions simplifying driver development, making it easier for non NDIS driver developers to write/maintain a NetAdapter based network driver. NDIS pushes many hard problems to the client driver such as data path synchronization, PnP and power handling. NetAdapterCx solves this by taking over the <STRONG> responsibility of synchronizing </STRONG> <STRONG> power and Pnp event with both data and control path </STRONG> IO so that individual client driver is not required to do so.&nbsp; In addition, NetAdapterCx also serializes input to the queues. With NDIS, the <B> client driver runs with elevated privileges </B> and can consequently <B> </B> cause instability to the entire system, resulting in bug checks. <BR /> <UL> <BR /> <LI> By moving complexities to the OS (such as DMA mapping) and using the polling-based IO model, NetAdapter model works towards graceful handling of such scenarios, resulting in improved driver quality and reliability. </LI> <BR /> </UL> <BR /> </LI> <BR /> </UL> <BR /> <UL> <BR /> <LI> <B> Accelerated performance: </B> Moving away from the legacy interrupt-driven model in NDIS, the <STRONG> NetAdapterCx builds on the updated data path and the polling-based IO model of PacketDirect </STRONG> . In the future, the polling model can be optimized for performance. <STRONG> NetPacket </STRONG> , the primary layer of abstraction in the NetAdapter data path, map <STRONG> directly to the NIC hardware queues </STRONG> . This allows Windows to intelligently and transparently scale out in a way that will make maximum use of your NIC’s hardware. Because of this direct mapping, scale out for features like Receive Side Scaling becomes more impactful. </LI> <BR /> </UL> <BR /> <STRONG> Going forward, all NetAdapter drivers will be state separated and DCHU compliant for new, secure systems. </STRONG> <BR /> <BR /> <STRONG> Architectural Overview </STRONG> <BR /> <BR /> The following pictures show the differences between the traditional NDIS architecture and the new NetAdapter model. <BR /> <BR /> [caption id="attachment_5065" align="alignleft" width="214"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75220i4726564D36B78A56" /> <I> Fig1: NDIS Miniport model </I> [/caption] <BR /> <BR /> [caption id="attachment_5075" align="alignleft" width="325"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75221iC4346D974294AE03" /> <I> Fig 2: NetAdapter Class Extension </I> [/caption] <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> Figure 1 above shows the traditional NDIS model. A typical NDIS miniport driver leverages NDIS for all its needs, including PnP, power, control, data and hardware interaction. <BR /> <BR /> Figure 2 above shows a new Miniport driver being a WDF client. It interacts with NDIS via the new NetAdapter Class Extension (NetAdapterCx) but only for data and control. For other needs, such as PnP, power and hardware interaction the new NetAdapter driver uses WDF interfaces. <BR /> <BR /> As you can see, NetAdapterCx still works behind the scenes with NDIS, but handles all the interaction with NDIS. <BR /> <BR /> <STRONG> What is available today? </STRONG> <BR /> <BR /> Starting in Windows 10, version 1703, the Windows Driver Kit (WDK) includes a Network Adapter WDF Class Extension module (NetAdapterCx), <STRONG> for preview </STRONG> .&nbsp;Many <STRONG> thanks to our networking partners and the Windows developer community </STRONG> , without you the NetAdapterCx framework wouldn’t be as mature as it is today. <BR /> <TABLE> <TBODY><TR> <TD> <STRONG> Milestone </STRONG> </TD> <TD> <STRONG> Comments </STRONG> </TD> </TR> <TR> <TD> <STRONG> RS2 Release, Windows 10, version 1703 </STRONG> </TD> <TD> Fully working <STRONG> preview </STRONG> with support for ethernet over any bus (USB, PCIe). <BR /> <UL> <BR /> <LI> We have built prototype NetAdpater based drivers with our partners to validate the framework (over both PCIe and USB bus) </LI> <BR /> <LI> Sample prototype drivers available in the <A href="#" target="_blank"> Github </A> page. </LI> <BR /> </UL> <BR /> </TD> </TR> <TR> <TD> <STRONG> RS3 Release, Windows 10 version 1709 </STRONG> </TD> <TD> Advancements in Performance over RS2 Release </TD> </TR> <TR> <TD> <STRONG> RS4 release, Windows 10, version 1803 </STRONG> </TD> <TD> Stabilization. </TD> </TR> <TR> <TD> <STRONG> Upcoming RS5 release, Windows 10, version 1809 </STRONG> </TD> <TD> Focus on the <STRONG> consumer devices with the Mobile broadband(MBB) network adapters </STRONG> , more details below. Go <A href="#" target="_blank"> here </A> for the latest specification. </TD> </TR> </TBODY></TABLE> <BR /> <B> Marching towards RS5 RTM </B> <BR /> <BR /> [caption id="attachment_5125" align="alignleft" width="300"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75222i557ECD10AC310202" /> <I> NetAdpaterCx in MBB modems </I> [/caption] <BR /> <BR /> <B> </B> The goal in RS5, is to commercialize the new <A href="#" target="_blank"> NetAdapter based Mobile Broadband class extension </A> (MBBCx) and class driver. This new framework will be compatible with in-market modems that rely on the NDIS based MBB USB class driver built using MBIM specification. <BR /> <BR /> <STRONG> We are excited to announce that this framework and class driver is shipping in <A href="#" target="_blank"> RS5 insider builds </A> for you to try! Test it out and do not forget to report issues <EM> . </EM> </STRONG> Help us make windows better for you! <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <STRONG> Long road ahead of us </STRONG> <BR /> <BR /> While we do believe NetAdapter framework is our path forward, we see NDIS 6 and NetAdapter co-existing for the foreseeable future. <BR /> <BR /> After RS5 stabilization for in-market MBB modems, the NetAdapter framework is slated to expand to upcoming consumer devices with PCIe mobile broadband modems. Future focus will be on increasing coverage across other network adapter types. <BR /> <BR /> <STRONG> Can’t wait to try? </STRONG> <BR /> <BR /> It is critical for us to get your feedback to drive adoption, influence future design decisions and roadmap. <BR /> <BR /> We have a ton of great information and resources to share with you. To that end, we invite you <BR /> <OL> <BR /> <LI> To visit our <A href="#" target="_blank"> Github </A> page and hit “follow”. </LI> <BR /> <LI> Visit documentation at <A href="#" target="_blank"> docs.microsoft.com </A> for the API specification. </LI> <BR /> <LI> Check out the video series <A href="#" target="_blank"> here </A> to dive deeper into the topics introduced in this article. </LI> <BR /> <LI> Peruse our NetAdapter based <A href="#" target="_blank"> driver samples </A> . </LI> <BR /> <LI> Experiment building a driver for your hardware. </LI> <BR /> <LI> Most importantly, reach out on NetAdapter@microsoft.com with issues and questions. </LI> <BR /> </OL> <BR /> We truly appreciate and look forward to your feedback and continued engagement! <BR /> <P> </P> </BODY></HTML> Thu, 14 Feb 2019 18:01:36 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/introducing-the-netadapter-driver-model-for-the-next-generation/ba-p/339722 Harini Ramakrishnan 2019-02-14T18:01:36Z Previewing support for same-site cookies in Microsoft Edge https://gorovian.000webhostapp.com/?exam=t5/networking-blog/previewing-support-for-same-site-cookies-in-microsoft-edge/ba-p/339717 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on May 17, 2018 </STRONG> <BR /> Please refer to our Edge blog: <BR /> <BR /> <A href="#" target="_blank">https://blogs.windows.com/msedgedev/2018/05/17/samesite-cookies-microsoft-edge-internet-explorer/</A> </BODY></HTML> Thu, 14 Feb 2019 18:00:59 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/previewing-support-for-same-site-cookies-in-microsoft-edge/ba-p/339717 Gabriel Montenegro 2019-02-14T18:00:59Z Network start-up and performance improvements in Windows 10 April 2018 Update and Windows Server, version 1803 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/network-start-up-and-performance-improvements-in-windows-10/ba-p/339716 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Apr 27, 2018 </STRONG> <BR /> <STRONG> <EM> Increased container density, faster network endpoint creation time, improvements to NAT network throughput, DNS fixes for Kubernetes, and improved developer features </EM> </STRONG> <EM> <STRONG> </STRONG> </EM> <BR /> <BR /> <BR /> <EM> A lot of enthusiasm and excitement surrounds the highly anticipated quality improvements to the container ecosystem on Windows; all shipping with Windows Server version 1803 (WS1803) and Windows 10 April 2018 Update. The range of improvements span long-awaited networking fixes, enhanced scalability and efficiency of containers, as well as new features to make the suite of container networking tools offered to developers more comprehensive. Let's explore some of these improvements and uncover how they will make containers on Windows better than ever before! </EM> <BR /> <H2> <STRONG> Improvements to deviceless vNICs </STRONG> </H2> <BR /> Deviceless vNICs for Windows Server Containers removes the overhead of using Windows PNP device management to make both endpoint creation and removal significantly faster. Network endpoint creation time in particular can have a notable impact on large-scale deployments, where scaling up and down can add unwanted delay. Windows 10 April 2018 Update and WS1803 achieves better performance than its predecessors, as the data below will show. <BR /> <BR /> WS1803 is Microsoft’s best-of-breed release to date in terms of providing a seamless scaling experience to consumers that expect things to “just work” in a timely fashion. <BR /> <BR /> To summarize the impact of these improvements: <BR /> <UL> <BR /> <LI> Increased scalability of Windows Server Containers from 50 to 500 containers on one host with linear network endpoint creation cost </LI> <BR /> <LI> Decreased Windows Server Container start-up time with 30% improvement in network endpoint creation time and 50% improvement in time taken for container deletion </LI> <BR /> </UL> <BR /> <H3> <STRONG> Before vs. after </STRONG> </H3> <BR /> As discussed above, container vNIC creation and deletion was one of the identified bottlenecks for scaling requirements of powerhouse enterprises today. In previous Windows releases, with PNPs required for container instantiation, we saw on average <EM> 10 container creations fail out of 500 </EM> . Now, with deviceless vNIC’s, we don’t see <STRONG> <EM> any failures </EM> </STRONG> for 500 container creations. <BR /> <BR /> See the graphs below for a quick visualization of the trends discussed: <BR /> <BR /> [caption id="attachment_4935" align="alignnone" width="512"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75213i6BEDC5169AFA781A" /> Figure 1 - Container Creation: PNP vs. deviceless vNICs[/caption] <BR /> <BR /> [caption id="attachment_4945" align="alignnone" width="480"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75215i4C5C44A7E8BE2E98" /> Figure 2 - Container Deletion: PNP vs. deviceless vNICs[/caption] <BR /> <BR /> In addition to this, check out the stress test below that captures the new, lightning-fast multi-container deployment creation time! <BR /> <H3> <STRONG> Stress test: c </STRONG> <STRONG> ontainer endpoint creation time </STRONG> </H3> <BR /> <BR /> <STRONG> Description </STRONG> <BR /> PowerShell script that creates and starts specified amount of recent <EM> “ </EM> <A href="#" target="_blank"> <EM> microsoft/windowsservercore </EM> </A> <EM> ” </EM> Windows Server containers (build 10.0.17133.73) on a Windows Server, version 1803 host (build 17133) using the default “NAT” network driver. <BR /> <STRONG> Hardware specification </STRONG> <BR /> <UL> <BR /> <LI> C6220 Server </LI> <BR /> <LI> Storage: 1 400GB SSD </LI> <BR /> <LI> RAM: 128GB </LI> <BR /> <LI> CPU: 2x E5-2650 v2 2.6Ghz 16c each (32c total) </LI> <BR /> <LI> Networking: 1 GB Intel(R) I350 Gigabit Network Connector </LI> <BR /> </UL> <BR /> <STRONG> Test results </STRONG> <BR /> <TABLE> <TBODY><TR> <TD> <STRONG> Number &nbsp;of Containers </STRONG> </TD> <TD> <STRONG> Average HNS endpoint creation time (switch+port+vfp) (ms) </STRONG> </TD> </TR> <TR> <TD> 10 </TD> <TD> 104.6 </TD> </TR> <TR> <TD> 50 </TD> <TD> 126.28 </TD> </TR> <TR> <TD> 100 </TD> <TD> 150.3 </TD> </TR> </TBODY></TABLE> <BR /> <EM> Figure 3 – Table of HNS endpoint creation time. <I> Wondering what HNS is? <A href="#" target="_blank"> See here </A> </I> </EM> <BR /> <BR /> <STRONG> Container endpoint creation time (ms) vs. number of container instances </STRONG> <EM> <B> </B> </EM> <BR /> [caption id="attachment_4975" align="alignnone" width="658"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75216iF9EED8982539BFD6" /> Figure 4 – Stress test: container endpoint creation time graph[/caption] <BR /> <H3> <STRONG> Test discussion </STRONG> </H3> <BR /> The results show that container creation performance follows a stable linear trend, with creation time scaling to an average of 150ms on servers with 100 endpoints&nbsp;. <BR /> <BR /> In other words, on our experimental hardware we can roughly estimate Windows server container creation time “ <EM> t </EM> ” against number of endpoints on server <EM> “n” </EM> very easily using the simple relationship <EM> t = n/2 +100 </EM> . <BR /> <BR /> This shows that the daunting task of twiddling your thumbs waiting for deployment to <EM> finally </EM> launch is much more agreeable and foreseeable on WS1803. <BR /> <H2> </H2> <BR /> <H2> <STRONG> NAT Performance Improvements </STRONG> </H2> <BR /> Several bespoke Windows use-cases including Windows Defender Application Guard in the Edge web browser or Docker for Windows rely heavily on network address translation (NAT), so investments into one comprehensive and performant NAT solution is another built-in benefit of moving to this new release. <BR /> <BR /> Alongside improvements in deviceless vNICs, here are some <EM> additional </EM> <EM> optimizations </EM> which are applicable to the NAT network datapath: <BR /> <UL> <BR /> <LI> Optimizations (CPU utilization) of machinery used for translation decisions of incoming traffic </LI> <BR /> <LI> Widened network throughput pipeline by 10-20% </LI> <BR /> </UL> <BR /> This alone is already a great advocate for moving to the new release, but watch this space for even more awesome optimization goodies ( <EM> in the near future!) </EM> that are actively being engineered! <BR /> <BR /> <BR /> <H2> <STRONG> Improvements to Developer Workflows and Ease of Use </STRONG> </H2> <BR /> In previous Windows releases, there existed gaps to the flexibility and mobility needs of modern developers and IT admins. Networking for containers was one such space where gaps were identified that prevented both developers and IT admins from having a seamless experience with containers; they couldn’t confidently develop containerized applications due to a lack of development convenience and network customization options&nbsp;. The goal in WS1803 was to target two fundamental areas of the developer experience around container networking that need improvement— <EM> localhost/loopback </EM> support, and <EM> HTTP proxy </EM> support for containers. <BR /> <H3> 1. &nbsp; &nbsp; HTTP proxy support for container traffic </H3> <BR /> In WS1803 and Windows 10 April 2018 Update, functionality is being added to allow container host machines to inject proxy settings upon container instantiation, such that container traffic is forced through the specified proxy. This feature will be supported on both Windows Server and Hyper-V containers, giving developers more control and flexibility over their desired container network setup. <BR /> <BR /> While simple in theory, this is easiest to explain with a quick example. <BR /> <BR /> Let’s say we have a host machine configured to pass through a proxy that is reachable under <EM> proxy.corp.microsoft.com </EM> and port number <EM> 5320 </EM> . &nbsp;Inside this host machine, we also want to create a Windows server container, and force any north/south traffic originating from the containerized endpoints to pass through the configured proxy. <BR /> <BR /> Visually, this would look as follows: <BR /> <BR /> [caption id="attachment_4955" align="alignnone" width="1392"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75217i8CC1F9401177E313" /> Figure 5 - Container proxy configuration[/caption] <BR /> <BR /> <A href="#" target="_blank"> </A> <BR /> <BR /> The corresponding actions to configure Docker to achieve this would be: <BR /> <BR /> For Docker 17.07 or higher: <BR /> <UL> <BR /> <LI> Add this to your config.json: </LI> <BR /> </UL> <BR /> { <BR /> "proxies": { <BR /> "default": { <BR /> "httpProxy": "<A href="#" target="_blank">http://proxy.corp.microsoft.com:5320</A>" <BR /> } <BR /> } <BR /> } <BR /> For Docker 17.06 or lower: <BR /> <UL> <BR /> <LI> Run the following command: </LI> <BR /> </UL> <BR /> docker run -e "HTTP_PROXY=<A href="#" target="_blank">http://proxy.corp.microsoft.com:5320&nbsp;</A>" -it&nbsp; microsoft/windowsservercore &lt;command&gt; <BR /> Diving deeper from a technical standpoint, this functionality is provided through three different registry keys that are being set <EM> inside the container </EM> : <BR /> <OL> <BR /> <LI> Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\[DefaultConnectionSettings\WinHttpSettings </LI> <BR /> <LI> Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings </LI> <BR /> <LI> Software\Policies\Microsoft\Windows\CurrentVersion\Internet Settings </LI> <BR /> </OL> <BR /> <BR /> <BR /> The configured proxy settings <EM> inside the container </EM> can then be queried using the command: <BR /> netsh winhttp show proxy <BR /> [caption id="attachment_4965" align="alignnone" width="524"] <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75218i0F447D41803C9DB4" /> Figure 6 – Viewing container proxy configuration[/caption] <BR /> <BR /> <A href="#" target="_blank"> </A> <BR /> <BR /> That’s it! Easy, right? The instructions to configure Docker to use a proxy server can be found in the <A href="#" target="_blank"> Docker documentation </A> . <BR /> <BR /> The preliminary PR can be tracked <A href="#" target="_blank"> here </A> . <BR /> <H3> 2. &nbsp; &nbsp; Localhost/loopback support for accessing containers </H3> <BR /> New with the Windows 10 April 2018 Update and WS1803 release is also support for being able to access containerized web services via “localhost” or 127.0.0.1 (loopback). Please see this <A href="#" target="_blank"> blog post </A> that does an excellent job portraying the added functionality. This feature has already been available to Windows Insiders via <A href="#" target="_blank"> Build 17025 </A> on Windows 10 and <A href="#" target="_blank"> Build 17035 </A> on Windows Server. <BR /> <BR /> <BR /> <H2> <STRONG> Networking </STRONG> <STRONG> Quality Improvements </STRONG> </H2> <BR /> One of the most important considerations of both developers and enterprises is a stable and robust container networking stack. Therefore, one of the biggest focus areas for this release was to remedy networking ailments that afflicted prior Windows releases, and to provide a healthy, consistent, and sustainable networking experience of the container ecosystem on Windows. <BR /> <BR /> Windows 10 April 2018 Update and WS1803 users can expect the following: <BR /> <UL> <BR /> <LI> Greatly stabilized DNS resolution within containers <EM> out-of-the-box </EM> </LI> <BR /> <LI> Enhanced stability of Kubernetes services on Windows </LI> <BR /> <LI> Improved recovery after Kubernetes container crashes </LI> <BR /> <LI> Fixes to address and port range reservations through WinNAT </LI> <BR /> <LI> Improved persistence &nbsp;&nbsp;of containers after Host Networking Service (HNS) restart </LI> <BR /> <LI> Improved persistence of containers after unexpected container host reboot </LI> <BR /> <LI> Better overall resiliency of NAT networking </LI> <BR /> </UL> <BR /> We continue being dedicated to stamp out pesky networking bugs. After all, sleepless nights playing whack-a-mole with HNS&nbsp; are no fun (even for us). If you still face container networking issues on the newest Windows release, check out these <A href="#" target="_blank"> preliminary diagnostics </A> and <EM> get in touch </EM> ! </BODY></HTML> Thu, 14 Feb 2019 18:00:56 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/network-start-up-and-performance-improvements-in-windows-10/ba-p/339716 David Schott 2019-02-14T18:00:56Z Windows Server 2016 Software Defined Networking: Updating the Network Controller Server certificate https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-server-2016-software-defined-networking-updating-the/ba-p/339700 <P><STRONG> First published on TECHNET on Mar 19, 2018 </STRONG> <BR />Network Controller uses a single certificate for northbound communication with REST clients (like System Center Virtual Machine Manager) and southbound communication with Hyper-V hosts and Software Load Balancers. A customer may wish to change this certificate after initial deployment, maybe because the certificate has expired or maybe because he wants to move from self-signed certificate to certificates issued by a Certificate Authority. Currently, the workflow to update certificates is broken if you are using System Center Virtual Machine Manager. This will be fixed in an upcoming release. For now, please follow the steps below to update the Network Controller Server certificate. <BR /><BR />NOTE: These steps are not required if you are renewing the existing certificate with the same key. <BR /><BR /><STRONG> Steps to update the Network Controller Server certificate </STRONG> <BR /><BR /></P> <OL> <OL> <LI>Install the new certificate in Personal store of LocalMachine account on a Network Controller node</LI> <LI>Export the certificate with private key and import it on the other Network Controller nodes (to ensure that the same certificate is provisioned on all the nodes). DO NOT remove the old certificate from the Network Controller nodes</LI> <LI>Update the server certificate using the Powershell command:</LI> </OL> </OL> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Set-NetworkController -ServerCertificate &lt;new cert&gt; </EM></P> <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4.&nbsp;Update the certificate used for encrypting the credentials stored in the Network Controller<SPAN style="font-family: inherit;">&nbsp;using the Powershell command:</SPAN></P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Set-NetworkControllerCluster -CredentialEncryptionCertificate &lt;new cert&gt; </EM> <BR />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;5. You will also need to update the certificate used for southbound authentication with Hyper-V hosts and Software Load Balancer MUX virtual machines. To update this, follow steps 6-8</P> <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;6. Retrieve a Server REST resource using the Powershell command:</P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Get-NetworkControllerServer -ConnectionUri &lt;REST uri of your deployment&gt; </EM> <BR />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;7. In the Server REST resource, navigate to the “Connections” object and retrieve the Credential&nbsp;resource with type “X509Certificate”</P> <P><BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Connections":&nbsp;[ </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{ </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "ManagementAddresses":[ “contoso.com" ], </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "CredentialType":&nbsp; "X509Certificate", </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Protocol":&nbsp; null, </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Port":&nbsp; null, </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Credential":{ </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Tags":&nbsp; null, </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "ResourceRef":&nbsp; "/credentials/41229069-85d4-4352-be85-034d0c5f4658",</EM><EM>&nbsp; </EM></P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</EM><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "InstanceId":&nbsp; "00000000-0000-0000-&nbsp; 0000-000000000000",&nbsp;&nbsp;</EM><BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;…&nbsp;&nbsp;</EM><BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;… </EM> <BR /><I>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;} </I> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;} <BR />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;] </EM> <BR />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8. Update the Credential REST resource retrieved above with the thumbprint of the new&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; certificate</P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; $cred </EM> <EM> = </EM> <EM> New-Object </EM> <EM> Microsoft.Windows.Networkcontroller.credentialproperties </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; $cred </EM> <EM> .type= </EM> <EM> "X509Certificate" </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; $cred </EM> <EM> .username= </EM> <EM> "" </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; $cred </EM> <EM> .value= </EM> <EM> "&lt;thumbprint of the new certificate&gt;" </EM> <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; New-NetworkControllerCredential -ConnectionUri &lt;REST uri of the deployment&gt; -</EM></P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ResourceId 41229069- 85d4-4352-be85-034d0c5f4658 -Properties $cred </EM> </P> <P>&nbsp;</P> <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;9. If the new certificate is a self-signed certificate, provision the certificate (without the private key) in the Trusted Root certificate store of all the Hyper-V hosts and Software Load Balancer MUX virtual&nbsp;machines. This is to ensure that the certificate presented by Network&nbsp;Controller is trusted by the southbound devices. If the certificate is not self-signed, ensure&nbsp;that the Certificate Authority that issued the certificate is also trusted by the Hyper-V hosts&nbsp;and the Software Load Balancer MUX virtual machines.</P> <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;10. System Center Virtual Machine Manager (SCVMM) also must be updated to use the new certificate. On the SCVMM machine, execute the following Powershell command:</P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Set-SCNetworkService -ProvisionSelfSignedCertificatesforNetworkService $true -Certificate</EM></P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;$cert -NetworkService $svc </EM> <BR />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Where <EM> NetworkService </EM> is the Network Controller service, <BR /><I>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Certificate </I> is the new Network Controller certificate, and <BR /><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ProvisionSelfSignedCertificatesforNetworkService is $true if you are using a self-signed cert</EM></P> <P><EM>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 11.&nbsp;</EM>Provision the Network Controller certificate (without the private key) in the Trusted Root certificate&nbsp;store of the SCVMM machine</P> <P><BR />After you have verified that the connectivity is working fine, you can go ahead and remove the old Network Controller certificate from the Network Controller nodes.</P> Tue, 11 Aug 2020 23:54:09 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-server-2016-software-defined-networking-updating-the/ba-p/339700 AnirbanPaul 2020-08-11T23:54:09Z The Evolution of RDMA in Windows: now extended to Hyper-V Guests https://gorovian.000webhostapp.com/?exam=t5/networking-blog/the-evolution-of-rdma-in-windows-now-extended-to-hyper-v-guests/ba-p/339699 <P><STRONG> First published on TECHNET on Nov 29, 2017 </STRONG> <BR /><EM> This post written by Don Stanwyck, Senior Program Manager, Windows Core Networking </EM> <BR /><BR />Remote DMA (RDMA) is an incredible technology that allows networked hosts to exchange information with virtually no CPU overhead and with extremely little latency in the end-system.&nbsp; Microsoft has been shipping support for RDMA in Windows Server since Windows Server 2012 and in Windows 10 (some SKUs) since its first release.&nbsp; With the release of Windows Server 1709 Windows Server supports RDMA in the guest.&nbsp; RDMA is presented over the SR-IOV path, i.e., with direct hardware access from the guest to the RDMA engine in the NIC hardware, and with essentially the same latency (and low CPU utilization) as seen in the host. <BR /><BR />This week we published a how-to guide ( Updated Link: <A href="#" target="_blank">https://aka.ms/ConvergedRDMA</A>&nbsp;) on deploying RDMA on native hosts, on virtual NICs in the host partition (Converged NIC), and in Hyper-V guests.&nbsp; This guide in intended to help reduce the amount of time our customers spend trying to get their RDMA networks deployed and working. <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 300px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75205iE958BB4E5225CD56/image-size/large?v=v2&amp;px=999" role="button" /></span> <BR /><BR />As many of my readers are aware, in Windows 2012 we shipped the first version of RDMA on Windows.&nbsp; It supported only native interfaces, i.e., direct binding of the SMB protocol to the RDMA capabilities offered by the physical NIC.&nbsp; Today we refer to that mode of operation as Network Direct Kernel Provider Interface (NDKPI) <STRONG> Mode 1 </STRONG> , or more simply, Native RDMA. <BR /><BR />SMB-Direct (SMB over RDMA) was popular, but if a customer wanted RDMA on a Hyper-V host they had to set up separate NICs for RDMA and for Hyper-V.&nbsp; That got expensive. <BR /><BR />. <BR /><BR /><BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 300px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75206iC0FAA144D9F6C58E/image-size/large?v=v2&amp;px=999" role="button" /></span> With Windows Server 2016&nbsp; came the solution: Converged NIC operation.&nbsp; Now a customer who wanted to use RDMA and Hyper-V at the same time could do so on the same NICs – and even have them in a team for bandwidth aggregation and failover protection.&nbsp;&nbsp; The ability to use a host vNIC for both host TCP traffic and RDMA traffic and share the physical NIC with Guest traffic is called NDKPI <STRONG> Mode 2 </STRONG> . <BR /><BR />New technologies were built on the Converged NIC.&nbsp; Storage Spaces Direct (S2D), for example, delivered the ability to user RDMA for low latency storage across all the hosts in a rack. <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 300px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75207iCB052997B0AE7460/image-size/large?v=v2&amp;px=999" role="button" /></span> That wasn’t enough.&nbsp; Customers told us they wanted RDMA access from within VMs.&nbsp; They wanted the same low latency, low CPU utilization path that the host gets from using RDMA to be available from inside the guest.&nbsp; We heard them. <BR /><BR />Windows Server 1709 supports RDMA in the guest.&nbsp; RDMA is presented over the SR-IOV path, i.e., with direct hardware access from the guest to the RDMA engine in the NIC hardware.&nbsp; (This is NDKPI <STRONG> Mode 3 </STRONG> .) This means that the latency between a guest and the network is essentially the same as between a native host and the network.&nbsp; Today this is only available on Windows Server 1709 with guests that are also Windows Server 1709.&nbsp; Watch for support in other guests to be announced in upcoming releases. <BR /><BR /><BR /><BR />This means that <STRONG> trusted </STRONG> applications in guests can now use any RDMA application, e.g., SMB Direct, S2D, or even 3rd party technologies that are written to our kernel RDMA interface, to communicate using RDMA to any other network entity. <BR /><BR /><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 300px;"><img src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75208i17247CA58839B987/image-size/large?v=v2&amp;px=999" role="button" /></span> Yes, there is that word “trusted” in the previous statement.&nbsp; What does that mean?&nbsp; It means that for today, just like with any other SR-IOV connected VM, the Hyper-V switch can’t apply ACLs, QoS policies, etc., so the VM may do some things that could cause some level of discomfort for other guests or even the host.&nbsp; For example, the VM may attempt to transmit a large quantity of data that would compete with the other traffic from the host (including TCP/IP traffic from non-SR-IOV guests). <BR /><BR />So how can that be managed?&nbsp; There are two answers to that question, one present, and one future.&nbsp; In the present Windows allows the system administrator to affinitize VMs to specific physical NICs, so a concerned administrator could affinitize the VM with RDMA to a separate physical NIC from the other guests in the system (the Switch Embedded Team can support up to 8 physical NICs).&nbsp; In the <STRONG> future </STRONG> , at a time yet to be announced, Windows Server expects to provide bandwidth management (reservations and limits) of SR-IOV-connected VMs for both their RDMA and non-RDMA traffic, and enforcement of ACLs programmed by the host administrator and applying to SR-IOV traffic (IP-based and RDMA).&nbsp; Our hardware partners are busy implementing the new interfaces that support these capabilities. <BR /><BR />What scenarios might want to use Guest RDMA today?&nbsp; There are several that come to mind, and they all share the following characteristics: <BR /><BR /></P> <UL> <UL> <LI>They want low-latency access to network storage;</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>They don’t want to waste CPU overhead on storage networking; and</LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>They are using SMB or one of the 3rd party solutions that runs on Windows Kernel RDMA.</LI> </UL> </UL> <P><BR /><BR />So whether you are using SMB storage directly from the guest, or you are running an application that uses SMB (e.g., SQL) in a guest and want faster storage access, or you are using a 3rd party NVMe or other RDMA-based technology, you can use them with our Guest RDMA capability. <BR /><BR />Finally, while High Performance Computing (HPC) applications rarely run in Guest OSs, some of our hardware partners are exposing the Network Direct Service Provider Interface (NDSPI), Microsoft’s user-space RDMA interface, in guests as well.&nbsp; So if your hardware vendor supports NDSPI (MPI), you can use that from a guest as well. <BR /><BR /><STRONG> RDMA and DCB </STRONG> <BR /><BR />RDMA is a great technology that uses very little CPU and has very low latency.&nbsp; Some RDMA technologies take a heavy reliance on Data Center Bridging (DCB).&nbsp; DCB has proven to be difficult for many customers to deploy successfully.&nbsp; As a result, the view of RDMA as a technology has been affected by the experiences customers have had with DCB – and that’s sad.&nbsp; The product teams at Microsoft are starting to say more clearly what we’ve said in quieter terms in the past: <BR /><BR /><STRONG> <I> Microsoft Recommendation: </I> </STRONG> While the Microsoft RDMA interface is RDMA-technology agnostic, in our experience with customers and partners we find that RoCE/RoCEv2 installations are difficult to get configured correctly and are problematic at any scale above a single rack.  If you intend to deploy RoCE/RoCEv2, you should a) have a small scale (single rack) installation, and b) have an expert network administrator who is intimately familiar with Data Center Bridging (DCB), especially the Enhanced Transmission Service (ETS) and Priority Flow Control (PFC) components of DCB.  If you are deploying in any other context iWarp is the safer alternative.  iWarp does not require any configuration of DCB on network hosts or network switches and can operate over the same distances as any other TCP connection. RoCE, even when enhanced with Explicit Congestion Notification (ECN) detection, requires network configuration to configure DCB/ETS/PFC and/or ECN especially if the scale of deployment exceeds a single rack.  Tuning of these settings, i.e., the settings required to make DCB and/or ECN work, is an art not mastered by every network engineer. <BR /><BR />RoCE vendors have been very actively working to reduce the complexity associated with RoCE deployments.&nbsp; See the list of resources (below) for more information about vendor specific solutions.&nbsp; Check with your NIC vendor for their recommended tools and deployment guidance. <BR /><BR />Additional resources: <BR /><BR /></P> <UL> <UL> <LI>Jose Barreto’s <A href="#" target="_blank" rel="noopener"> 100Gb/s RDMA demo </A></LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Claus Jorgensen’s <A href="#" target="_blank" rel="noopener"> "S2D on Cavium 41000" Blog (iWarp - RoCE comparison) </A></LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Microsoft’s <A href="#" target="_blank" rel="noopener"> sample switch DCB configurations for RoCE </A></LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Mellanox’s <A href="#" target="_blank" rel="noopener"> RDMA/RoCE Community page </A></LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Chelsio's <A href="#" target="_blank" rel="noopener"> Storage Spaces Direct Throughput with iWarp </A></LI> </UL> </UL> <P>&nbsp;</P> <UL> <UL> <LI>Your vendor’s User Guides and Release Notes for your specific network adapter <BR /><BR /></LI> </UL> </UL> <P>&nbsp;</P> Tue, 19 Oct 2021 19:43:13 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/the-evolution-of-rdma-in-windows-now-extended-to-hyper-v-guests/ba-p/339699 Jason Messer 2021-10-19T19:43:13Z SDN Troubleshooting: UDP Communication failures and changing the Network Controller Certificate https://gorovian.000webhostapp.com/?exam=t5/networking-blog/sdn-troubleshooting-udp-communication-failures-and-changing-the/ba-p/339694 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Aug 25, 2017 </STRONG> <BR /> With this blog post, I wanted to highlight a couple of issues that we have encountered recently with Software Defined Networking (SDN) customer deployments in Windows Server 2016. <BR /> <H2> Issue #1: UDP communication isn't working when outbound NAT is configured </H2> <BR /> Customer had configured outbound NAT access for his virtual network through <A href="#" target="_blank"> SCVMM </A> (this internally uses SDN Software Load Balancer), so that machines in the virtual network could access the Internet. The customer noticed that TCP traffic to the Internet was working fine, but all User Datagram Protocol (UDP) traffic was getting dropped. Moreover, this only happened when the Software Load Balancer MUX was on a different HyperV host than the tenant VM. <BR /> <BR /> On deeper analysis, it was revealed that the destination VM was rejecting the packet because the UDP checksum was incorrect. Further investigations revealed a physical NIC issue. The customer was using a physical NIC which was not certified for SDN with Windows Server 2016. The NIC was incorrectly marking the <STRONG> UdpChecksumFailed </STRONG> flag when the inner packet had a valid checksum. <BR /> <BR /> If you are planning to use SDN with Windows Server 2016, please ensure that you use certified NICs. You can verify whether a network adapter is or is not certified by checking the <A href="#" target="_blank"> Windows Server Catalog </A> . <BR /> <BR /> Click <STRONG> Software-Defined Data Center (SDDC) Premium </STRONG> to filter the Windows Server Catalog LAN card list. <BR /> <H2> Issue #2: Changing the Network Controller Server certificate </H2> <BR /> A customer wanted to change the Network Controller server certificate used for communication with the Northbound clients. He was using self-signed certificates and wanted to move to Certificate Authority based certificates. After installing the new certificate on all the Network Controller nodes, he used the <STRONG> Set-NetworkController </STRONG> Powershell command to point Network Controller to the new certificate. <BR /> <BR /> Although the command succeeded, Network Controller communication with SCVMM stopped working. <BR /> <BR /> This is due to a bug in the product where the certificate binding is only changed on one Network Controller node (where the command was run) and is not updated on the other nodes. We are planning to release a fix soon. <BR /> <BR /> As a workaround, you need to manually change the certificate binding on the other Network Controller nodes. Process is as follows: <BR /> <UL> <BR /> <LI> Install the new certificate in Personal store of LocalMachine account </LI> <BR /> <LI> Execute the Powershell command: <STRONG> Set-NetworkController -ServerCertificate &lt;new cert&gt; </STRONG> </LI> <BR /> <LI> Retrieve the thumbprint of the new SSL certificate that you want to use with Network Controller </LI> <BR /> <LI> Double click the certificate and click on Details, note the value of Thumbprint parameter. Remove any spaces in between </LI> <BR /> </UL> <BR /> The following illustration depicts the Thumbprint property of the certificate. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75204i83313CC5A4316B3D" /> <BR /> <BR /> On each Network Controller node, check the SSL binding by executing the following command from a command prompt: <BR /> <BR /> <STRONG> netsh http show sslcert <EM> </EM> </STRONG> <BR /> <BR /> If the result shows the thumbprint of the old certificate, change the binding by executing the following commands: <BR /> <BR /> <STRONG> netsh http delete sslcert ipport=0.0.0.0:443 </STRONG> <BR /> <BR /> <STRONG> netsh http add sslcert certhash= &lt;thumbprint of the new certificate&gt; appid=&lt;application ID&gt; ipport=0.0.0.0:443 certstorename=MY <EM> </EM> </STRONG> <BR /> <BR /> You can retrieve the <STRONG> appid </STRONG> from the output parameter Application ID of the <STRONG> netsh http show sslcert </STRONG> command. <BR /> <BR /> If the binding shows the thumbprint of the new certificate, no further action is needed on that node. <BR /> <BR /> <STRONG> Additional Information </STRONG> <BR /> <BR /> Here are a few links to SDN topics to assist with your planning and deployment: <BR /> <BR /> If you plan to assess your needs and environment for deploying SDN, see the topic <A href="#" target="_blank"> Plan a Software Defined Network Infrastructure </A> . <BR /> <BR /> If you want to deploy SDN using System Center Virtual Machine Manager, see the topic <A href="#" target="_blank"> Set up a Software Defined Network (SDN) infrastructure in the VMM fabric </A> . <BR /> <BR /> If you have any questions/feedback about SDN, send an email to <STRONG> <A href="https://gorovian.000webhostapp.com/?exam=mailto:sdn_feedback@microsoft.com" target="_blank"> sdn_feedback@microsoft.com </A> </STRONG> . </BODY></HTML> Thu, 14 Feb 2019 17:59:05 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/sdn-troubleshooting-udp-communication-failures-and-changing-the/ba-p/339694 AnirbanPaul 2019-02-14T17:59:05Z Core Network Stack Features in the Creators Update for Windows 10 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/core-network-stack-features-in-the-creators-update-for-windows/ba-p/339676 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Jul 13, 2017 </STRONG> <BR /> <STRONG> By: </STRONG> Praveen Balasubramanian and Daniel Havey <BR /> <BR /> This blog is the sequel to our first Windows Core Networking features announcements post.&nbsp; It describes the second wave of core networking features in the Windows Redstone series.&nbsp; The first wave of features is described here: <A href="#" target="_blank"> Announcing: New Transport Advancements in the Anniversary Update for Windows 10 and Windows Server 2016 </A> .&nbsp; We encourage the Windows networking enthusiast community to experiment and provide feedback.&nbsp; If you are interested in Windows Transport please follow our Facebook feedback and discussion page: <A href="#" target="_blank"> @Windows.10.Data.Transport </A> . <BR /> <BR /> <BR /> <H2> TCP Improvements: </H2> <BR /> <H3> TCP Fast Open (TFO) updates and server side support </H3> <BR /> In the modern age of popular Web services and e-commerce , latency is a killer when it comes to page responsiveness. We're adding support in TCP for <A href="#" target="_blank"> TCP Fast Open (TFO) </A> to cut down on round trips that can severely impact how long it takes for a page to load. <A href="#" target="_blank"> Here's how it works: </A> TFO establishes a secure TFO cookie in the first connection using a standard 3-way handshake.&nbsp; Subsequent connections to the same server use the TFO cookie to connect without the 3-way handshake (zero RTT).&nbsp; This means TCP can carry data in the SYN and SYN-ACK. <BR /> <BR /> What we found together with <A href="#" target="_blank"> others in the industry </A> is that <A href="#" target="_blank"> middleboxes are interfering </A> with such traffic and <A href="#" target="_blank"> dropping connections </A> . Together with our large population of Windows enthusiasts (that's you!), we conducted experiments over the past few months, and tuned our algorithms to avoid usage of this option on networks where improper middlebox behavior is observed.&nbsp; Specifically, we enabled TFO in Edge using a checkbox in about:flags. <BR /> <BR /> To harden against such challenges, Windows automatically detects and disables TFO on connections that traverse through these problematic middleboxes. &nbsp;For our Windows Insider Program community, we enabled TFO in Edge (About:flags) by default for all insider flights in order to get a better understanding of middlebox interference issues as well as find more problems with anti-virus and firewall software.&nbsp; The data helped us improve our fallback algorithm which detects typical middlebox issues.&nbsp; We intend to continue our partnership with our Windows Insider Program (WIP) professionals to improve our fallback algorithm and identify unwanted anti-virus, firewall and middlebox behavior.&nbsp; Retail and non WIP releases will not participate in the experiments.&nbsp; If you operate infrastructure or software components such as middleboxes or packet processing engines that make use of a TCP state machine, please incorporate support for TFO.&nbsp; In the future, the combination of TLS 1.3 and TFO is expected to be more widespread. <BR /> The Creators Update also includes a fully functional server side implementation of TFO. The server side implementation also supports a pre-shared key for cases where a server farm is behind a load balancer. The shared key can be set by the following knob (requires elevation): <BR /> <BR /> reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v TcpFastopenKey /t REG_BINARY /f /d 0123456789abcdef0123456789abcdef <BR /> netsh int tcp reload <BR /> We encourage the community to test both client and server side functionality for interop with other operating system network stacks. The subsequent releases of Windows Server will include TFO functionality allowing deployment of IIS and other web servers which can take advantage of reduced connection setup times. <BR /> <BR /> <BR /> <H3> Experimental Support for the High Speed CUBIC Congestion Control Algorithm </H3> <BR /> <A href="#" target="_blank"> CUBIC </A> is a TCP Congestion Control (CC) algorithm featuring a cubic congestion window (Cwnd) growth function.&nbsp; The Cubic CC is a high-speed TCP variant and uses the amount of time since the last congestion event instead of ACK clocking to advance the Cwnd.&nbsp; In large BDP networks the <A href="#" target="_blank"> Cubic algorithm </A> takes advantage of throughput much faster than ACK clocked CC algorithms such as New Reno TCP.&nbsp; There have been reports that CUBIC can cause bufferbloat in networks with <A href="#" target="_blank"> unmanaged queues </A> (LTE and ADSL).&nbsp; In the Creators Update, we are introducing a Windows native implementation of CUBIC.&nbsp; We encourage the community to experiment with CUBIC and send us feedback. <BR /> The following commands can be used to enable CUBIC globally and to return to the default Compound TCP (requires elevation): <BR /> <BR /> netsh int tcp set supplemental template=internet congestionprovider=cubic <BR /> netsh int tcp set supplemental template=internet congestionprovider=compound <BR /> *** The Windows implementation of Cubic does not have the "Quiescence bug"&#157; that was recently <A href="#" target="_blank"> uncovered </A> in the Linux implementation. <BR /> <BR /> <BR /> <H3> Improved Receive Window Autotuning </H3> <BR /> TCP autotuning logic computes the "receive window"&#157; parameter of a TCP connection as described in <A href="#" target="_blank"> TCP autotuning logic </A> .&nbsp; High speed and/or long delay connections need this algorithm to achieve good performance characteristics.&nbsp; The takeaway from all this is that using the SO_RCVBUF socket option to specify a static value for the receive buffer is almost universally a <A href="#" target="_blank"> bad idea </A> .&nbsp; For those of you who choose to do so anyways please remember that calculating the correct size for TCP send/receive buffers is complex and requires information that applications do not have access to.&nbsp; It is far better to allow the Windows autotuning algorithm to size the buffer for you.&nbsp; We are working to identify such suboptimal usage of SO_RCVBUF/SO_SENDBUF socket options and to convince developers to move away from fixed window values.&nbsp; If you are an app developer and you are using either of these socket options please contact us. <BR /> <BR /> In parallel to our developer education effort we are improving the autotuning algorithm.&nbsp; Before the Creators Update the TCP receive Window autotuning algorithm depended on correct estimates of the connection's bandwidth and RTT.&nbsp; There are two problems with this method.&nbsp; First, the TCP RTT estimate is only measured on the sending side as described in <A href="#" target="_blank"> RFC 793 </A> .&nbsp; However, there are many examples of receive heavy workloads such as OS updates etc.&nbsp; The RTT estimate taken at the receive heavy side could be inaccurate.&nbsp; Second, there could be a feedback loop between altering the receive window (which can change the estimated bandwidth) and then measuring the bandwidth to determine how to alter the receive window. <BR /> <BR /> These two problems caused the receive window to constantly vary over time.&nbsp; We eliminated the unwanted behavior by modifying the algorithm to use a step function to converge on the maximum receive window value for a given connection.&nbsp; The step function algorithm results in a larger receive buffer size, however, the advertised receive window size is not backed by non-paged pool memory allocation and system resources are not used unless data is received and queued so the larger size is fine.&nbsp; Based on experimental results, the new algorithm adapts to the BDP much more quickly than the old algorithm.&nbsp; We encourage user and system administrators to also take note of our earlier post: <A href="#" target="_blank"> An Update on Windows TCP AutoTuningLevel </A> .&nbsp; This should clear misconceptions that autotuning and receive window scaling are bad for performance. <BR /> <H3> TCP stats API </H3> <BR /> The <A href="#" target="_blank"> Estats </A> API requires elevation and enumerates statistics for all connections.&nbsp; This can be inefficient especially on busy servers with lots of connections.&nbsp; In the Creators Update we are introducing a new API called SIO_TCP_INFO.&nbsp; &nbsp;SIO_TCP_INFO allows developers to query rich information on individual TCP connections using a socket option. The SIO_TCP_INFO API is versioned and we plan to add more statistics over time. &nbsp;In addition, we plan to add SIO_TCP_INFO &nbsp;to .Net NCL and HTTP APIs in subsequent releases. <BR /> The MSDN documentation for this API will be up soon and we will add a link here as soon as it is available. <BR /> <H2> IPv6 improvements </H2> <BR /> The Windows networking stack is dual stack and supports both IPv4 and IPv6 by default since Windows Vista.&nbsp; Over the Windows 10 releases, we are actively working on improving the support for IPv6. &nbsp;The following are some of the advancements in Creators Update. <BR /> <H3> RFC 6106 support </H3> <BR /> The Creators Update includes support for <A href="#" target="_blank"> RFC 6106 </A> which allows for DNS configuration through router advertisements (RAs). &nbsp;RDNSS and DNSSL ND options contained in router advertisements are validated and processed as described in the RFC. &nbsp;The implementation supports a max of 3 RDNSS and DNSSL entries each per interface. &nbsp;If there are more than 3 entries available from one or more routers on an interface, then entries with greater lifetime are preferred. &nbsp;In the presence of both DHCPv6 and RA DNS information, Windows gives precedence to DHCPv6 DNS information, in accordance with the RFC. <BR /> <BR /> In Windows, the lifetime processing of RA DNS entries deviates slightly from the RFC.&nbsp; In order to avoid implementing timers to expire DNS entries when their lifetime ends, we rely on the periodic Windows DNS service query interval (15 minutes) to remove expired entries, unless a new RA DNS message is received in which case the entry is updated immediately.&nbsp; This enhancement eliminates the complexity and overhead of kernel timers while keeping the DNS entries fresh.The following knob can be used to control this feature (requires elevation): <BR /> The following command can be used to control this feature (requires elevation): <BR /> netsh int ipv6 set interface &lt;ifindex&gt; rabaseddnsconfig=&lt;enabled | disabled&gt; <BR /> <BR /> <H3> Flow Labels </H3> <BR /> <BR /> Before the Creators update, the FlowLabel field in the IPv6 header was set to 0.&nbsp; Beginning with the Creators Update, outbound TCP and UDP packets over IPv6 have this field set to a <A href="#" target="_blank"> hash of the 5-tuple </A> (Src IP, Dst IP, Src Port, Dst Port). &nbsp;Middleboxes can use the FlowLabel field to perform ECMP for in-encapsulated native IPv6 traffic without having to parse the transport headers.&nbsp; This will make IPv6 only datacenters doing load balancing or flow classification more efficient. <BR /> <BR /> You can use this admin only knob to enable/disable IPv6 flow labels : <BR /> netsh int ipv6 set flowlabel=[disabled|enabled] (enabled by default) <BR /> <BR /> The following knob can be used to control this feature (requires elevation): <BR /> netsh int ipv6 set global flowlabel=&lt;enabled | disabled&gt; <BR /> <BR /> <BR /> <H3> ISATAP and 6to4 disabled by default </H3> <BR /> IPv6 continues to see uptake and IPv6 only networks are no longer a rarity. ISATAP and 6to4 are IPv6 transition technologies that have been enabled by default in Windows since Vista/Server 2008. As a step towards future deprecation, the Creators Update will have these technologies disabled by default. There are administrator and group policy knobs to re-enable them for specific enterprise deployments. An upgrade to the Creators Update will honor any administrator or group policy configured settings. By disabling these technologies, we aim to increase native IPv6 traffic on the Internet. Teredo is the last transition technology that is expected to be in active use because of its ability to perform NAT traversal to enable peer-to-peer communication. <BR /> <H3> Improved 464XLAT support </H3> <BR /> <A href="#" target="_blank"> 464XLAT </A> was originally designed for cellular only scenarios since mobile operators are some of the first ISPs with IPv6 only networks.&nbsp; However, some apps are not IP-agnostic and still require IPv4 support.&nbsp; Since a major use case for mobile is tethering, 464XLAT should provide IPv4 connectivity to tethered clients as well as to apps running on the mobile device itself. Creators Update adds support for 464XLAT on cellular-equipped desktops and tablets too. We also enabled support for TCP Large Send Offload (LSO) over 464XLAT improving throughput and reducing CPU usage. <BR /> <H2> Multi-homing improvements </H2> <BR /> Devices with multiple network interfaces are becoming ubiquitous. &nbsp;The trend is especially prevalent on mobile devices, but, 3G and LTE connectivity is becoming common on laptops, hybrids and many other form factors.&nbsp; For the Creators Update we collaborated with the Windows Connection Manager (WCM) team to make the WiFi to cellular handover faster and to improve performance when a mobile device is docked with wired Ethernet connectivity and then undocked causing a failover to WiFi. <BR /> <H3> Dead Gateway Detection (DGD) </H3> <BR /> Windows has always had a DGD algorithm that automatically transitions connections over to another gateway when the current gateway is unreachable, but, that algorithm was designed for server scenarios.&nbsp; For the Creators update we improved the DGD algorithm to respond to client scenarios such as switching back and forth between WiFi to 3G or LTE connectivity.&nbsp; DGD signals WCM whenever transport timeouts suggest that the gateway has gone dead.&nbsp; WCM uses this data to decide when to migrate connections over to the cellular interface.&nbsp; DGD also periodically re-probes the network so that WCM can migrate connections back to WiFi.&nbsp; This behavior only occurs if the user has opted in for automatic failover to cellular. <BR /> <H3> Fast connection teardown </H3> <BR /> In Windows, TCP connections are preserved for about 20 seconds to allow for fast reconnection in the case of a temporary loss of wired or wireless connectivity.&nbsp; However, in the case of a true disconnection such as docking and undocking this is an unacceptably long delay.&nbsp; Using the Fast Connection Teardown feature WCM can signal the Windows transport layer to instantly tear down TCP connections for a fast transition. <BR /> <H3> Improved diagnostics using Test-NetConnection </H3> <BR /> <A href="#" target="_blank"> Test-NetConnection </A> (alias tnc) is a built-in cmdlet in powershell that performs a variety of network diagnostics.&nbsp; In Creators Update we have enhanced this cmdlet to provide detailed information about both route selection as well as source address selection. <BR /> The following command when run elevated will describe the steps to select a particular route per RFC 6724. This can be particularly useful in multi-homed systems or when there are multiple IP addresses on the system. <BR /> <BR /> Test-NetConnection -ComputerName "<A href="#" target="_blank">www.contoso.com</A>" -ConstrainInterface 5 -DiagnoseRouting -InformationLevel "Detailed" </BODY></HTML> Thu, 14 Feb 2019 17:58:12 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/core-network-stack-features-in-the-creators-update-for-windows/ba-p/339676 Daniel Havey 2019-02-14T17:58:12Z HTTPS Client Certificate Request freezes when the Server is handling a large PUT/POST Request https://gorovian.000webhostapp.com/?exam=t5/networking-blog/https-client-certificate-request-freezes-when-the-server-is/ba-p/339672 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Jul 12, 2017 </STRONG> <BR /> <H2> HTTPS Client Certificate Request freezes when the Server is handling a large PUT/POST Request </H2> <BR /> <BR /> <BR /> There is a class of problems that may occur when using client-side certificates in HTTPS. <BR /> <BR /> Sometimes, the server’s request for a client certificate will <EM> freeze </EM> (until the timeout of two minutes or so) when processing PUT/POST request with a large payload (e.g., &gt;40KB). <BR /> <BR /> Ideally, the server should request the client certificate before any large request exchange. <BR /> <BR /> Otherwise, the server should request the client certificate immediately after either: <BR /> <UL> <BR /> <LI> a request has been completely received, or </LI> <BR /> <LI> a request has been responded to. </LI> <BR /> </UL> <BR /> Otherwise, the large payload fills the network buffers, which cannot be emptied until the certificate is received and everything processed. This leads to deadlock if the server issues a synchronous call for the client certificate. Although it is not illegal, this is what causes the problem. Furthermore, this represents a trivial DoS vector against any such server. <BR /> <BR /> This may depend on the component sitting directly above http.sys. IIS for example, tries to read as much entity body as possible before requesting the client certificate. <BR /> <BR /> These are some alternatives to fix this issue, but only the first one listed below is deterministic. <BR /> <H2> By Modifying Only the Server side (when the client cannot be modified): </H2> <BR /> <OL> <BR /> <LI> ( <STRONG> recommended </STRONG> ) Set “client certificate required” on the SSL binding so that client certificate is requested at SSL/TLS connection time, before any HTTP request exchange. This forces client certificate to be requested for every connection on that binding. Depending on your configuration, you might need a dedicated VIP and/or SSL SNI name for this communication. This requires no server code changes, but a configuration change via “netsh http” on the SSL binding: <I> </I> <EM> clientcertnegotiation=enable </EM> </LI> <BR /> </OL> <BR /> <STRONG> Note </STRONG> : If the server is IIS-based the change needs to be done through IIS. Otherwise, since IIS has a different config, it may overwrite any changes made directly to Http.sys. <BR /> <OL> <BR /> <LI> If the server sees this is a PUT/POST request, you need to ensure that the server’s TCP buffers have enough space for the client certificate when it arrives. This leads to strategies such as <BR /> <OL> <BR /> <LI> reading as much entity body as possible requesting for the client certificate using an asynchronous call, or, </LI> <BR /> <LI> Modify your web server app so that it asynchronously pulls the request body while it waits for client certificate retrieval to finish.&nbsp; If too much entity body is pulled (e.g., several MB) and client certificate retrieval has still not finished then cancel the request/connection. Requires server code changes. </LI> <BR /> <LI> even better, issuing the asynchronous call for the client certificate as early as possible and draining as much of the entity body as possible as you wait for the client certificate to arrive. </LI> <BR /> </OL> <BR /> </LI> <BR /> </OL> <BR /> This requires server code changes for sure. To increase the chances of this working, *all* relevant buffers on the server as well as the client and in between, need to have enough space for the client certificate to not be stuck behind large payloads. So modifying the client to drain buffers (in addition to the server) helps, but is not sufficient, as intermediate buffers along the way may also pose a problem (e.g., bufferbloat). This is not a deterministic method. <BR /> <BR /> <BR /> <H2> By Modifying the Client side in addition to the Server side: </H2> <BR /> <OL> <BR /> <LI> ( <STRONG> recommended </STRONG> ) Use requests such as GET or HEAD to prime the connection so that the server can request for the certificate without being blocked to receive the entity body. This also implies an extra round trip for the priming request, but if client certificates are involved the application is already making some latency tradeoffs. This is not deterministic, as the immediately following “real” request may use a different connection, but usually reuses the “primed” connection from the connection pool. This will require client-side changes and requires that the server expose such an endpoint, as well. </LI> <BR /> <LI> Use the status code 100 Continue (requires client to send “ <EM> Expect: </EM> <EM> 100-continue </EM> ” header). This may require both client and server changes to be supported. Furthermore, it is not a deterministic mechanism. </LI> <BR /> </OL> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> <BR /> </BODY></HTML> Thu, 14 Feb 2019 17:58:03 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/https-client-certificate-request-freezes-when-the-server-is/ba-p/339672 Gabriel Montenegro 2019-02-14T17:58:03Z Troubleshooting certificate issues in Software Defined Networking (SDN) https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshooting-certificate-issues-in-software-defined/ba-p/339671 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on May 19, 2017 </STRONG> <BR /> As you may be aware, Network Controller in Windows Server 2016 uses certificate based authentication for communicating with Hyper-V hosts and Software Load Balancer MUX virtual machines (VMs). <BR /> <BR /> Some SDN customers have complained about communication issues between Network Controller and hosts, although certificates were correctly configured on both the entities. <BR /> <BR /> On debugging, we found that the customer had installed a non self-signed certificate into the computer's <STRONG> Trusted Root Certification Authorities </STRONG> store. Although this certificate was not involved in communication between Network Controller and the hosts, the presence of such a certificate broke client authentication. Here is a view of some of the certificate properties: <BR /> <BR /> <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75201i46B5105B594B1BB9" /> <BR /> <BR /> The following Knowledge Base article provides information about this issue: <A href="#" target="_blank"> Internet Information Services (IIS) 8 may reject client certificate requests with HTTP 403.7 or 403.16 errors </A> <BR /> <BR /> To resolve this issue, you can uninstall the non self-signed certificate from the <STRONG> Trusted Root Certification Authorities </STRONG> certificate store for the Local Computer, or move the certificate to the <STRONG> Intermediate Certification Authorities </STRONG> store. <BR /> <BR /> One more thing to note is that that the <STRONG> Personal </STRONG> (My - cert:\localmachine\my) certificate store on the Hyper-V host must have exactly one X.509 certificate with Subject Name (CN) as the host FQDN. This certificate is used for communication with the Network Controller. <BR /> <BR /> This behavior is due to a bug in the system and will be fixed shortly. For now, please ensure that you have only one certificate with the Subject Name (CN) as the host FQDN. <BR /> <BR /> For more information, see the following topics in the Windows Server 2016 Technical Library. <BR /> <UL> <BR /> <LI> <A href="#" target="_blank"> Network Controller Security </A> </LI> <BR /> <LI> <A href="#" target="_blank"> Managing certificates for Software Defined Networking </A> </LI> <BR /> </UL> <BR /> Anirban Paul, Senior Program Manager </BODY></HTML> Thu, 14 Feb 2019 17:58:01 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshooting-certificate-issues-in-software-defined/ba-p/339671 AnirbanPaul 2019-02-14T17:58:01Z Windows network performance suffering from bad buffering https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-network-performance-suffering-from-bad-buffering/ba-p/339668 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on May 08, 2017 </STRONG> <BR /> Daniel Havey, Praveen Balasubramanian <BR /> <BR /> Windows telemetry results have indicated that a significant number of data connections are using the SO_RCVBUF and/or the SO_SNDBUF winsock options to statically allocate TCP buffers. There are many websites that recommend setting the TCP buffers with these options in order to improve TCP performance. This is a myth. Using Winsock options (SO_RCVBUF and/or SO_SNDBUF) to statically allocate TCP buffers will not make Windows networking stack “faster”. In fact, static allocation of the TCP buffers will degrade performance in terms of how fast the connection responds (latency) and how much data it delivers (bandwidth). The Windows transports team officially recommends <STRONG> not </STRONG> doing this. <BR /> <BR /> TCP buffers need to be dynamically allocated in proportion to the Bandwidth Delay Product (BDP) of the TCP connection. There are two good reasons why we should let the Windows networking stack dynamically set the TCP buffers for us and not set them statically at the application layer. 1.) The application does not know the BDP (TCP does) so it cannot properly set the TCP buffers and 2.) Dynamic buffer management requires complex algorithmic control which TCP already has. In summary, Windows 10 has autotuning for TCP. Let the autotuning algorithm manage the TCP buffers. <BR /> <BR /> <STRONG> Example: </STRONG> I am going to use the Cygwin application as an example since they recently fixed their buffering (thank you Corinna). The experiment is conducted across the Internet to an iperf server in France (from my desk in Redmond). <BR /> <H3> <STRONG> Experiment 1 -- Cygwin (Bad buffering): </STRONG> </H3> <BR /> Pinging 178.250.209.22 with 32 bytes of data: <BR /> Reply from 178.250.209.22: bytes=32 time=176ms TTL=35 <BR /> Reply from 178.250.209.22: bytes=32 time=173ms TTL=35 <BR /> Reply from 178.250.209.22: bytes=32 time=173ms TTL=35 <BR /> Reply from 178.250.209.22: bytes=32 time=172ms TTL=35 <BR /> Ping statistics for 178.250.209.22: <BR /> Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), <BR /> Approximate round trip times in milli-seconds: <BR /> Minimum = 172ms, Maximum = 176ms, Average = 173ms <BR /> ------------------------------------------------------------ <BR /> Client connecting to 178.250.209.22, TCP port 5001 <BR /> TCP window size: 208 KByte (default) <BR /> ------------------------------------------------------------ <BR /> [ 3] local 10.137.196.108 port 56758 connected with 178.250.209.22 port 5001 <BR /> [ ID] Interval Transfer Bandwidth <BR /> [ 3] 0.0- 1.0 sec 512 KBytes 4.19 Mbits/sec <BR /> [ 3] 1.0- 2.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 2.0- 3.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 3.0- 4.0 sec 1.25 MBytes 10.5 Mbits/sec <BR /> [ 3] 4.0- 5.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 5.0- 6.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 6.0- 7.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 7.0- 8.0 sec 1.25 MBytes 10.5 Mbits/sec <BR /> [ 3] 8.0- 9.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 9.0-10.0 sec 1.50 MBytes 12.6 Mbits/sec <BR /> [ 3] 0.0-10.1 sec 13.6 MBytes 11.3 Mbits/sec <BR /> <BR /> <BR /> We can see that the RTT is the same for both Experiment 1 &amp; 2 about 177ms. However, in Experiment 1 Cygwin has bad buffering and the throughput averages 11.3 Mbps and tops out at 12.6 Mbps. This is because in Experiment 1 Cygwin was using SO_RCVBUF to allocate 278,775 bytes for the TCP receive buffer and the throughput is buffer limited to 12.6 Mbps. <BR /> <H3> <STRONG> Experiment 2 -- Cygwin (Good buffering): </STRONG> </H3> <BR /> Pinging 178.250.209.22 with 32 bytes of data: <BR /> Reply from 178.250.209.22: bytes=32 time=172ms TTL=35 <BR /> Reply from 178.250.209.22: bytes=32 time=172ms TTL=35 <BR /> Reply from 178.250.209.22: bytes=32 time=172ms TTL=35 <BR /> Reply from 178.250.209.22: bytes=32 time=173ms TTL=35 <BR /> Ping statistics for 178.250.209.22: <BR /> Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), <BR /> Approximate round trip times in milli-seconds: <BR /> Minimum = 172ms, Maximum = 173ms, Average = 172ms <BR /> ------------------------------------------------------------ <BR /> Client connecting to 178.250.209.22, TCP port 5001 <BR /> TCP window size: 64.0 KByte (default) <BR /> ------------------------------------------------------------ <BR /> [&nbsp; 3] local 10.137.196.108 port 56898 connected with 178.250.209.22 port 5001 <BR /> [ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bandwidth <BR /> [&nbsp; 3]&nbsp; 0.0- 1.0 sec&nbsp;&nbsp; 768 KBytes&nbsp; 6.29 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 1.0- 2.0 sec&nbsp; 11.8 MBytes&nbsp; 98.6 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 2.0- 3.0 sec&nbsp; 18.0 MBytes&nbsp;&nbsp; 151 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 3.0- 4.0 sec&nbsp; 16.6 MBytes&nbsp;&nbsp; 139 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 4.0- 5.0 sec&nbsp; 16.4 MBytes&nbsp;&nbsp; 137 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 5.0- 6.0 sec&nbsp; 18.0 MBytes&nbsp;&nbsp; 151 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 6.0- 7.0 sec&nbsp; 18.0 MBytes&nbsp;&nbsp; 151 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 7.0- 8.0 sec&nbsp; 18.0 MBytes&nbsp;&nbsp; 151 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 8.0- 9.0 sec&nbsp; 15.6 MBytes&nbsp;&nbsp; 131 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 9.0-10.0 sec&nbsp; 17.4 MBytes&nbsp;&nbsp; 146 Mbits/sec <BR /> [&nbsp; 3]&nbsp; 0.0-10.0 sec&nbsp;&nbsp; 151 MBytes&nbsp;&nbsp; 126 Mbits/sec <BR /> <BR /> <BR /> In Experiment 2 we see Cygwin perform without static application level buffering.&nbsp; The average throughput is 126 Mbps and the maximum is 151 Mbps which is the true unloaded line speed of this connection.&nbsp; By statically allocating the receive buffer using SO_RCVBUF we limited ourselves to a top speed of 12.6 Mbps.&nbsp; By letting Windows TCP autotuning dynamically allocate the buffers we achieved the true unloaded line rate of 151 Mbps.&nbsp; That is about an order of magnitude better performance.&nbsp; Static allocation of TCP buffers at the app level is a bad idea. <STRONG> Don’t do it </STRONG> . <BR /> <BR /> Sometimes there are corner cases where as a developer one might think that there is justifiable cause to statically allocate the TCP buffers.&nbsp; Let’s take a look at three of the most common causes for thinking this: <BR /> <BR /> 1.)&nbsp;Setting the buffers for performance sake. <STRONG> Don’t. </STRONG> TCP autotuning is a kernel level algorithm and can do a better job than any application layer algorithm. <BR /> 2.)&nbsp;Setting the buffers because you are trying to rate limit traffic. <STRONG> Be Careful! </STRONG> The results may not be what you expect.&nbsp; In the Cygwin example the connection is buffer limited to 12.6 Mbps maximum.&nbsp; However, if the RTT were to change to about 40 ms then the connection would be limited to about 50 Mbps.&nbsp; You cannot reliably set a bandwidth cap in this manner (See the BDP equations). <BR /> 3.)&nbsp;Setting the buffers for some other reason.&nbsp; Let’s have a discussion.&nbsp; Please comment on the post and we will respond. <BR /> <BR /> </BODY></HTML> Thu, 14 Feb 2019 17:57:51 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-network-performance-suffering-from-bad-buffering/ba-p/339668 Daniel Havey 2019-02-14T17:57:51Z Windows Networking for Kubernetes https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-networking-for-kubernetes/ba-p/339667 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Apr 04, 2017 </STRONG> <BR /> A seismic shift is happening in the way applications are developed and deployed as we move from traditional three-tier software models running in VMs to "containerized" applications and micro-services deployed across a cluster of compute resources. Networking is a critical component in any distributed system and often requires higher-level orchestration and policy management systems to control IP address management (IPAM), routing, load-balancing, network security, and other advanced network policies. The Windows networking team is swiftly adding new features ( <A href="#" target="_blank"> Overlay networking and Docker Swarm Mode on Windows 10 </A> ) and working with the larger containers community (e.g. <A href="#" target="_blank"> Kubernetes sig-windows group </A> ) by contributing to open source code and ensuring native networking support for any orchestrator, in any deployment environment, with any network topology. <BR /> <BR /> Today, I will be discussing how Kubernetes networking is implemented in Windows and managed by an extensible Host Networking Service (HNS) - which is used in both Azure Container Service (ACS) Windows worker nodes and on-premises deployments - to plumb network policy in the OS . <BR /> <BR /> <EM> Note: A video recording of the 4/4 #sig-windows meetup where I describe this is posted here: </EM> <A href="#" target="_blank"> <EM> https://www.youtube.com/watch?v=P-D8x2DndIA&amp;t=6s&amp;list=PL69nYSiGNLP2OH9InCcNkWNu2bl-gmIU4&amp;index=1 </EM> </A> <BR /> <H2> Kubernetes Networking </H2> <BR /> Windows containers can be orchestrated using either <A href="#" target="_blank"> Docker Swarm </A> or <A href="#" target="_blank"> Kubernetes </A> <A href="#" target="_blank"> </A> <A href="#" target="_blank"> </A> to help "automate the deployment, scaling, and management of 'containerized' applications". However, the networking model used by these two systems is different. <BR /> <BR /> Kubernetes networking is built on the fundamental requirements listed <A href="#" target="_blank"> here </A> and is either agnostic to the network fabric underneath or assumes a flat Layer-2 networking space where all containers and nodes can communicate with all other containers and nodes across a cluster <STRONG> without using NAT (encapsulation is permitted) </STRONG> . Windows can support these requirements using a few different networking modes exposed by HNS and working with external IPAM drivers and route configurations. <BR /> <BR /> The other large difference between Docker and Kubernetes networking is the scope at which IP assignment and resource allocation occurs. Docker assigns an IP address to every container whereas Kubernetes assigns IP addresses to a <A href="#" target="_blank"> Pod </A> which represents a network namespace and could consist of multiple containers running inside the Pod. Windows also has a network namespace concept called a <STRONG> <EM> network compartment </EM> </STRONG> and a management surface is being built in Windows to allow for multiple containers in a Pod to communicate with each other through localhost. <BR /> <BR /> Connectivity between pods located on different nodes in a Kubernetes cluster can be accomplished either by using an overlay (e.g. vxlan) network or without an overlay by configuring routes and IPAM on the underlying (virtual) network fabric. Realizing this network model can be done through: <BR /> <UL> <BR /> <LI> CNI Network Plugin </LI> <BR /> <LI> Implementing the "Routing" interface in Kubernetes code </LI> <BR /> <LI> External configuration </LI> <BR /> </UL> <BR /> The sig-windows community (led by <A href="#" target="_blank"> Apprenda </A> ) did a lot of work to come up with an initial solution for getting Kubernetes networking to work on Windows. The networking teams at Microsoft are building on this work and continues to partner with the community to add support for the native Kubernetes networking model - <EM> defined by the </EM> <A href="#" target="_blank"> <EM> Container Network Interface (CNI) </EM> </A> <EM> which, is different from the <A href="#" target="_blank"> Cloud Network Model (CNM) </A> </EM> <A href="#" target="_blank"> </A> <EM> used by Docker </EM> - and surfacing policy management capabilities through HNS. <BR /> <H2> Kubernetes networking in Azure Container Service (ACS) </H2> <BR /> Azure Container Service recently announced <A href="#" target="_blank"> Kubernetes general availability </A> which uses a routable-vip approach (no overlay) to networking and configures <A href="#" target="_blank"> User-Defined Routes (UDR) </A> in the Hyper-V (virtualization) host for Pod communication between Linux and Windows cluster node VMs. A /24 CIDR IP pool (routable between container host VMs) is allocated to each container host with one IP assigned per Pod (one container). <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75199iC5BFA86A10D60B67" /> <BR /> <BR /> With the recent Azure VNet for Containers announcement which includes support for a CNI network plugin used in Azure (pre-released here: <A href="#" target="_blank"> https://github.com/Azure/azure-container-networking/releases/tag/v0.7 </A> ), tenants can connect their ACS clusters (containers and hosts) directly to Azure Virtual Networks. This means that individual IPs from a tenant's Azure VNet IP space will be assigned to Kubernetes nodes and pods in potentially the same subnet. The Windows networking team is also working to build a CNI plugin to support and extend container management through Kubernetes on Windows for on-premises deployments. <BR /> <H2> Kubernetes networking in Windows </H2> <BR /> Microsoft engineers across Windows and Azure product groups actively contributed code to the Kubernetes repo to enhance kube-proxy (used for DNS and service load-balancing) and kubelet (for Internet access) binaries which are installed on ACS Kubernetes Windows worker nodes. This overcame gaps <A href="#" target="_blank"> previously identified </A> so that both DNS and service load-balancing worked correctly without the need for Routing and Remote Access Services (RRAS) or netsh port proxy. In this implementation, the Windows network uses Kubernetes' default kubenet plugin without CNI plugin. <BR /> <BR /> Using HNS, one <EM> transparent </EM> and one <EM> NAT </EM> network is created on each Windows container host for inter-Pod and external communication respectively. Two container endpoints - connected to the Service and Pod networks - are required for each Windows container which will participate in the Kubernetes service. Static routes must be added to the running Windows containers themselves on the container endpoint attached to the service network. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75200i227DA041087EC074" /> <BR /> <BR /> In the absence of ACS-managed User-Defined Routes, Out-of-Band (OOB) configuration of these routes need to be realized in the Cloud Service Provider network, implemented using the "routing" interface of the Kubernetes cloud provider, or connected via overlay networks. Other solutions include using the HNS overlay network driver for inter-Pod communication or using the OVS Hyper-V switch extension with OVN Controller. <BR /> <BR /> Today, with the publicly available versions of Windows server and client you can deploy Kubernetes with the following restrictions: <BR /> <UL> <BR /> <LI> One container per Pod </LI> <BR /> <LI> CNI Network Plugins are not supported </LI> <BR /> <LI> Each container requires two container endpoints (vNICs) with IP routing manually plumbed </LI> <BR /> <LI> Service IPs can only be associated with one Container Host and will not be load-balanced </LI> <BR /> <LI> Policy specifications (e.g. network security) are not supported </LI> <BR /> </UL> <BR /> <H2> What's Coming Next? </H2> <BR /> Windows is moving to a faster release cadence such that new platform features will be made available in a matter of months rather than years. In some circumstances, early builds can be made available to Insiders as well as to TAP customers and EEAP partners for early feature validation. <BR /> <BR /> Stay tuned for new features which will be made available soon... <BR /> <H2> Summary </H2> <BR /> In this blog post, I described some of the nuances of the Kubernetes networking model and how it differs from the Docker networking model. I also talked about the code updates made by Microsoft engineering teams to the kubelet and kube-proxy binaries for Windows in open source repos to enable networking support. Finally, we ended with how Kubernetes networking is implemented in Windows today and the plans for how it will be implemented through a CNI plugin in the near future... </BODY></HTML> Thu, 14 Feb 2019 17:57:47 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/windows-networking-for-kubernetes/ba-p/339667 Jason Messer 2019-02-14T17:57:47Z How to find the SDN gateway local address for BGP peering in Windows Server 2016 https://gorovian.000webhostapp.com/?exam=t5/networking-blog/how-to-find-the-sdn-gateway-local-address-for-bgp-peering-in/ba-p/339663 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Mar 23, 2017 </STRONG> <BR /> A few days back, I&nbsp;wrote a blog post about some issues being faced by Software Defined Networking (SDN) customers. The issue was specific to changing VPN bandwidth settings in Windows Server 2016. You can read more about that issue and the solution <A href="#" target="_blank"> here </A> . <BR /> <BR /> Another area where we have seen customers struggle is finding out the local SDN gateway server address. The local SDN gateway Server address is required for the following reasons: <BR /> <OL> <BR /> <LI> When you configure the remote VPN endpoint (in your enterprise or your local datacenter), you need to provide the local SDN gateway server address as the destination IP. This is the IP address advertised by the gateway for external connectivity </LI> <BR /> <LI> If you are using BGP for learning dynamic routes over VPN, you will need the local SDN gateway server address to configure the BGP peering information. Note that this address will be different from the destination IP I have mentioned above, since this is the IP address of the internal interface of the VPN server. </LI> <BR /> </OL> <BR /> <H2> Finding the external address of SDN gateway </H2> <BR /> This address will be used as the destination IP address when you configure the on-premise VPN server (or a GRE endpoint in the same datacenter). This address may be different for different tenants&nbsp;because the SDN gateway is a multi-tenant server. <BR /> <BR /> This address is displayed in the&nbsp;System Center Virtual Machine Manager (SCVMM) console&nbsp;when you configure the connection, as depicted in the illustration below. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75198i4E7870383ECB5B45" /> <BR /> <BR /> <BR /> <H2> Finding the BGP router IP address of the SDN gateway </H2> <BR /> <H3> BGP Router IP for tenant connections </H3> <BR /> If you are using&nbsp;Border Gateway Protocol (BGP)&nbsp;with your tenant IPsec, GRE or L3 connections for dynamically learning remote routes, you&nbsp;need to know the BGP router IP address so that you can configure that address as the peer address on the remote router. When you configure the VPN connections through SCVMM,&nbsp;SCVMM automatically assigns an IP Address from the gateway routing subnet to the tenant compartment of the gateway VM.&nbsp;SCVMM uses this IP address as the BGP router IP address.&nbsp;Because this router is&nbsp;tenant-specific, the router address&nbsp;is different for each tenant. <BR /> <BR /> First, execute the following Windows Powershell commands on a Network Controller machine or a machine that is configured as a Network Controller client: <BR /> <BR /> <EM> $gateway = Get-NetworkControllerVirtualGateway -ConnectionUri &lt;REST uri of your deployment&gt; </EM> <BR /> <BR /> <EM> $gateway.Properties.NetworkConnections.Properties.DestinationAddress </EM> <BR /> <BR /> The results of this command can display multiple virtual gateways, depending on how many tenants have configured gateway connections.&nbsp;Also, each virtual gateway can have multiple connections (IPSec, GRE, L3).&nbsp;Because you already know the destination address of the connection, you can identify the correct connection based on the destination address.&nbsp;After you have the correct network connection,&nbsp;run the following command (on the corresponding virtual gateway) to get the BGP router IP address of the virtual gateway <BR /> <BR /> <EM> $gateway.Properties.BgpRouters.Properties.RouterIp </EM> <BR /> <BR /> The result of this command provides the IP address that you must configure on the remote router as the peer IP Address. <BR /> <H3> BGP router IP for GRE gateway </H3> <BR /> If you are using GRE connectivity in your SDN deployment, you must create a GRE VIP logical network and advertise the GRE VIPs from your SDN gateways to the physical network using internal BGP peering. You can get more details in the SDN planning document <A href="#" target="_blank"> here </A> . <BR /> <BR /> You need to create a BGP peer on the Top of Rack router (ToR) that is used by your SDN infrastructure to receive routes for the GRE VIP logical network advertised by the SDN Gateways. BGP peering only needs to occur one way (from SDN Gateway to external BGP peer). To configure the BGP peer, you will need to provide the peer IP i.e, the BGP router IP of the SDN gateways. <BR /> <BR /> To get the BGP router IP of the SDN gateway, execute the following Powershell commands on a Network Controller machine or a machine that is configured as a Network Controller client: <BR /> <BR /> <EM> $gateway = Get-NetworkControllerGateway -ConnectionUri &lt;REST uri of your deployment&gt; </EM> <BR /> <BR /> <EM> $gateway.Properties.BgpConfig.RouterIp </EM> <BR /> <BR /> The result of this command provides the&nbsp;IP address that you must&nbsp;configure on the remote router as the peer IP Address. <BR /> <BR /> <BR /> <BR /> If you want to setup SDN through SCVMM, there is a bunch of detailed documentation on Technet <A href="#" target="_blank"> here </A> . Before starting the deployment, please go through the SDN planning guidance <A href="#" target="_blank"> here </A> . </BODY></HTML> Thu, 14 Feb 2019 17:57:28 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/how-to-find-the-sdn-gateway-local-address-for-bgp-peering-in/ba-p/339663 AnirbanPaul 2019-02-14T17:57:28Z Troubleshoot Configuring SDN RAS Gateway VPN Bandwidth Settings in Virtual Machine Manager https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshoot-configuring-sdn-ras-gateway-vpn-bandwidth-settings/ba-p/339661 <HTML> <HEAD></HEAD><BODY> <STRONG> First published on TECHNET on Mar 06, 2017 </STRONG> <BR /> I wanted to share some of my experiences with debugging Windows Server 2016 Software Defined Networking (SDN) related customer issues. These issues&nbsp;are related to SDN RAS Gateways.If you’ve deployed Software Defined Networking (SDN) in Windows Server 2016 Datacenter by using System Center Virtual Machine Manager (SCVMM), you might have encountered problems configuring the RAS Gateway virtual private network (VPN) connection inbound and outbound bandwidth settings. <BR /> <BR /> Gateways are used in SDN to provide external connectivity to a virtual network. This can be connectivity to an on-premises network or to the physical network in the same datacenter. You can get more information about gateways&nbsp;in the topic <A href="#" target="_blank"> RAS Gateway for SDN </A> . <BR /> <BR /> <STRONG> Issue #1 </STRONG> <BR /> <BR /> The customer was unable to change VPN connection inbound and outbound bandwidth settings by using the SCVMM user interface (UI) setting <STRONG> Maximum Incoming </STRONG> and <STRONG> Maximum Outgoing </STRONG> . <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75195i2532814E0C03EE08" /> <BR /> <BR /> When the customer tried to change these gateway bandwidth settings from the SCVMM UI, he received the error ID 26909, <STRONG> Network service ‘SA19N30NC’ doesn’t support this type of traffic metering, </STRONG> as depicted in the following screen shot. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75196i99332FEB3DC8D166" /> <BR /> <BR /> <STRONG> Solution for Issue #1 </STRONG> <BR /> <BR /> SCVMM currently does not support changing bandwidth settings for a VPN connection. They will start supporting this shortly. By default, the inbound and outbound bandwidth is set as 500 Kbps. <BR /> <BR /> Meanwhile, if you want to change bandwidth settings, you can use the Network Controller Windows PowerShell command <A href="#" target="_blank"> New-NetworkControllerVirtualGatewayNetworkConnection </A> with the parameters <STRONG> OutboundKiloBitsPerSecond </STRONG> and <STRONG> InboundKiloBitsPerSecond </STRONG> . <BR /> <BR /> <EM> NOTE: If you make any other changes to the VPN connection through SCVMM after changing the bandwidth settings, the bandwidth settings will be reset to the default (500 Kbps). So, you will need to run the Network Controller Powershell again to update the bandwidth settings. </EM> <BR /> <BR /> <STRONG> Issue #2 </STRONG> <BR /> <BR /> Even after changing the VPN network connection bandwidth settings to 200 Mbps by using&nbsp;Network Controller Windows PowerShell commands, the customer observed a bandwidth cap of about 150 Mbps for the connection. <BR /> <BR /> <STRONG> Solution for Issue #2 </STRONG> <BR /> <BR /> The customer had set the gateway capacity as 1000 Mbps (this is the default value in the SCVMM UI). The <STRONG> Gateway capacity (Mbps) </STRONG> parameter denotes the normal TCP bandwidth which is expected out of the gateway VM. Customer should fill this accordingly based on his underlying network speed. <BR /> <BR /> <IMG src="https://techcommunity.microsoft.com/t5/image/serverpage/image-id/75197i2A27E19C3D77FBBD" /> <BR /> <BR /> Maximum IPsec tunnel bandwidth is limited to (3/20)* Gateway Capacity on a particular gateway. So, if the gateway capacity is set to 1000 Mbps, the equivalent IPsec tunnel capacity would be 150 Mbps. <BR /> <BR /> The equivalent ratios for GRE and L3 tunnels are 1/5 and 1/2, respectively. <BR /> <BR /> NOTE: You must be wondering why the customer was allowed to add a connection with 200 Mbps bandwidth if the gateway did not have available capacity. Actually, the configuration change never succeeded. This configuration change is an asynchronous operation. After changing the settings, if the customer had executed <STRONG> Get-NetworkControllerVirtualGatewayNetworkConnection </STRONG> and checked the <STRONG> ConfigurationState </STRONG> of the resource, the “Status” would have been “Failure” with “DetailedInfo” giving more details about the error. <BR /> <BR /> If you want to setup SDN through SCVMM, see the topic <A href="#" target="_blank"> Set up a Software Defined Network (SDN) infrastructure in the VMM fabric </A> . Before starting the setup,&nbsp;you can review the&nbsp;SDN planning guidance&nbsp;in the topic <A href="#" target="_blank"> Plan a Software Defined Network Infrastructure </A> . <BR /> <BR /> Anirban Paul, Senior Program Manager </BODY></HTML> Thu, 14 Feb 2019 17:57:17 GMT https://gorovian.000webhostapp.com/?exam=t5/networking-blog/troubleshoot-configuring-sdn-ras-gateway-vpn-bandwidth-settings/ba-p/339661 AnirbanPaul 2019-02-14T17:57:17Z