Custom containers in Azure Machine Learning managed online endpoints

Published Jun 17 2021 01:48 PM 1,657 Views
Microsoft

Today, we are announcing the public preview of the ability to use custom Docker containers in Azure Machine Learning online endpoints. In combination with our new 2.0 CLI, this feature enables you to deploy a custom Docker container while getting Azure Machine Learning online endpoints’ built-in monitoring, scaling, and alerting capabilities.

 

Below, we walk you through how to use this feature to deploy TensorFlow Serving with Azure Machine Learning. The full code is available in our samples repository.

 

Sample deployment with TensorFlow Serving

 

To deploy a TensorFlow model with TensorFlow Serving, first create a YAML file:

 

name: tfserving-endpoint
type: online
auth_mode: aml_token
traffic:
  tfserving: 100

deployments:
  - name: tfserving
    model:
      name: tfserving-mounted
      version: 1
      local_path: ./half_plus_two
    environment_variables:
      MODEL_BASE_PATH: /var/azureml-app/azureml-models/tfserving-mounted/1
      MODEL_NAME: half_plus_two
    environment:
      name: tfserving
      version: 1
      docker:
        image: docker.io/tensorflow/serving:latest
      inference_config:
        liveness_route:
          port: 8501
          path: /v1/models/half_plus_two
        readiness_route:
          port: 8501
          path: /v1/models/half_plus_two
        scoring_route:
          port: 8501
          path: /v1/models/half_plus_two:predict
    instance_type: Standard_F2s_v2
    scale_settings:
      scale_type: manual
      instance_count: 1
      min_instances: 1
      max_instances: 2

 

Then create your endpoint:

 

az ml endpoint create -f endpoint.yml

 

And that’s it! You now have a scalable TensorFlow Serving endpoint running on Azure ML-managed compute.

Next steps

%3CLINGO-SUB%20id%3D%22lingo-sub-2460558%22%20slang%3D%22en-US%22%3ECustom%20containers%20in%20Azure%20Machine%20Learning%20managed%20online%20endpoints%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2460558%22%20slang%3D%22en-US%22%3E%3CP%3EToday%2C%20we%20are%20announcing%20the%20public%20preview%20of%20the%20ability%20to%20use%20custom%20Docker%20containers%20in%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-ai%2Fannouncing-managed-endpoints-in-azure-machine-learning-for%2Fba-p%2F2366481%22%20target%3D%22_self%22%3EAzure%20Machine%20Learning%20online%20endpoints%3C%2FA%3E.%20In%20combination%20with%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-ai%2Fannouncing-the-new-cli-and-arm-rest-apis-for-azure-machine%2Fba-p%2F2393447%22%20target%3D%22_blank%22%3Eour%20new%202.0%20CLI%3C%2FA%3E%2C%20this%20feature%20enables%20you%20to%20deploy%20a%20custom%20Docker%20container%20while%20getting%20Azure%20Machine%20Learning%20online%20endpoints%E2%80%99%20built-in%20monitoring%2C%20scaling%2C%20and%20alerting%20capabilities.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBelow%2C%20we%20walk%20you%20through%20how%20to%20use%20this%20feature%20to%20deploy%20%3CA%20href%3D%22https%3A%2F%2Fwww.tensorflow.org%2Ftfx%2Fguide%2Fserving%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3ETensorFlow%20Serving%3C%2FA%3E%20with%20Azure%20Machine%20Learning.%20The%20full%20code%20is%20available%20in%20our%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2Fazureml-examples%2Ftree%2Fmain%2Fcli%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Esamples%20repository.%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-1359422235%22%20id%3D%22toc-hId-1359422263%22%3ESample%20deployment%20with%20TensorFlow%20Serving%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20deploy%20a%20TensorFlow%20model%20with%20TensorFlow%20Serving%2C%20first%20create%20a%20YAML%20file%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-yaml%22%3E%3CCODE%3Ename%3A%20tfserving-endpoint%0Atype%3A%20online%0Aauth_mode%3A%20aml_token%0Atraffic%3A%0A%26nbsp%3B%20tfserving%3A%20100%0A%0Adeployments%3A%0A%26nbsp%3B%20-%20name%3A%20tfserving%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%20model%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20name%3A%20tfserving-mounted%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20version%3A%201%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20local_path%3A%20.%2Fhalf_plus_two%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%20environment_variables%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20MODEL_BASE_PATH%3A%20%2Fvar%2Fazureml-app%2Fazureml-models%2Ftfserving-mounted%2F1%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20MODEL_NAME%3A%20half_plus_two%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%20environment%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20name%3A%20tfserving%0A%26nbsp%3B%20%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bversion%3A%201%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20docker%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20image%3A%20docker.io%2Ftensorflow%2Fserving%3Alatest%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20inference_config%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20liveness_route%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20port%3A%208501%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20path%3A%20%2Fv1%2Fmodels%2Fhalf_plus_two%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20readiness_route%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20port%3A%208501%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20path%3A%20%2Fv1%2Fmodels%2Fhalf_plus_two%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20scoring_route%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20port%3A%208501%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20path%3A%20%2Fv1%2Fmodels%2Fhalf_plus_two%3Apredict%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%20instance_type%3A%20Standard_F2s_v2%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%20scale_settings%3A%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%20scale_type%3A%20manual%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20instance_count%3A%201%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20min_instances%3A%201%0A%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20max_instances%3A%202%0A%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThen%20create%20your%20endpoint%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-bash%22%3E%3CCODE%3Eaz%20ml%20endpoint%20create%20-f%20endpoint.yml%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAnd%20that%E2%80%99s%20it!%20You%20now%20have%20a%20scalable%20TensorFlow%20Serving%20endpoint%20running%20on%20Azure%20ML-managed%20compute.%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--448032228%22%20id%3D%22toc-hId--448032200%22%3ENext%20steps%3C%2FH1%3E%0A%3CUL%3E%0A%3CLI%3ERead%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fmachine-learning%2Fhow-to-deploy-custom-container%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Eour%20documentation%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3ESee%20the%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2Fazureml-examples%2Fblob%2Fmain%2Fcli%2Fdeploy-torchserve.sh%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Esample%20with%20TorchServe%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3ELearn%20more%20about%20our%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fmachine-learning%2Fconcept-prebuilt-docker-images-inference%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EAzure-built%20inference%20images%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3ELook%20out%20for%20future%20samples%20showing%20ML.NET%20and%20R%20support%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2460558%22%20slang%3D%22en-US%22%3E%3CP%3EUse%20TensorFlow%20Serving%2C%20TorchServe%2C%20and%20other%20custom%20containers%20in%20Azure%20Machine%20Learning%20managed%20online%20endpoints%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2460558%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Machine%20Learning%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMachine%20Learning%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMLOps%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Jun 17 2021 01:48 PM
Updated by:
www.000webhost.com