TL; DR: Use MSAL and OAuth ROPC with scope 499b84ac-1321-427f-aa17-267ca6975798/user_impersonation.

Microsoft’s Graph API (MS Graph) [1] is a convenient way to access a vast amount of Azure data programmatically. Its use is straight forward and generally speaking painless. However, there are still many Azure services APIs that haven’t been integrated, such as the Azure DevOps API (AZ DevOps) [2]. The AZ DevOps API originates from the Team Foundation Server (TFS) which had its API designed long before MS Graph. Therefore, it might be a little confusing if working with these APIs of different concepts. This post helps to lay out the foundation for accessing the AZ DevOps API using OAuth 2.0 and a token provided by Azure Active Directory (AAD).

The AZ DevOps API comes with a particularity, which is that it does not allow to access the API as an application (so-called application permissions), but requires user credentials (so-called delegated permissions). Why Microsoft made this design decision is unclear to me :confused:. As a result, it is recommended to create a personal access token (PAT) as a user and use it in your automation stack [3]. This approach is not desirable if you plan to run some scripts as Azure Functions or somewhere in a VM. Everyone that has access to these scripts also has access to your account through the PAT.

In order to be somewhat “compliant” to OAuth 2.0 [4], this post focusses on registering an app with Azure and granting access via a service account (SA).

The following graphic illustrates the involved components:

architecure

In this post’s scenario, AAD has been connected to AZ DevOps and acts as endpoint to log in.

Step-by-Step

Create Service Account

The SA is a dedicated account for the purpose of communicating with the AZ DevOps API. It is a simple Active Directory account, just like any other ordinary account for a user. This account might be created in AAD, or in an on-premises AD that synchronizes user and groups to an AAD.

Add this SA to the AZ DevOps organization (and projects) that is intended for automation through the API and ensure the SA has all the required privileges.

Get the SA object id using AZ CLI:

az ad user show --id SA-NAME@DOMAIN | jq ".objectId"
"some-UUID-string"

Remember the object id for the next steps.

Choose OAuth 2.0 Grant Type

The AAD supports a number of grant types. Each of these grants cover a different scenario on how to obtain an access token.

If presented the situation to implement an OAuth 2.0 grant flow for a new application, make sure to find the right grant type for your scenario. Read about all of them and pick the one that fits your requirements: https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-v2-protocols

At the time of writing this post, the following grant types are supported:

OAuth 2.0 implicit grant flow
OAuth 2.0 auth code grant
OAuth 2.0 on-behalf-of flow
OAuth 2.0 client credentials grant
OAuth 2.0 device code flow
OAuth 2.0 resource owner password credentials grant
OAuth 2.0 SAML bearer assertion flow

In this specific case, the OAuth 2.0 resource owner password credentials grant (ROPC) suits our scenario. How do we know that? The AZ DevOps API only supports user credentials-based authentication rather than “application authentication”. All other grant types require “application authentication”. Application authentication isn’t exactly a correct OAuth term :wink:, more on this topic during the next chapters.

Create App Registration

Let’s begin with the definition on an AAD application:

An application that has been integrated with Azure AD has implications that go beyond the software aspect. “Application” is frequently used as a conceptual term, referring to not only the application software, but also its Azure AD registration and role in authentication/authorization “conversations” at runtime. - MS [5]

For the sake of simplicity, a registered application is called app in this post.

An application registration creates an application object that represents the application uniquely within the AAD tenant. Further, a service principal is created which comparable to a service account. It is used to:

.. access resources that are secured by an Azure AD tenant, the entity that requires access must be represented by a security principal. This is true for both users (user principal) and applications (service principal). - MS [5]

In lay mans terms, a service principal (SP) is an entity in AAD that represents an app. Access policies and permissions are configured, based on SPs in AAD. If an app is mentioned, it is most likely that the SP is meant.

In order to create an app registration, go to the Azure Portal -> App registrations -> Create.

Name

Give the app a memorable name and create it. Once created, remember the Application (client) ID (it seems the Object ID can be used interchangeably, but I recommend to go with the app id).

Authentication

Next, go to Authentication and select Treat application as public client: YES. This allows us to work with the ROPC grant type.

Certificates & Secrets

Because this application is configured a public client, there is no need to configure a client secret. With this configuration, the SP cannot be used for authentication, instead the user’s credentials are used.

API permissions

The next required configuration step is setting API permissions. Add permission for the AZ DevOps service. You will notice that it is not possible to select Application Permissions. Again, it is unclear why Microsoft is not offering this option. Choose Delegated permissions and user_impersonation as the only available option.

Generally speaking, if an app is configured with application permissions, then the user gets redirected to AAD for authentication. Once authentication is completed, the app receives a token which it uses to authenticate. It never gets access to the user credentials. When the app contacts an API with said token, it uses the SP as its identity. Accordingly, the application permissions define effectively what an app is allowed to do - even if the user would have additional permissions. Application permissions are represented by the scope.

If an app is configured with delegated permissions, then it has access to user credentials and uses them to perform the authentication on behalf of the user. This is obviously a security risk, because a third-party is in possession of a user’s credentials. After successful authentication, the app receives a token which it uses to talk to APIs. However, the app is not represented by a SP but by the logged in user. Accordingly, the only permission of the app is called user_impersonation. An app with delegated permissions is allowed to do everything the user is allowed to do.

Manifest

Go to the Manifest and remember the resourceAppId under requiredResourceAccess. resourceAppId is the unique resource ID of the AZ DevOps service. This is a permanent, static id of value 499b84ac-1321-427f-aa17-267ca6975798. This id is required to configure the scope of our OAuth 2.0 grant.

Choose Token Version

It is essential to use the appropriate token version. The token is issued by a Microsoft Authority service. This might be the AAD if you are an enterprise user. In case of a non-enterprise user, a token is issued by another Authority (e.g. AZ DevOps Authority at https://app.vssps.visualstudio.com/oauth2/token). The token is a JWT token and can be used with a variety of authentication protocols. For example, Basic Authentication or OAuth 2.0. In most situations, it is recommended to choose OAuth 2.0 in which case the issued authorization token from an Authority is referred to as access token.

Depending on the token version, the scope and or resource information are defined differently. If that information is incorrect, you might get a token from the Authority service, which is not accepted by the endpoint (where you want to log in).

Generally speaking, if you use AAD as endpoint, you need v1 tokens and if you use Microsoft’s identity platform as endpoint, you need v2 tokens [6].

With the Microsoft Authentication Library (MSAL), you’re using Microsoft’s identity platform endpoint by default (but it also allows to access v1 endpoints). The ADAL Library is was used to authenticate against AAD as endpoint.

An AAD (v1) endpoint requires both resource identifiers and scope information. For example, resource=https://graph.microsoft.com/&scope=Directory.Read.

A Microsoft Identity Platform endpoint only requires a scope. For example, https://graph.microsoft.com/directory.read.

While ADAL (v1) acquires tokens for resources, MSAL (v2) acquires them for scopes.

The AAD can be used both as authority to get access tokens and as endpoint to validate them. Both endpoints (AAD and MS identity platform) accept tokens from AAD as authority.

To determine what endpoint is supported by the designated AZ service (in this case AZ DevOps), check the accessTokenAcceptedVersion field in the application manifest of said service:

Possible values for accessTokenAcceptedVersion are 1, 2, or null. If the value is null, this parameter defaults to 1, which corresponds to the v1.0 endpoint. - [7]

accessTokenAcceptedVersion: null and accessTokenAcceptedVersion: 1 mean AAD (v1) and accessTokenAcceptedVersion: 2 means MS identity platform (v2).

I have not found a consistent way to retrieve the accepted access token version of Azure services, such as Azure DevOps or Data Lake Analytics (Trial & Error or search the web..:persevere:) [8].

So far, I have seen three different scope formats to access the AZ DevOps API:

Format	Description
`vso.graph_manage`	This format can be used if `PAT` tokens are created from AZ DevOps for AZ DevOps [9]. AZ DevOps acts both as authority and endpoint (v2).
`499b84ac-1321-427f-aa17-267ca6975798/user_impersonation`	This format is used if tokens are issued from AAD for AZ DevOps (see appendix for a complete OAuth 2.0 token for this scenario). `499b84ac-1321-427f-aa17-267ca6975798` is the unique, permanent `resourceAppId` for AZ DevOps (v1).
`https://app.vssps.visualstudio.com/.default`	It’s unclear to me when this is the appropriate format (v2). (TODO: figure out.)

AZ DevOps supports AAD but not MS identity platform as endpoint, hence the token version is v1. And because AZ DevOps only supports delegated app permissions, the token scope is always: 499b84ac-1321-427f-aa17-267ca6975798/user_impersonation.

Choose Authentication Library

You can use any library that supports OAuth 2.0. However, Microsoft’s current standard library for authentication is MSAL [10].

In order to implement ROPC using MSAL, configure the following properties:

Authority: https://login.microsoftonline.com/AAD-TENANT-ID/v2.0
Application ID (or object id): UUID
User: SA username
Password: SA password
Scope: 499b84ac-1321-427f-aa17-267ca6975798/user_impersonation

Once configured, the grant flow looks similar to:

Request token from AAD (often via ADFS if on-prem AD is source of truth)
Receive consent message
Manually consent using SA
Request token again
Receive token
Call AZ DevOps API and provide token

This concludes all steps necessary to get a valid token from AAD to access the AZ DevOps API. Once translated into code, you will notice it is just a few lines… :sweat_smile:.

Example: Get valid AAD Token for AZ DevOps API

A demo app using Python 3.

Dependencies:

azure-devops==6.0.0b2
msal==1.2.0
msrest==0.6.13

Code:

import msal
from azure.devops.connection import Connection
from msrest.authentication import OAuthTokenAuthentication

config = {
    "authority": "https://login.microsoftonline.com/AAD-TENANT-ID",
    "client_id": "APP-ID",
    "username": "SA-EMAILADDRESS",
    "password": "SA-PASSWORD",
    "scope": ["499b84ac-1321-427f-aa17-267ca6975798/.default"], # or 499b84ac-1321-427f-aa17-267ca6975798/user_impersonation
    "org_url": "https://dev.azure.com/YOURCOMPANY"
}

# PublicClientApplication for OAuth 2.0 ROPC
app = msal.PublicClientApplication(
    config["client_id"], authority=config["authority"])

token = app.acquire_token_by_username_password(
    config["username"],
    config["password"],
    scopes=config["scope"])

# Check if consent is required
if 65001 in token.get("error_codes", []):
    print("Visit this to consent:",
            app.get_authorization_request_url(config["scope"]))

    raise Exception("Consent required. See message above")

# BasicAuthentication() as alternative
credentials = OAuthTokenAuthentication('SA-OBJECT-ID', token)
connection = Connection(base_url=config["org_url"], creds=credentials)

# For demo purpose, get user profile of SA
profile_client = connection.clients.get_profile_client()
profile = profile_client.get_profile('me')
print(profile)

Output:

{
  'additional_properties': {
    'displayName': 'SA-USERNAME',
    'publicAlias': 'UUID',
    'emailAddress': 'SA-EMAIL'
  },
  'application_container': None,
  'core_attributes': None,
  'core_revision': 325423388,
  'id': 'UUID',
  'profile_state': None,
  'revision': 325423388,
  'time_stamp': datetime.datetime(2020, 4, 24, 11, 23, 8, 456666, tzinfo=<FixedOffset '+00:00'>)
}

Sources

Appendix

A valid OAuth 2.0 payload (as of May 2020) from AAD for AZ DevOps:

{
  "token_type": "Bearer",
  "scope": "499b84ac-1321-427f-aa17-267ca6975798/user_impersonation",
  "expires_in": 3599,
  "ext_expires_in": 3599,
  "access_token": "JWT-TOKEN",
  "refresh_token": "JWT-TOKEN",
  "id_token": "JWT-TOKEN",
  "client_info": "...",
  "id_token_claims": {
      "aud": "UUID",
      "iss": "https://login.microsoftonline.com/AAD-TENANT-ID/v2.0",
      "iat": 1588870189,
      "nbf": 1588870189,
      "exp": 1588874089,
      "name": "SA-USERNAME",
      "oid": "SA-OBJECT-ID",
      "preferred_username": "SA-EMAILADDRESS",
      "sub": "SOMESTRING",
      "tid": "UUID",
      "uti": "SOMESTRING",
      "ver": "2.0"  # OAuth v2.0 (this is not the token version)
  }
}

The access_token of the OAuth 2.0 payload:

{
  "aud": "499b84ac-1321-427f-aa17-267ca6975798",  # AZ DevOps resource id
  "iss": "https://sts.windows.net/UUID/",
  "iat": 1588870189,
  "nbf": 1588870189,
  "exp": 1588874089,
  "acr": "1",
  "aio": "SOMESTRING",
  "amr": [
    "pwd"
  ],
  "appid": "UUID",
  "appidacr": "0",
  "ipaddr": "SOURCE-IP-ADDRESS",
  "name": "SA-USERNAME",
  "oid": "SA-OBJECT-ID",
  "onprem_sid": "S-1-5-UUID",
  "puid": "SOMESTRING",
  "scp": "user_impersonation",
  "sub": "SOMESTRING",
  "tid": "UUID",
  "unique_name": "SA-EMAIL-ADDRESS",
  "upn": "SA-EMAIL-ADDRESS",
  "uti": "SOMESTRING",
  "ver": "1.0"    # it's a v1 token (AAD)
}

Step-by-Step#

Create Service Account#

Choose OAuth 2.0 Grant Type#

Create App Registration#

Choose Token Version#

Choose Authentication Library#

Example: Get valid AAD Token for AZ DevOps API#

Sources#

Appendix#