Thursday, September 19, 2024

Part 2: How to optimize increase speed of .Net solution build process in Azure Devops Pipeline?

 To improve the time taken by the VSBuild@1 task in an Azure DevOps pipeline for building .NET applications, you can optimize various parameters and configurations similar to the general techniques used for improving build times. Below are specific recommendations for optimizing the VSBuild@1 task:

1. Enable Multi-Processor Builds (Parallel Builds)

You can pass the /m flag to enable multi-core builds, which can significantly reduce build time for solutions with multiple projects.

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    msbuildArgs: '/m'

Explanation:

  • /m enables multi-threaded building, allowing multiple projects to be built concurrently on agents with multiple processors.

2. Use Incremental Builds

Incremental builds can save time by only building projects that have changed since the last build. By default, the VSBuild task cleans the project before building, which can be inefficient. Disable cleaning unless necessary:

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    clean: false

Explanation:

  • clean: false ensures that the build output from previous builds is not deleted, enabling incremental builds where only the projects that have changed are rebuilt.

3. Optimize Build Configuration

Build the solution in Release mode only if necessary (e.g., for production builds). For development or non-critical builds, Debug mode is often faster.

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    configuration: 'Debug'

Explanation:

  • configuration: 'Debug' builds the solution in Debug mode, which generally takes less time than Release mode.

4. Leverage Build Caching

Use caching to prevent repetitive tasks from being executed. Specifically, caching the NuGet packages and intermediate build outputs can significantly reduce the build time.

Cache NuGet Packages:

Use the Cache@2 task to cache your NuGet packages:

steps:
- task: Cache@2
  inputs:
    key: 'nuget | "$(Agent.OS)" | **/*.csproj'
    path: $(NuGetPackageRoot)
    cacheHitVar: NUGET_RESTORED

Explanation:

  • The Cache@2 task saves the downloaded NuGet packages from one pipeline run and reuses them in subsequent runs, reducing the time spent on restoring packages.

Cache Intermediate Build Outputs:

You can also cache build outputs (e.g., compiled libraries) to avoid recompiling them:

steps:
- task: Cache@2
  inputs:
    key: 'build | $(Agent.OS) | $(Build.SourcesDirectory)'
    path: $(Build.BinariesDirectory)
    cacheHitVar: BUILD_CACHE_HIT

5. Use Specific MSBuild Arguments

You can pass additional MSBuild arguments to fine-tune the build process:

  • /p:DeployOnBuild=false prevents deploying during the build, which is useful if you don’t need deployment in your pipeline.

  • /p:RestorePackages=false skips NuGet package restore if it's already handled in a previous step.

  • /p:WarningLevel=1 lowers the verbosity of build warnings, though use this cautiously.

Example:

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    msbuildArgs: '/m /p:DeployOnBuild=false /p:RestorePackages=false'

6. Reduce Verbosity

Reducing the verbosity of logs can slightly speed up the build process by minimizing the amount of data logged:

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    logLevel: 'minimal'

Explanation:

  • logLevel: 'minimal' reduces the amount of output generated during the build, which can improve performance and also make the logs easier to read.

7. Disable Tests During Build (if not needed)

If you are running tests separately in your pipeline, you can disable test execution during the build by using the /p:RunTests=false argument.

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    msbuildArgs: '/p:RunTests=false'

8. Use Hosted Agents with More Resources

If your builds are still taking a long time, consider using agents with more resources (e.g., more CPU and memory). Azure DevOps offers different tiers of hosted agents, and using more powerful agents can reduce build times.

To specify a more powerful hosted agent, use:

pool:
  vmImage: 'windows-latest'  # or 'windows-2022' for better performance

9. Enable NuGet Restore with Parallel Processing

To make NuGet restore faster, use parallel processing:

steps:
- task: NuGetCommand@2
  inputs:
    command: 'restore'
    restoreSolution: '**/*.sln'
    feedsToUse: 'select'
    noCache: false
    disableParallelProcessing: false

Explanation:

  • disableParallelProcessing: false ensures NuGet restore is performed in parallel, which can be faster for solutions with many dependencies.

10. Split Build and Test Phases

If your build process includes running tests, you can separate the build and test phases to avoid unnecessary rebuilds and gain better control over parallelism:

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    msbuildArgs: '/m'
- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\$(BuildConfiguration)\*.Test.dll
    searchFolder: '$(Build.SourcesDirectory)'

This helps by only running tests after the build completes and can allow parallel execution of tests.

11. Use Self-Hosted Agents

If your builds are large and resource-intensive, you can use self-hosted agents with better hardware configurations (e.g., more CPU and RAM). You can configure self-hosted agents to avoid the time it takes for the default Azure DevOps hosted agents to spin up and download all required tools.

Conclusion:

To summarize, improving the time for the VSBuild@1 task in Azure DevOps involves using:

  • Parallel builds with /m.

  • Incremental builds by disabling the clean option.

  • Build caching for NuGet and build outputs.

  • Specific MSBuild arguments like /p:DeployOnBuild=false and /p:RestorePackages=false.

  • Lower log verbosity and splitting build and test tasks.

By fine-tuning the pipeline with these optimizations, you can reduce the build time for your .NET solution.

Part 1: How to optimize azuredevops build pipeline checkout ?

 Optimizing the checkout steps in an Azure DevOps build pipeline can significantly reduce the overall build time. The checkout step is where the source code is pulled from your repository, and by default, it checks out the entire repository with full history, which can be time-consuming, especially for large repositories.

Here are some ways to optimize the checkout step:

1. Shallow Fetch (Checkout Latest Commit Only)

By default, the pipeline checks out the entire repository including the full history. For most build jobs, only the latest commit is necessary. You can use shallow fetch to reduce the time by fetching only the latest commit.

Add this to your pipeline YAML to checkout only the latest commit:

steps:
- checkout: self
  fetchDepth: 1

Explanation:

  • fetchDepth: 1 ensures that only the latest commit is fetched, rather than the entire history.

2. Skip Submodules (if not needed)

If your repository contains submodules and you don’t need them for the build process, you can skip checking them out.

To skip submodules:

steps:
- checkout: self
  submodules: false

Explanation:

  • submodules: false skips fetching and checking out submodules, speeding up the process.

3. Limit File Paths for Checkout (Sparse Checkout)

If you don’t need the entire repository for your build, you can specify a sparse checkout, which limits the files or folders that get checked out. Azure DevOps doesn't natively support sparse checkout in YAML, but you can use git commands to achieve this.

For example:

steps:
- script: |
    git init
    git remote add origin $(Build.Repository.Uri)
    git fetch --depth=1 origin $(Build.SourceBranch)
    git sparse-checkout init --cone
    git sparse-checkout set path/to/folder
    git checkout $(Build.SourceVersion)
  displayName: "Sparse Checkout"

Explanation:
This uses git sparse-checkout to limit the files being checked out to only what’s necessary for the build.

4. Use Caching for Dependencies

If your pipeline frequently checks out the same dependencies (e.g., libraries or tools), use caching to avoid downloading the same files over and over again.

For example, caching NuGet packages:

steps:
- task: Cache@2
  inputs:
    key: 'nuget | "$(Agent.OS)" | **/packages.lock.json'
    path: $(NuGetCacheFolder)
    cacheHitVar: CACHE_RESTORED

Explanation:
This caches dependencies between builds, so the next time the pipeline runs, it can reuse them instead of downloading them again.

5. Use Parallel Checkout (For Multiple Repositories)

If your build pipeline checks out multiple repositories, ensure they are checked out in parallel rather than sequentially.

You can check out multiple repositories as follows:

resources:
  repositories:
  - repository: RepoA
    type: git
    name: Project/RepoA
  - repository: RepoB
    type: git
    name: Project/RepoB

steps:
- checkout: RepoA
- checkout: RepoB

If your resources allow, you can speed this up by running those checkouts in parallel.

6. Use Predefined Agents or Self-Hosted Agents with Git Preconfigured

Azure DevOps uses hosted agents by default. Hosted agents may take time to set up the environment, including installing git or setting up credentials. You can use self-hosted agents with pre-installed Git configurations and credentials to avoid this overhead.

7. Checkout Specific Branches

If you only need a specific branch for the build, ensure that only the required branch is checked out.

steps:
- checkout: self
  persistCredentials: true
  clean: true
  lfs: false
  fetchDepth: 1

8. Disable Clean Option (If Not Necessary)

By default, the pipeline may clean the repository before each build, which deletes the local working directory. If this isn’t required for your build, disable this option to avoid redundant cleanup.

steps:
- checkout: self
  clean: false

9. Increase Agent Resources (For Larger Repos)

If your repository is large and checkout still takes too long, consider using more powerful agents. Azure DevOps provides various tiers of hosted agents, and you can also configure self-hosted agents with more CPU or memory.

10. Run Checkout Asynchronously (For Large Teams)

If your organization has a lot of pipelines running concurrently, you may want to run the checkout step asynchronously across builds. Using the "Run in parallel" feature can improve performance in a large CI/CD setup.

Conclusion:

By applying these techniques—especially shallow checkout, sparse checkout, skipping submodules, and caching—you can significantly reduce the checkout time in your Azure DevOps pipelines. These optimizations can speed up the entire build pipeline and reduce the total time for CI/CD.

Monday, September 16, 2024

How to inject scope service into Singleton Service ?

 ASP.NET Core has three service lifetimes:

  • Transient
  • Singleton
  • Scoped

    How to resolve inject scope service object into singleton service?  Below SomeSingletonService will inject scope service object and resolve it.

Wednesday, September 11, 2024

How to create task to add new variable in existing variablegroups in Azure Devops library

Step 1: Create a Personal Access Token (PAT)

To interact with Azure DevOps REST API, you'll need a Personal Access Token (PAT) with the necessary permissions. Follow these steps:

  1. Go to Azure DevOps portal.

  2. Navigate to User settings (top right corner) > Personal access tokens.

  3. Click on New Token and provide the required access (e.g., "Read and Write to Build").

  4. Save the token for use in the script.

Step 2: Use the Azure DevOps REST API to Update Variable Group

To update the Variable Group, use a PowerShell or cURL script in your pipeline that calls the Azure DevOps REST API.

API Endpoint to Update Variable Group:

PATCH <https://dev.azure.com/{organization}/{project}/_apis/distributedtask/variablegroups/{groupId}?api-version=7.1-preview.1>

Request Body Example:

To add or update variables within the group, you can use the following structure:

{ "variables": { "newVariable": { "value": "newValue" }, "existingVariable": { "value": "updatedValue" } } }

Step 3: PowerShell Script for Azure DevOps Pipeline

Here’s how you can write a PowerShell script to update the Variable Group. Add this as an inline PowerShell task in your pipeline.

steps: - task: PowerShell@2 inputs: targetType: 'inline' script: | $token = "$(System.AccessToken)" $organization = "your-org" $project = "your-project" $groupId = "your-variable-group-id" $url = "<https://dev.azure.com/$organization/$project/_apis/distributedtask/variablegroups/$groupId?api-version=7.1-preview.1>" $headers = @{ Authorization = "Bearer $token" "Content-Type" = "application/json" } $body = @{ variables = @{ "newVariable" = @{ value = "newValue" } "existingVariable" = @{ value = "updatedValue" } } } | ConvertTo-Json -Depth 10 Invoke-RestMethod -Uri $url -Method Patch -Headers $headers -Body $body env: System.AccessToken: $(System.AccessToken) # Ensure "Allow scripts to access OAuth token" is enabled

Key Points:

  • Ensure the System.AccessToken is provided and enabled in the pipeline's Agent Job (under "Allow scripts to access the OAuth token").

  • Replace your-org, your-project, and your-variable-group-id with your actual Azure DevOps values.

  • Adjust the body of the request based on the variables you want to add/update.

Step 4: Add Task to Your Pipeline

Add the PowerShell task to your Azure DevOps pipeline YAML file. This script will invoke the Azure DevOps REST API to update the variable group by adding new variables.

Step 5: Run the Pipeline

Once the task is in place, run the pipeline, and it will update the variable group with the new variables.

This approach should allow you to automate the management of variable groups directly from within your Azure DevOps pipelines.