Microsoft Teams on Citrix Virtual Apps and Desktops, part #3 – 18 tips for optimizing performance

So, let’s do part #3 of a series on Teams. Mainly, around some actions you can take to help optimize the performance for your user base!

When we are discussing optimizing Teams, what most admins and users are interested in is performance in audio, video, screen sharing, calling and remote control situations. The other features of Teams – chat and document sharing – is generally not a pain point. It’s only when the functions move into audio/visual territory, and the hardware that accompanies this, that the brown stuff really hits the fan.

Free Video Chat and Collaboration | Microsoft Teams

Microsoft Teams

I’m sure we all know what Teams is by now, no introductions required! 🙂

When using Windows Server RDSH instances/Windows 10 Multi-User instances and/or Windows client VDI instances, there are implications for performance and user density given the shared nature of RDSH/Win10 MU platforms, and/or the shared resource nature of Windows client VDI. Ensuring that all users have an optimal experience that does not impact other users on the shared platform and/or shared hypervisor fabric is of paramount importance. The following key points should allow Teams to operate in an optimal mode that provides the best user experience.

Set user expectations

It is important to note that there is a noticeable discrepancy between the traditional “desktop” version of Teams and the “VDI” version of Teams that is commonly used in RDSH or virtual desktop environments. Feature parity lags quite starkly between the two, with the “VDI” version the poorer relation.

Features such as “background blur” and the Gallery View are commonly called out by users who move to the VDI version of Teams and are already familiar with the desktop version. Unawareness of these limitations can often convince the users that Teams is “broken” and lead to helpdesk calls.

Setting the user expectations around what is possible or limited within the VDI version of Teams can help to head off these issues. Citrix publish a roadmap of CVAD features that can sometimes provide an insight into when Teams features will “catch up” with the desktop version. It can be useful to share this either with users or support personnel (https://www.citrix.com/en-gb/support/citrix-virtual-apps-and-desktops-roadmap.html)

Key action #1 – set user expectations as to differences between the desktop and VDI versions of Teams, provide roadmap to support this

User self-help

Many problems that users encounter within Teams can be resolved by learning some basic troubleshooting techniques. Exiting Teams correctly by right-clicking the notification area icon and choosing “Quit” can help with many issues.

Encouraging users to check that their devices are detected correctly is another way you can enable users to head off potential service desk calls. If they are not, exiting Teams correctly in the way described above will often resolve the issue.

Key action #2 – educate users in basic Teams troubleshooting, with supporting documentation as necessary

Session disconnections

It is also worth noting that when users disconnect sessions rather than logging out, the Citrix HDX virtual channel that handles Teams can disconnect and not re-establish properly. Around this, it can be useful to set up robust timeout and disconnection policies to prevent users from experiencing this issue. There are a number of factors to consider when deploying these policies, namely around understanding how users make use of the system, and this understanding will drive the policies that you implement. A thorough discussion of disconnection and timeout policies within RDSH or VDI environments is provided here – https://4sysops.com/archives/securing-timeouts-in-remote-desktop-session-host-rdsh-and-virtual-desktop-infrastructure-vdi-environments/

Key action #3 – implement suitable timeout and disconnection policies within the customer environment

Teams installation and configuration

Installation

We already covered this in part #1 of this series, but lets reiterate that it is crucially important to install Teams correctly for the environment. The deployment command line will vary depending on whether you are using RDSH/non-persistent VDI, or fully persistent VDI.

If the installation is not done correctly and a mix of desktop and VDI versions is allowed to proliferate, or even a mix of individual VDI versions on shared systems, users will experience high levels of Teams instability and errors.

For RDSH or other non-persistent, the following command must be used for installation:-

msiexec /i <path_to_msi> /l*v <install_logfile_name> ALLUSER=1 ALLUSERS=1

For persistent VDI, the following command must be used:-

msiexec /i <path_to_msi> /l*v <install_logfile_name> ALLUSERS=1

The first command, which is the most common one to use, disables the automatic update feature and is crucial to maintaining control within virtual environments with an administrator-managed image.

Key action #4 – ensure that Teams installation is done correctly dependent on the environment

First-run configuration

It is also prudent to disable the automatic first run of Teams to prevent a storm of resource utilization when Teams is initially deployed. However, once it is run user-initiated for the first time, it is also prudent to then allow it to automatically launch for every user session thereafter.

This cannot be configured using GPOs or command-line switches, contrary to what it may say in Microsoft documentation or online guides. The ideal way to configure this is by following these steps:-

  • Delete the HKLM Run key for Teams from the golden or master image
  • Create a Group Policy Preference, WEM config item or other method that creates a conditional Registry value in the user profile (details in next point)
  • Creates a HKCU\Software\Microsoft\Windows\CurrentVersion\Run entry that calls the following command line (C:\Program Files (x86)\Microsoft\Teams\current\Teams.exe” –process-start-args “–system-initiated”) but ONLY when the following folder exists (%APPDATA%\Microsoft\Teams\Cache)

For a full rundown of the installation and first-run config, see part #1 of this series.

Key action #5 – remove auto-start, configure Registry settings to disable first-run storm

Pre-staging options

Again, this was covered in more detail in part #2 of this series, but we’ve put a summary here just for posterity.

Microsoft recommend that the option to “disable GPU acceleration” within Teams is enabled for non-GPU workloads and disabled for GPU workloads. In order to set this appropriately without user intervention, it is recommended to “pre-stage” a JSON file within the default user profile in the master image that matches the requirements of the workload.

This can be done by creating a file within the master image at c:\users\Default\AppData\Roaming\Microsoft\Teams called desktop-config.json which has the following format (changed to match the workload requirement)

{
   "appPreferenceSettings": {
      "runningOnClose": true,
      "disableGpu": true,
      "callingMWEnabledPreferenceKey": false
   },
   "theme": "default"
}

disableGPU should be set to true for non-GPU workloads and false for GPU-enabled workloads. Other settings can be pre-configured here as required, although some of them do not take properly (RegisterAsIMProvider is a particularly troublesome setting).

Key action #6 – pre-stage a JSON file into the master image to optimize base user settings as required

Teams HDX optimization

For shared systems (whether sharing the actual VM resource or the underlying hypervisor resource), Teams optimization is the most crucial setting to enable. Citrix HDX optimization allows the Citrix virtual channel to “offload” Teams audio and video traffic to the connecting endpoint, removing the impact to CPU and other VM resources that is commonly observed when using audio and video calling features.

In order to enable Teams optimization, a number of pre-requisites must be checked.

Connecting client specs

The following table details the recommended client specifications:-

Device classSpecifications and notes
Windows clientWindows 7, Windows 8, Windows 8.1 or Windows 10 1607+
4GB RAM+
x86 or x64
2.2Ghx+ dual-core CPU or 1.5Ghz+ quad-core CPU with boost up to 2.4Ghz
Windows thin clientHP t360/t640, t730/740 or mt44/mt45
Dell 5070 or 5470
10Zig 4510 and 5810q
Full list here – filter by Thin Clients and MS Teams Optimized Windows 10 IoT Enterprise 2016 or 2019
MacOSmacOS Catalina (10.15)
Linux clientx64 Linux distribution
2.2Ghx+ dual-core CPU or 1.5Ghz+ quad-core CPU with boost up to 2.4Ghz
GStreamer 1.0 or later or Cairo 2
libcc++9.0 or later
libgdk 3.22 or later
OpenSSL 1.1.1d
Linux thin clientHP t360/t640, t730/740 or mt44/mt45
Dell 5070 or 5470
10Zig 4510 and 5810q
Full list here – filter by Thin Clients and MS Teams Optimized x64 Linux distribution
ChromebookChromeOS v50 or later
AndroidNo optimization currently possible
iOSNo optimization currently possible

Citrix client specifications

In addition to the requirements specified above, there is also a minimum Workspace App (Citrix client) version that must be present for offloading to be possible. The required and recommended versions are detailed below.

Device classVersions
Windows client (Windows 8 and higher)Citrix Workspace App 1907+ (minimum)
Citrix Workspace App latest n-1 (recommended)
Windows client (Windows 7)Citrix Workspace App 1907+ (minimum)
Citrix Workspace App 2002 (recommended – any higher not supported)
Windows thin clientCitrix Workspace App 1907+ (minimum)
Citrix Workspace App latest n-1 (recommended)
MacOSCitrix Workspace App 2009+ (minimum)
Citrix Workspace App latest n-1 (recommended)
Linux clientCitrix Workspace App 2006+ (minimum)
Citrix Workspace App latest n-1 (recommended)
Linux thin clientCitrix Workspace App 2009+ (minimum)
Citrix Workspace App latest n-1 (recommended)
ChromebookCitrix Workspace App 2105+ (minimum)
Citrix Workspace App latest n-1 (recommended)
AndroidNo optimization currently possible
iOSNo optimization currently possible

Teams client version

There is also a required Teams version that must be in use in order for optimization to be possible. The minimum is 1.2.00.31357, but again, it is recommended that the latest possible “VDI Installer” or “Machine Installer” of Teams is applied to the target Citrix devices. This should be updated as regularly as possible.

Key action #7 – ensure Teams client versions, Workspace app versions and connecting client specifications meet the requirements stated

Citrix infrastructure versions

The latest Citrix VDA should be applied to the target devices to take advantage of Teams enhancements that may be incorporated into the VDA itself. There are also requirements around the Citrix DDC versions that must be met. However, these points introduce a further discussion – namely, Current Release (CR) versus Long Term Service Release (LTSR).

Citrix VDA version

The minimum VDA version required is 1906.2 for CR, and 1912 for LTSR. However, it is recommended that the latest possible VDA version is applied (for CR) or the latest possible Cumulative Update (CU) (for LTSR). Additionally, the VDAs must be running Windows 10 1607+, or Windows Server 2012 R2+, in order to support Teams optimization.

DDC version

The Citrix DDCs (Desktop Delivery Controllers) must also be at a version of 1906.2+(CR) or 1912 (LTSR) to support Teams optimization features.

CR or LTSR

Citrix have a CR schedule in which updates to the core infrastructure components appear quite regularly (often quarterly). LTSR, on the other hand, putatively states that these versions can stay in a supported configuration for up to five years.

However, this is somewhat misleading. LTSR versions are supported as long as customers remain two cumulative updates or less behind the current CU release, and CR versions continue to be supported within two releases of the current latest version. In real terms, this means that the support window for LTSR is approximately two years, and that for CR the window of support can extend for approximately 18 months. Given that CR allows easier and quicker deployment of new features as they become available, which is particularly relevant to Teams optimizations, and the achievement of feature parity between the desktop and VDI Teams versions, it is clear that adopting the CR method is more desirable when it comes to a quality Teams experience.

Key action #8 – adopt the CR model where possible (n-1 from latest release recommended), or ensure that LTSR model remains up-to-date with Cumulative Update releases (currently latest CU at time of writing is CU3 which means minimum supported version would be CU1)

Updating client devices

In many environments, updating the Workspace App version on the endpoints can be difficult, particularly where the customer has embraced “Bring Your Own Device” initiatives, or if third parties are connecting to the solution. Ideally, devices should run the Workspace App with the auto-update feature enabled, but this is not always possible in every situation.

To mitigate the risks posed by this, there are a few considerations that are worth thinking about.

Old Citrix infrastructure

It is important to check for old Citrix infrastructure that might mandate an older Citrix client, such as old NetScaler instances, legacy Storefront or Web Interface, and/or older Citrix farms. Often these older instances can force users into having to use an older Citrix client, commonly the 4.9 Receiver version which is still updated by Citrix because of its wide install base.

If these situations exist, there are two choices:-

  1. Accept that these users will have to run without Teams optimization, and either decrease user density to cope, silo these users, or disable resource-intensive Teams functionality (see Teams fallback mode, discussed later)
  2. Apply a workaround to allow users to access multiple Citrix client versions. This can be achieved using FSLogix, but this is an unsupported configuration and will require detailed testing

Key action #9 – validate whether older clients are required, and implement a solution if this is found to be the case

Warning for unoptimized users

It is possible to configure a “nag” alert that will warn users that they are on an older Citrix client version. Scripts are available that will alert the user if they are at a client version older than a certain defined level, and can advise them to download the newer version or contact support.

The script runs via PowerShell so there may be security constraints which could affect the viability of this solution, however there are options such as converting it to an executable which may make it more viable.

CTA Dennis Mohrmann provides the script that can do this which is linked here. We are going to do a separate post running through using this in the very near future. Dennis also has a script to check optimization status but this runs every time a Teams process spawns – we’ll talk more about that too in the dedicated article.

Other options around checking the client version exist, such as using ADC Endpoint Analysis or other contextual posture checking. Dependent on what is available to the customer, this may also be worth investigating.

Key action  #10 – investigate the possibility of a posture check or nag on the connecting client version to reduce unoptimized sessions

HTML5 Receiver

Citrix have an option to use the HTML5 version of the Workspace App which runs embedded in the browser instead of using the full client. Some users find this useful, however it does not support Teams optimization and should be avoided.

Key action #11 – educate users on how to use the full client to launch their resources rather than the HTML5 version

Teams fallback mode

There is an option to disable what is known as “fallback mode”. This means that when optimization is unavailable, the connecting client “falls back” to using the resources on the VM they are connected to.

To reduce the impact that non-optimized users can have on other sessions, it is possible to selectively disable fallback mode so that they cannot launch resource-intensive tasks within Teams (such as audio and video calling). This is done by setting a Registry value either on a computer or user level. The available Registry values are listed below:-

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Teams\DisableFallback

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Office\Teams\DisableFallback

Both values should be set as DWORD values and can be either 0 (fallback is enabled), 1 (audio and video are disabled for non-optimized users) or 2 (video is disabled for non-optimized users.

It is recommended that this is set to 1, so that the impact of unoptimized users is reduced, as well as encouraging them to contact support to resolve their problems with optimization failure.

Note – this feature is currently broken! I have a case open with Microsoft to try and allow it to work properly and will update when the case moves forwards.

Key action #12 – implement DisableFallback to reduce the impact of non-optimized users across the estate

Turn off incoming video

There is an option to “turn off incoming video” which can be used in meetings that seems to increase performance by an order of magnitude. We are currently exploring how this can be defaulted to “off” for users – it may be possible on a tenant level but we have not fully investigated this as yet.

Web client

Some people are keen to explore the use of the Teams web client rather than the full fat client, and then leverage Citrix features such as Browser Content Redirection to offload the traffic. This is not desirable on a number of levels. Firstly, the feature set is not complete. Secondly, using Browser Content Redirection to offload Teams is not straightforward and involves a lot of integration with authentication providers and security devices such as internet proxies. Finally, there are many entry points providing web client users with paths to install the “desktop” client version of Teams, and restricting this (because it is a user profile installer) is difficult and means using technologies like AppLocker or FSLogix App Masking. At this moment, it is not advisable to fall back to the web version of Teams in Citrix environments unless no other option is available. There may be a post forthcoming where we discuss how to use it as best as possible.

Profile management

Teams has a sizeable user cache which is written to very regularly. This means that managing the user profile under Citrix can be somewhat challenging.

VHD solutions

In order to manage the Teams cache effectively, it is important to adopt a “virtual hard disk” based method to management. This can be done either by capturing the entire user profile into a VHD, or simply the Teams cache itself.

There are three main solutions that can be used to capture the profile or individual caches into a VHD:-

  • FSLogix Profile Containers or Office Containers
  • Citrix User Profile Management with Profile Container feature
  • Ivanti User Workspace Manager

Dependent on the customer entitlements, any of these will suffice to capture the required data into a VHD and avoid the performance issues associated with older, file-based profile management systems.

FSLogix would be the recommended solution, as the entitlement is free with Microsoft365, Office365, Windows 10 licenses or RDSH CALs. If a different profile management solution is incumbent, the Office Container solution rather than the Profile Container can be used to complement the existing one.

If FSLogix cannot be used or the customer wish to leverage an existing solution, then Citrix User Profile Management and Ivanti User Workspace Manager both offer a method of capturing either the entire profile or the Teams cache folder into a VHD file.

Key action #13 – deploy a container-based VHD solution for profile management when using Teams

Exclusions

Again, this is something we’ve covered in detail in a previous post, but for completeness let’s mention it with the other optimizations. In order to minimize the storage footprint required, it is recommended, if possible, to exclude certain folders from the Teams cache. The most crucial folder to exclude from the capture is the following below:-

%APPDATA%\Microsoft\Teams\Service Worker\Cache Storage

This ensures that a total of 5GB per user is trimmed from the cache storage requirement at the first run of Teams.

I’d also recommend excluding the Teams processes from Citrix WEM CPU spike protection, if you are using this feature.

Key action #14 – configure profile exclusions as above

Maintenance

The trade-off with VHD file solutions is that they add state, rather than removing it. In order to keep the size of the VHDs down, it is recommended to run regular pruning and compaction on the cache stores. This can be configured to run offline using community scripts from a variety of locations. Again, we’re going to have a follow-up post about VHD maintenance – stay tuned!

Antivirus exclusions

Microsoft recommend that AV exclusions for Teams are configured to allow best performance. The following folders should be excluded from reactive AV and other security software suites such as HIPS, DLP, etc.

  • %userprofile%\AppData\Local\Microsoft\Teams\current\teams.exe
  • %userprofile%\AppData\Local\Microsoft\Teams\update.exe
  • %userprofile%\AppData\Local\Microsoft\Teams\stage\squirrel.exe

Key action #15 – configure AV exclusions as above

Network performance

Robust and performant network infrastructure is required to support Teams, particularly when using the audio and video calling features within meetings. The following metrics suggest thresholds for performance on a network level

There is a Microsoft tool called “Advisor for Teams” that will assess the suitability of network links which can be run prior to deployment. Technologies such as Citrix SD-WAN can also be used to increase performance.

Key action #16 – validate and track network routing and performance

Monitoring

It is recommended that monitoring on both a network, hypervisor and session level is put in place to help with continually trending, baselining and assessing Teams performance. Without these metrics, judgements on performance become subjective. The monitoring should encompass the entire “service” provided by the solution and be collated centrally for easy access by operations and executives alike.

Key action #17 – ensure appropriate monitoring is in place from the start

Summary

Microsoft Teams is a difficult piece of software to deploy virtually, given the ever-changing nature of the software and the fast update cadence. Microsoft can make changes on a tenant level which affect the layout and performance of the user interface, without any change being made on the customer end. This makes it a uniquely challenging application.

With good monitoring, solid controls, continuous development and appropriate management buy-in, it can be deployed in a manner which provides a reliable user experience. It must be reiterated, though, that this involves keeping on top of the deployment in a way that is possibly unfamiliar to operations teams.

Given the way that both the Teams client, the Citrix Workspace App, the Citrix VDA and Citrix infrastructure components must be regularly updated to keep pace with Teams feature developments, it is incredibly useful to start exploring the concept of “evergreen” installations and rolling rebuilds rather than maintaining a single golden or master image. When the Teams client is updated, it must be uninstalled fully prior to updating – it cannot be upgraded in place. This makes a rolling rebuild mechanism using technologies such as Terraform, Packer, Ansible, etc. much more desirable for Teams deployment than the traditional ways such as MCS, PVS or SCCM. Whilst it is appreciated that this is a big sea-change and not something many enterprises could undertake lightly, the shift towards infrastructure-as-code and a more devops-style deployment method would reap benefits in the longer term as more Office applications evolve to the same methodology as Teams.

Key action #18 – explore infrastructure-as-code and devops pipeline deployment methods

So there you have it – a whole list of possible things you can do to optimize your Teams experience on Citrix. of course, a lot of this we’ve covered already, but it seems useful to try and bring them together in a guide.

As I said, a lot of spin-off articles are hinted at here – they should be forthcoming pretty soon!

Loading

21 comments

  1. how can i always run teams machine wide installer automatically with custom desktop-json (openashidden) without existing of the folder cache.

  2. Great write up. A couple of things we have found to help the experience is to limit the video to 360p on the thin client (Windows OS). This can be done via Registry key described in https://docs.citrix.com/en-us/citrix-workspace-app-for-windows/configure.html.

    In addition the 7×7 Large Gallery view is more efficient than the 2×2 view. So encourage users to select that. Of course latest workspace for Windows 2106 (out sometime in June) will include a bunch of fixes to help screenshare issues.

  3. How can i update multiple parameters like disableGpu on existing users that doesn’t load the desktop-json from default user?

    1. I already fixed this by using this script on login:

      $filepath = “$env:APPDATA\microsoft\teams\desktop-config.json”
      $existing = Get-Content $filepath -raw|ConvertFrom-Json
      $existing.appPreferenceSettings.openAsHidden = $true
      $existing.appPreferenceSettings.runningOnClose = $true
      $existing.appPreferenceSettings.disableGpu = $true
      $existing.appPreferenceSettings.registerAsIMProvider = $true
      $existing.appPreferenceSettings.enableMediaLoggingPreferenceKey = $false
      $existing |ConvertTo-Json|Set-Content $filepath

      We run this script on every login.
      The only issue i have is that enableMediaLoggingPreferenceKey is being ignored.
      registerAsIMProvider is set after a user has started word or any other office product.

  4. Couldnt schedule the meeting when clicking on the Outlook add-in?
    Do you know what can cause this error/issue on Citrix with FSLogix?
    Not always but very rare this happens to users.

    We enabled the add-in with registry and Teams is machine wide installed.

  5. James, did you get an answer about the disablefallback: Note – this feature is currently broken! I have a case open with Microsoft to try and allow it to work properly and will update when the case moves forwards.

    1. I did. Their answer is that you should use Teams policies to do this on a per-user basis. Not very satisfactory, as I wanted to be able to define it on the device the user was using (so invoke it when they’re on a non-capable device), but they simply say it should only be done per-user)

  6. AppData\Roaming\Microsoft\Teams\Service Worker\CacheStorage NOT

    AppData\Roaming\Microsoft\Teams\Service Worker\Cache Storage

Leave a Reply

Your email address will not be published. Required fields are marked *