Part 2 of 2: Testing & Troubleshooting Workspace ONE UEM Automated Domain Join

April 21, 2021   |   by bgarmon

Last updated 7/01/2021 – Added section named For Your Information

This is Part 2 of a 2-part guide to configure Workspace ONE UEM’s Domain Join feature in order to allow newly provisioned Windows 10 devices to automatically join Active Directory from any remote network. Part 1 covered the end-to-end configuration, Part 2 covers testing and troubleshooting of the configuration detailed in Part 1. If you missed Part 1, you can go back to it by clicking this link.

For the end users device, Domain Join via Workspace ONE UEM is a one-time operation. There is no retry if things fail, so when you are configuring this solution you need to test every aspect of it before you send it out in the wild. This post is written so that each Test Case builds on the previous Test case. If you skip around you will miss important technical nuggets.

Understanding Offline Domain Join

Microsoft’s official name for the process being used here is an “Offline Domain Join (ODJ).” The name confuses a lot of people because the machine will be connected to the Internet when this happens and most people association offline to mean “having no network connection.” The “Offline” portion of the name refers to the Windows 10 computer not being on the same network as the Domain Controller when the domain join completes. The technology that makes ODJ function is Microsoft’s djoin.exe, a program included in the Windows OS since the release of Windows 7.

To explain ODJ the following is a summary of what ODJ does and a brief summary of how it’s done. A more detailed look at each of these steps will be discussed later in this post as part of the testing and troubleshooting process.

What HappensHow Microsoft does thisHow Workspace ONE UEM does this
A program named Djoin.exe is run to generate a computer account record in AD for a computer not yet on the network. The result is a text file called a blob.From a Windows Server on the domain, an IT Administrator runs the command djoin.exe /provision /domain /machine /savefile blob.txtThe djoin command is run automatically by the Airwatch Cloud Connector (ACC) following an MDM Enrollment. The resulting blob file is saved in the Airwatch Database.
The blob file must be copied to the Windows 10 computer not yet on the networkAn IT Administrator copies the blob.txt to the computer or adds the contents of the blob.txt to an unattend.xml (which is consumed during OOBE)After MDM enrollment completes, a command is triggered to extract the blob from the database and transfer the blob through the Airwatch Cloud Messaging service (AWCM) to the Windows 10 computer.
Djoin.exe is run on the Windows 10 device to load the blob.txtIT Admin runs djoin.exe /requestodj /loadfile blob.txt or this same action is triggered by the Windows OS setup process based on the unattend.xml.In the previous step UEM used a Microsoft CSP to download the blob which is the equivalent of the Microsoft process without requiring the IT admin to run the command line
The final step involves rebooting the computer.An IT Admin reboots computer, or this reboot is automatically triggering during OOBE by the Windows setup process.If the UEM device is using Factory Provisioning the reboot will happen as part of the final shut-down step during the Sysprep Audit Mode process. A manual reboot will only be required if you are testing this from an existing Workgroup joined machine
After the computer reboot, but before the login screen appears, line-of-site to the DC is established by the VPN App. This enables the end user to use their AD account to logon to Windows.Pre-login VPN solutions like Microsoft Direct Access or a 3rd party VPN solution is used to establish a VPN connection to the DC to complete this step.Workspace One Tunnel establishes the pre-login VPN to complete this step.
The computer is now joined to the domain and the process is complete<-- Look left. <-- Look left.

Useful log files (in no particular order):

Below are the logs for troubleshooting Domain Join, in no particular order. More details about each of these log files will be discussed alongside the test cases below.

SourceLog FileNotes
ACCC:\VMware\AirWatch\Logs\CloudConnector\CloudConnector.logNeeds to be in Debug mode
Windows 10 Device%windir%\debug\Netsetup.logRecords Domain Join status
Windows 10 DeviceHKLM\Software\Airwatch\Prior to the Blob being applied to the device, the Windows Registry will show the domain join configuration in several keys including DomainJoinConfiguration, DomainJoinType, DomainName, HasDomainJoined and a few others
Windows 10 DeviceC:\VMware\AirWatch\Logs\FactoryProvisioning.Service.logThe Factory Provisioning Service records status messages in this log file
Windows 10 DeviceC:\VMware\AirWatch\Logs\Factory Provisioning Services/Website/VMware.Workspace One.FactoryProvisioning.API.logThis log shows what happens during the runtime of the Factory Provisioning Windows API responsible for receiving the PPKG package generation request
Windows 10 DeviceC:\VMware\AirWatch\Logs\Factory Provisioning Service\Services\WMware.Workspace One.FactoryProvisioning.Service.exe.configThe configuration file used for Factory Provisioning
Windows 10 DeviceC:\Windows\Panther\UnattendGC\setupact.logIt's tempting to look at c:\Windows\Panther\setupact.log but that will not show you much. The UnattendGC subfolder has the file you are looking for. Search the file for DJOIN.EXE to see what's going on

VMware maintains a comprehensive list of all of the Workspace ONE UEM log files for all platforms and services at this link.

The remainder of this blog will run through several test scenarios to help validate you have a working configuration.

For Your Information

The Workspace ONE UEM Console and its supporting infrastructure are upgraded monthly by the VMware Cloud Operations teams. The Airwatch Cloud Connector (ACC) is a major component involved in this process. Be aware that the “Auto-Upgrade” checkbox is enabled for the ACC, and it works you should not have anything to worry about as the automated auto-upgrade is downloading DLL files and updating them to the newest version.

On the other hand if you are like me and have never seen an ACC Auto-upgrade work properly, you’ll be manually triggering the ACC upgrade direct from the server via the use of the ACC MSI. If you use the ACC MSI, by design the MSI will remove any customizations to the ACC service “Login As” settings as part of the installation reverting this back to the default of local system. This will break the Domain Join component you previously had working. Back in part 1 of this blog you would have changed this setting to use a service account in order for Offline Domain Join to function. After the ACC MSI finishes the installation to upgrade ACC to the latest version, make sure you stop the ACC Service, change the Logon AS back to what you setup before and then restart ACC.

Test 0: Does basic MDM Enrollment work?

If this is your first time enrolling a Windows 10 device in UEM step away from this blog and come back when you have successfully enrolled a Windows 10 device in Workspace ONE UEM. This blog assumes Workspace ONE UEM and the Airwatch Cloud Connector (ACC) are functional and that all of the pre-reqs are in place for the basic Workspace ONE UEM Console to device MDM based functionality. Before continuing you should complete the following:

  1. In the Workspace ONE UEM Console, configure at least 1 Windows 10 Profile to auto deploy to a Windows 10 device
  2. In the Workspace ONE UEM Console use the Enterprise App Repository (EAR) to configure at least 1 Windows application to auto deploy to the device. I recommend Google Chrome or Adobe Reader. They are small, easy installs and illustrate that the basic Device to Server communications are working properly.
  3. From a Windows 10 computer not yet enrolled into Workspace ONE UEM, open Start Menu > Settings > Accounts > Access work or school > and choose Enroll into MDM only. This enrollment method will validate that email auto-discovery is working. If you are using Microsoft Azure it will confirm that the connection between UEM and Azure is working and if you have an IDP like Ping or Okta integrated with either UEM or Azure, it will validate you have all of this setup properly.

Test 1: Windows 10 in a WorkGroup using Ethernet (NO WIFI)

In this test we are removing a few dozen things that can go wrong just to confirm all the configuration you completed back in the original Part 1 of this blog series. Recommendations for this test:

  1. This test must be completed on the same network as your Domain Controller.
  2. Use Ethernet (not Wifi) to complete this test.
  3. Use a Virtual Machine with Snapshots. If you need help, reference my blog for how to build a VM and follow it by clicking here: How to build a VM

There are several ways to initiate a Domain Join through Workspace ONE UEM including Workspace ONE UEM DropShip Provisioning Online, Offline, or through Microsoft’s Autopilot via a Hybrid Domain Join profile. For the initial test we are skipping all of those methods because we need to first validate basic Microsoft Domain connectivity, then extend the tests to basic network connectivity through the UAG and the Tunnel configuration. We will do this using a Windows 10 device configured in Workgroup Mode.

Note: With a Workgroup joined machine, the automated Domain Join from Workspace ONE UEM is triggered when the device enrolls into MDM and the device is added to a Smart Group.

Note: For production environments, bulk-registration of devices is possible through either a .CSV or as an automated part of the Workspace ONE Drop Ship Provisioning Online process. Microsoft Autopilot Hybrid Domain join scenarios do not require device pre-registration.

  1. Create the Windows 10 VM and complete OOBE using the Workgroup method.
  2. Login to the Windows 10 VM as the local administrator.
    The requirement to use a local administrator account is because later in this process you will install software that requires local admin rights or the install will fail.
  3. Get the Serial number of the windows 10 device by running the Powershell gwmi win32_bios
  4. From the Windows 10 VM, download the Workspace ONE UEM Intelligent Hub from https://getwsone.com and save the AirwatchAgent.msi to C:\temp. DO NOT start the installer!
  5. On your host machine launch a web browser and navigate to your Workspace ONE UEM Console. The next step is pre-registering the device as follows:
    1. Navigate to Devices > Lifecycle > Enrollment Status
    2. Choose Add > Register a Device
    3. In the User Search Text field type in a Username and choose Search User
    4. Under the Search Results the column named “Action” will have a Select button. Click “Select” and the page will refresh displaying the user’s details
    5. Under the Device section make the following changes:
      1. Platform: Windows Desktop
      2. Check the box next to “Show advanced device information options”
      3. Model: Desktop
      4. Serial Number: Supply the Serial Number for the device
      5. Scroll down to the Messaging section and change the message Type to None
      6. At the top of the page choose the Tab named Tags
      7. Click Add
      8. Select the Tag used to build the Smart Group where the Domain Join is assigned. In my example the tag is named “DSP”
      9. Choose Save
  6. Back on the Windows 10 VM, run the AirwatchAgent.msi and complete the enrollment using the username defined. Here’s a sample command line enrollment to use if you want to automate this step
    (update the command line to your environment).

    msiexec.exe /i c:\temp\AirWatchAgent.msi /quiet ENROLL=Y IMAGE=N SERVER=YourDSServerName.awmdm.com LGName=YourOG USERNAME=YourUsername PASSWORD=YourPasswordNoQuotesRequired DEVICEOWNERSHIPTYPE=CD ASSIGNTOLOGGEDINUSER=N DOWNLOADWSBUNDLE=false /log c:\temp\airwatch.log
  7. The next step is to connect to the ACC Server and open the ACC log so that you can see if the djoin.exe process functions. If you followed Part 1 of this blog series you already have the ACC logs verbosed. The log file is found in:

    C:\VMware\AirWatch\Logs\CloudConnector\CloudConnector.log
  8. Search the CloudConnector.log file for the entry “GetOfflineDomainJoinBlobs”
  9. If djoin.exe worked as expected you will see an entry like this:
2021/04/15 13:33:23.860	UEMCONNECTOR	368741b3-4817-438e-90bf-02c77843c861	[0000000-0000000]   (33) 	Info 	AirWatch.CloudConnector.AccServiceListener.ProcessServiceRequest/1	Processing Query request AirWatch.CloudConnector.DirectoryService.IDirectoryService:GetOfflineDomainJoinBlobs  
2021/04/15 13:33:23.876	UEMCONNECTOR	5c46a87f-8488-48b5-8566-3f2321397cd7	[0000000-0000000]   (33) 	Debug	AirWatch.CloudMessaging.Client.AwcmClient+<SendMessageRetryAsync>d__74.MoveNext	Sending Idle message to AWCM: https://awcm1688.awmdm.com/awcm, MessageId: ce190bd2-511b-4427-b964-a4e06bf838bc, CorrelationId: , OriginId: 458c520c-638d-45f5-95ad-99ba6d9e5233(https://cn1688.awmdm.com/789/acc), DestinationId:   
2021/04/15 13:33:23.892	UEMCONNECTOR	368741b3-4817-438e-90bf-02c77843c861	[0000000-0000000]   (13) 	Debug	AirWatch.CloudConnector.DirectoryService.DirectoryService.ExecuteDjoinCommand	Executing ODJ command - djoin /PROVISION /DOMAIN "lab.aftersixcomputers.com" /MACHINE "AS-7812008" /MACHINEOU "OU=UEMDomainJoin,DC=lab,DC=aftersixcomputers,DC=com" /SAVEFILE "odjBlob-AS-7812008.txt"  
2021/04/15 13:33:25.433	UEMCONNECTOR	368741b3-4817-438e-90bf-02c77843c861	[0000000-0000000]   (13) 	Debug	AirWatch.CloudConnector.DirectoryService.DirectoryService.ExecuteDjoinCommand	Result of ODJ command: 

Provisioning the computer...

Successfully provisioned [AS-7812008] in the domain [lab.aftersixcomputers.com].

Provisioning data was saved successfully to [odjBlob-AS-7812008.txt].



Computer provisioning completed successfully.

The operation completed successfully.

  
2021/04/15 13:33:25.465	UEMCONNECTOR	368741b3-4817-438e-90bf-02c77843c861	[0000000-0000000]   (13) 	Debug	AirWatch.CloudConnector.DirectoryService.DirectoryService+<GetOfflineDomainJoinBlobsAsync>d__68.MoveNext	Returning successful response to ODJ blob request for 1 resources.  
2021/04/15 13:33:25.465	UEMCONNECTOR	368741b3-4817-438e-90bf-02c77843c861	[0000000-0000000]   (13) 	Info 	AirWatch.CloudConnector.AccServiceListener.ProcessServiceRequest/1	Sending reply message  
2021/04/15 13:33:25.465	UEMCONNECTOR

If the CloudConnector.log file does not show the message:


Computer provisioning completed successfully. The operation completed successfully.

then use the ACC log file as your first path to troubleshoot. I don’t have a lot of failures with this process so I’m not sure exactly what else will show up here, but if you have samples to share, please share them in the comments below.

Back on the Windows 10 device, as far as Workspace ONE UEM is concerned the domain join has completed, but the Windows OS doesn’t know this yet. The Windows OS won’t know to switch to the domain model until a reboot. It’s a bit frustrating that there is no visual indication that the device needs a reboot, as the IT Admin you just have to remember this is the next step. Before the reboot, you may be able to confirm the UEM side of the equation worked on the device by opening the Windows registry editor and navigating to HKLM\Software\Airwatch\ where you might see a number of entries associated with the Domain Join including the name of the Domain being joined and the current status. Do not panic if you do not see these registry entries because the entries are removed if the domain operation was successful. I suspect the removal of these registry entries happens as part of the device reboot (more testing required).

If you have not done so already, on the Windows 10 device you should now trigger a device reboot to complete the Domain Join process.

Upon reboot, the expected result is that the Windows login screen will change to reflect the domain login page. You you will see the “Other User” option in the lower left of the login screen. Choose “Other User” and you will see the login screen ask for a username, but you will also notice that it will have your domain name automatically selected.

Did the computer name change? Remember that part of the Domain Join configuration in the Workspace ONE UEM Console also includes renaming the existing computer to whatever standard you defined in the configuration. Login to Windows 10 as a user on your domain and validate from the System Settings that the computer name matches the name defined in the Domain Join Configuration. You’ll also see in the Control Panel > Accounts page that the machine is joined to the domain. If you used the command line installer provided above, what you will not see is Workspace ONE UEM assigning the device to the end user that just logged in. This is on purpose based on the flags used in the command line.

If the domain join fails, there should be a log file that shows why. To troubleshoot the failure, log back into the Windows 10 device as the local administrator and pull up the %windir%\debug\Netsetup.log file.

This completes Test 1 and proves that you properly configured the ACC and the UEM Domain Join configuration and that over Ethernet with line-of-site to the Domain Controller the Domain Join works successfully. Pat yourself on the back and get ready for another test.

If you plan to continue your testing with a VM I recommend resetting the VM using the following process before beginning the next test:

  1. In the UEM Console navigate to Devices > List View >
  2. Select the Win10 VM > Select More Actions > Select Enterprise Wipe
  3. On the Win10 VM ensure the Enterprise Command Succeeds by watching for the Toast Notification
  4. In the UEM Console Navigate to Devices > List View >
    1. Delete the device record for the VM from the console
  5. Navigate to Devices > Lifecycle > Enrollment Status >
    1. Select the device record for the VM
    2. Choose More Actions > Reset Token
  6. On the Windows 10 VM revert to your snapshot or run through an OS reinstall to get you back to a clean OS install.

Test 2: Repeat Test 1 using a staging account

All of the automated Windows onboarding flows use a staging account. It is only after the domain join occurs and the AD user account successfully logs into Windows that UEM flips the staging user over to the end user. For this flip to happen the computer must have line of site to the domain controller during that first Windows 10 logon. Line-of-site to the DC is the function of Workspace ONE Tunnel. We need to make sure the staging user is able to enroll the device so this test is all about the verification of the staging account.

In the Workspace ONE UEM Console, there are two staging accounts created by default for Windows devices. Both are located in the UEM Console at Groups & Settings > All Settings > Devices & Users > Windows > Windows Desktop > Staging & Provisioning.

The first staging account listed is used for command line enrollment and Drop Ship Provisioning (Offline). The second staging account on this page is specific to Drop Ship Provisioning (Online). In this test, using the staging account UPN and password, repeat everything you did in Test 1 except that in Step 7 adjust the command line to use the first staging user listed and change AssigntoLoggedInUser to Y as shown here:

msiexec.exe /i c:\temp\AirWatchAgent.msi /quiet ENROLL=Y IMAGE=N SERVER=YourDSServerName.awmdm.com LGName=YourOG USERNAME=STAGINGUSER PASSWORD=STAGINGUSERPASSWORD DEVICEOWNERSHIPTYPE=CD ASSIGNTOLOGGEDINUSER=Y DOWNLOADWSBUNDLE=false /log c:\temp\airwatch.log

This test may seem redundant, but trust me, there are plenty of things that can still break at this point.

If all goes well, the change you are expected to see in the UEM Console is that the Device is now assigned to the AD End User that logged in.

If something breaks at this point check if the Intelligent Hub is able to complete the switch from staging user to the end user. Logs to assist with this portion of troubleshooting include:

%ProgramData%\AirWatch\UnifiedAgent\Logs\TaskScheduler.log

%ProgramData%\AirWatch\UnifiedAgent\Logs\AWProcessCommands.log

At the end of Test 2, the computer has been renamed, the domain has been joined, the user has logged in and UEM has assigned the device to the end user. We need to add Workspace ONE Tunnel into the mix but before we do there is one more test that is recommended.

Test 3 – We’re going off network!

In this test you need to no longer have line-of-site to your domain controller. Take your test device to your friend’s house, to Starbucks, or fake it via networking magic, but the requirement is that the device needs to have an Internet connection without having line-of-site to your domain controller.

If you are leaving the comfort of your office a few tips before you leave:

  1. Build an automated Windows 10 OS USB key that works with the test laptop you are bringing. Formatting the hard drive is faster than trying to uninstall things and hope that goes cleanly. My favorite automated OS install method is the USB key method Brooks Peppin includes on his blog. You can read about it by clicking here to make your own.
  2. Don’t forget your primary work laptop because you’ll need a second device to troubleshoot with.
  3. Download a copy of the VMware Workspace ONE Provisioning Tool and store it on the USB key. The file can be downloaded from https://my.workspaceone.com .

Arrived at your test location? Let’s continue

  1. Using the bootable USB key build yourself a new Windows 10 OS.
  2. At the first OOBE prompt, boot into Audit Mode
  3. In Part 1 of this blog series, the Per App VPN configuration is set up to establish a VPN connection back to your network whenever you attempt to map a network drive. Map a network drive using a DNS name of a file share on your network. If this works, you’ve validated that the Tunnel app is able to establish the tunnel. If it does not, it’s time to start digging into log files.
    1. The VMware Tunnel logs are located at c:\VMware\AirWatch\Logs\var\log\vmware\tunnel\vpnd\
      1. Start with the tunnel_init.log which should show if the Tunnel is configured and initialization
      2. Next check tunnel.log
  4. The next test is to try and logon to the computer as a different domain user. If your Wifi credentials are saved, log out of Windows 10 and try to login to the computer as a different AD account. Workspace ONE UEM Tunnel will establish the pre-login VPN connection back to the Domain Controller which will allow you to login to the computer as a new user.
  5. If these tests are failing, consult the Workspace ONE UEM Tunnel app and the Diagnostics setting. These logs should provide you with the reason why the Tunnel is unable to establish connectivity. If you can troubleshoot while you are remote

To be continued…

Tags:

Leave Your Comment