Part 2 of 2: Testing & Troubleshooting Workspace ONE UEM Automated Domain Join
April 21, 2021 | by bgarmon
Last updated 5/6/2021
Below is a rough draft, work in progress, updating every few hours.
This is Part 2 of a 2 part guide to configure Workspace ONE UEM’s Domain Join feature in order to allow newly provisioned Windows 10 devices to automatically join Active Directory from any remote network. Part 1 covers the end-to-end configuration, Part 2 covers testing and troubleshooting of the configuration detailed in Part 1. If you missed Part 1, click this link.
For the end users device, Domain Join via Workspace ONE UEM is a one-time operation. There is no retry if things fail, so when you are configuring this solution you need to test every aspect of it before you send it out in the wild. This post is written so that each Test Case builds on the previous Test case. If you skip around you will miss important technical nuggets.
Microsoft’s official name for the process is an “Offline Domain Join (ODJ).” Confusing, I know, because the machine will be connected to the Internet when this happens. The “Offline” portion of the name refers to the Windows 10 computer not being on the same network as the Domain Controller when the domain join completes. The technology that makes ODJ function is Microsoft’s djoin.exe, a program included in the Windows OS since the release of Windows 7.
To explain ODJ the following is a summary of what ODJ does and a brief summary of how it’s done. A more detailed look at each of these steps will be discussed later in this post as part of the testing and troubleshooting process.
|What Happens||How Microsoft does this||How Workspace ONE UEM does this|
|A program named Djoin.exe is run to generate a computer account record in AD for a computer not yet on the network. The result is a text file called a blob.||From a Windows Server on the domain, an IT Administrator runs the command djoin.exe /provision /domain ||The djoin command is run automatically by the Airwatch Cloud Connector (ACC) following an MDM Enrollment. The resulting blob file is saved in the Airwatch Database.|
|The blob file must be copied to the Windows 10 computer not yet on the network||An IT Administrator copies the blob.txt to the computer or adds the contents of the blob.txt to an unattend.xml (which is consumed during OOBE)||After MDM enrollment completes, a command is triggered to extract the blob from the database and transfer the blob through the Airwatch Cloud Messaging service (AWCM) to the Windows 10 computer.|
|Djoin.exe is run on the Windows 10 device to load the blob.txt||IT Admin runs djoin.exe /requestodj /loadfile blob.txt or this same action is triggered by the Windows OS setup process based on the unattend.xml.||In the previous step UEM used a Microsoft CSP to download the blob which is the equivalent of the Microsoft process without requiring the IT admin to run the command line|
|The final step involves rebooting the computer.||An IT Admin reboots computer, or this reboot is automatically triggering during OOBE by the Windows setup process.||If the UEM device is using Factory Provisioning the reboot will happen as part of the final shut-down step during the Sysprep Audit Mode process. A manual reboot will only be required if you are testing this from an existing Workgroup joined machine|
|After the computer reboot, but before the login screen appears, line-of-site to the DC is established by the VPN App. This enables the end user to use their AD account to logon to Windows.||Pre-login VPN solutions like Microsoft Direct Access or a 3rd party VPN solution is used to establish a VPN connection to the DC to complete this step.||Workspace One Tunnel establishes the pre-login VPN to complete this step.|
|The computer is now joined to the domain and the process is complete||<-- Look left.||<-- Look left.|
Here’s a list of the log files used in troubleshooting (in no particular order):
If you just want a cheat sheet to refer to later, here’s the most useful log files for troubleshooting Domain Join, in no particular order. I’ll keep adding to this list. More details about each of these log files will be discussed alongside the test cases below.
|ACC||C:\VMware\AirWatch\Logs\CloudConnector\CloudConnector.log||Needs to be in Debug mode|
|Windows 10 Device||%windir%\debug\Netsetup.log||Records Domain Join status|
|Windows 10 Device||HKLM\Software\Airwatch\||Prior to the Blob being applied to the device, the Windows Registry will show the domain join configuration in several keys including DomainJoinConfiguration, DomainJoinType, DomainName, HasDomainJoined and a few others|
|Windows 10 Device||C:\VMware\AirWatch\Logs\FactoryProvisioning.Service.log||The Factory Provisioning Service records status messages in this log file|
|Windows 10 Device||C:\VMware\AirWatch\Logs\Factory Provisioning Services/Website/VMware.Workspace One.FactoryProvisioning.API.log||This log shows what happens during the runtime of the Factory Provisioning Windows API responsible for receiving the PPKG package generation request|
|Windows 10 Device||C:\VMware\AirWatch\Logs\Factory Provisioning Service\Services\WMware.Workspace One.FactoryProvisioning.Service.exe.config||The configuration file used for Factory Provisioning|
VMware maintains a comprehensive list of all of the Workspace ONE UEM log files for all platforms and services at this link.
The remainder of this blog will run through several test scenarios to help validate you have a working configuration.
Test 0: Successfully enroll a Windows 10 device in UEM.
If this is your first time enrolling a Windows 10 device in UEM step away from this blog and come back when you have successfully enrolled a Windows 10 device in Workspace ONE UEM. This blog assumes Workspace ONE UEM and the Airwatch Cloud Connector (ACC) are functional and that all of the pre-reqs are in place for the basic Workspace ONE UEM Console to device MDM based functionality. Before continuing you should complete the following:
- MDM enroll a Windows 10 device into Workspace ONE UEM
- Configure at least 1 Windows 10 Profile to auto deploy to the device
- Configure at least 1 Windows application to auto deploy to the device. I recommend Google Chrome or Adobe Reader from the Workspace ONE UEM Enterprise App Repository be your initial application test case. They are small, easy installs and illustrate that the basic Device to Server communications are working properly.
With Test 0 out of the way, let’s continue.
Test 1: Windows 10 in a WorkGroup using Ethernet (NO WIFI)
In this test we are removing a few dozen things that can go wrong just to confirm all the configuration you completed in Part 1 is set up properly. While Test 0 focused on basic MDM connectivity, Test 1 is all about the Domain Join Process.
Recommendations for this test:
- This test must be completed on the same network as your Domain Controller.
- Use Ethernet to complete this test.
- Use a virtual machine (VM) with a snapshot taken just after the OS installs to easily try again if something fails.
- Edit the VM’s .VMX file to use the short format. Reference my blog How to build a VM if you need help with this setting.
There are several ways to initiate a Domain Join through Workspace ONE UEM. The most common scenario will be a new device coming from the Dell factory using Drop Ship Provisioning. A second method is through Microsoft Autopilot via a Hybrid Domain Join profile. For the initial test we are not going to leverage either of these processes because we need to first validate basic Microsoft Domain connectivity then extend those tests to basic network connectivity through the UAG and the Tunnel configuration. We are going to start from a pre-installed Windows 10 device that is part of a Workgroup. This configuration has the added bonus of a nice Windows GUI for easily digesting log files and registry keys.
With Workgroup joined machines, the automated Domain Join from Workspace ONE UEM is triggered when the device enrolls into MDM and the device is added to a Smart Group. In Part 1 you defined the Smart Group and linked the Domain Join configuration to the Smart Group. However Part 1 did not include pre-registering devices in the UEM Console. The steps below walk through how to pre-register a single device in UEM, give that device a UEM tag that is used to define the Smart Group, then trigger a domain join.
For production environments, bulk-registration of devices is possible through either a .CSV or as an automated part of the Workspace ONE Drop Ship Provisioning Online process. Microsoft Autopilot Hybrid Domain join scenarios do not require device pre-registration.
Complete the following:
- Create a virtual machine (VM) and install Windows 10 Professional or Enterprise in Workgroup mode. Reference my blog How to build a VM if you need help with this step.
- Boot the Windows 10 VM
- Login to the Windows 10 VM as the local administrator.
The requirement to use a local administrator account is because later in this process you will install software that requires local admin rights or the install will fail.
- Open the Windows Command Prompt and run:
wmic bios get serialnumber
Make a note of the serial number for use later on.
Special Note: At the time of this writing I learned that Microsoft is deprecating wmic from future versions of Windows 10 so this step will be updated as soon as I find an alternative command to gather the serial number.
- From the Windows 10 VM, download the Workspace ONE UEM Intelligent Hub from https://getwsone.com and save the AirwatchAgent.msi to C:\temp. DO NOT start the installer!
- On your host machine launch a web browser and navigate to your Workspace ONE UEM Console. The next step is pre-registering the device as follows:
- Navigate to Devices > Lifecycle > Enrollment Status
- Choose Add > Register a Device
- In the User Search Text field type in a Username and choose Search User
- Under the Search Results the column named “Action” will have a Select button. Click “Select” and the page will refresh displaying the user’s details
- Under the Device section make the following changes:
- Platform: Windows Desktop
- Check the box next to “Show advanced device information options”
- Model: Desktop
- Serial Number: Supply the Serial Number for the device
- Scroll down to the Messaging section and change the message Type to None
- At the top of the page choose the Tab named Tags
- Click Add
- Select the Tag used to build the Smart Group where the Domain Join is assigned. In my example the tag is named “DSP”
- Choose Save
- Back on the Windows 10 VM, run the AirwatchAgent.msi and complete the enrollment using the username defined. Here’s a sample command line enrollment to use if you want to automate this step
(update the command line to your environment).
msiexec.exe /i c:\temp\AirWatchAgent.msi /quiet ENROLL=Y IMAGE=N SERVER=YourDSServerName.awmdm.com LGName=YourOG USERNAME=YourUsername PASSWORD=YourPasswordNoQuotesRequired DEVICEOWNERSHIPTYPE=CD ASSIGNTOLOGGEDINUSER=N DOWNLOADWSBUNDLE=false /log c:\temp\airwatch.log
- The next step is to connect to the ACC Server and open the ACC log so that you can see if the djoin.exe process functions. The log file is found in:
- Search the CloudConnector.log file for the entry “GetOfflineDomainJoinBlobs”
- If djoin.exe worked as expected (and you followed Part 1 to verbose the CloudConnector.log file) you will see an entry like this:
2021/04/15 13:33:23.860 UEMCONNECTOR 368741b3-4817-438e-90bf-02c77843c861 [0000000-0000000] (33) Info AirWatch.CloudConnector.AccServiceListener.ProcessServiceRequest/1 Processing Query request AirWatch.CloudConnector.DirectoryService.IDirectoryService:GetOfflineDomainJoinBlobs 2021/04/15 13:33:23.876 UEMCONNECTOR 5c46a87f-8488-48b5-8566-3f2321397cd7 [0000000-0000000] (33) Debug AirWatch.CloudMessaging.Client.AwcmClient+<SendMessageRetryAsync>d__74.MoveNext Sending Idle message to AWCM: https://awcm1688.awmdm.com/awcm, MessageId: ce190bd2-511b-4427-b964-a4e06bf838bc, CorrelationId: , OriginId: 458c520c-638d-45f5-95ad-99ba6d9e5233(https://cn1688.awmdm.com/789/acc), DestinationId: 2021/04/15 13:33:23.892 UEMCONNECTOR 368741b3-4817-438e-90bf-02c77843c861 [0000000-0000000] (13) Debug AirWatch.CloudConnector.DirectoryService.DirectoryService.ExecuteDjoinCommand Executing ODJ command - djoin /PROVISION /DOMAIN "lab.aftersixcomputers.com" /MACHINE "AS-7812008" /MACHINEOU "OU=UEMDomainJoin,DC=lab,DC=aftersixcomputers,DC=com" /SAVEFILE "odjBlob-AS-7812008.txt" 2021/04/15 13:33:25.433 UEMCONNECTOR 368741b3-4817-438e-90bf-02c77843c861 [0000000-0000000] (13) Debug AirWatch.CloudConnector.DirectoryService.DirectoryService.ExecuteDjoinCommand Result of ODJ command: Provisioning the computer... Successfully provisioned [AS-7812008] in the domain [lab.aftersixcomputers.com]. Provisioning data was saved successfully to [odjBlob-AS-7812008.txt]. Computer provisioning completed successfully. The operation completed successfully. 2021/04/15 13:33:25.465 UEMCONNECTOR 368741b3-4817-438e-90bf-02c77843c861 [0000000-0000000] (13) Debug AirWatch.CloudConnector.DirectoryService.DirectoryService+<GetOfflineDomainJoinBlobsAsync>d__68.MoveNext Returning successful response to ODJ blob request for 1 resources. 2021/04/15 13:33:25.465 UEMCONNECTOR 368741b3-4817-438e-90bf-02c77843c861 [0000000-0000000] (13) Info AirWatch.CloudConnector.AccServiceListener.ProcessServiceRequest/1 Sending reply message 2021/04/15 13:33:25.465 UEMCONNECTOR
If the CloudConnector.log file does not show the message:
Computer provisioning completed successfully. The operation completed successfully.
then use this log file as your first path to troubleshoot. I don’t have a lot of failures with this process so I’m not sure exactly what else will show up here, but if you have samples to share, please share them in the comments below.
Back on the Windows 10 device, as far as Workspace ONE UEM is concerned the domain join has completed, but the Windows OS doesn’t know this yet. The Windows OS won’t know to switch to the domain model until a reboot. It’s a bit frustrating that there is no visual indication that the device needs a reboot, as the IT Admin you just have to remember this is the next step. Before the reboot, you may be able to confirm the UEM side of the equation worked on the device by opening the Windows registry editor and navigating to
HKLM\Software\Airwatch\ where you might see a number of entries associated with the Domain Join including the name of the Domain being joined and the current status. Do not panic if you do not see these registry entries because the entries are removed if the domain operation was successful. I suspect the removal of these registry entries happens as part of the device reboot (more testing required).
If you have not done so already, on the Windows 10 device you should now trigger a device reboot to complete the Domain Join process.
Upon reboot, the expected result is that the Windows login screen will change to reflect the domain login page. You you will see the “Other User” option in the lower left of the login screen. Choose “Other User” and you will see the login screen ask for a username, but you will also notice that it will have your domain name automatically selected.
Did the computer name change? Remember that part of the Domain Join configuration in the Workspace ONE UEM Console also includes renaming the existing computer to whatever standard you defined in the configuration. Login to Windows 10 as a user on your domain and validate from the System Settings that the computer name matches the name defined in the Domain Join Configuration. You’ll also see in the Control Panel > Accounts page that the machine is joined to the domain. If you used the command line installer provided above, what you will not see is Workspace ONE UEM assigning the device to the end user that just logged in. This is on purpose based on the flags used in the command line.
If the domain join fails, there should be a log file that shows why. To troubleshoot the failure, log back into the Windows 10 device as the local administrator and pull up the
This completes Test 1 and proves that you properly configured the ACC and the UEM Domain Join configuration and that over Ethernet with line-of-site to the Domain Controller the Domain Join works successfully. Pat yourself on the back and get ready for another test.
If you plan to continue your testing with a VM I recommend resetting the VM using the following process before beginning the next test:
- In the UEM Console navigate to Devices > List View >
- Select the Win10 VM > Select More Actions > Select Enterprise Wipe
- On the Win10 VM ensure the Enterprise Command Succeeds by watching for the Toast Notification
- In the UEM Console Navigate to Devices > List View >
- Delete the device record for the VM from the console
- Navigate to Devices > Lifecycle > Enrollment Status >
- Select the device record for the VM
- Choose More Actions > Reset Token
- On the Windows 10 VM revert to your snapshot or run through an OS reinstall to get you back to a clean OS install.
Test 2: Test the Staging User account while still on same network as the DC and still on Ethernet
MDM enrollment flows for Microsoft Autopilot and Workspace ONE UEM Drop Ship Provisioning are completed using a Workspace ONE UEM staging account. It’s only after the domain join occurs and the AD user account successfully logs into Windows that UEM flips the staging user over to the end user. For this flip to happen the computer must have line of site to the domain controller during that first Windows 10 logon. Line-of-site to the DC is the function of Workspace ONE Tunnel. We need to make sure the staging user is able to enroll the device so this test is all about the verification of the staging account.
In the Workspace ONE UEM Console, there are two staging accounts created by default for Windows devices. Both are located in the UEM Console at Groups & Settings > All Settings > Devices & Users > Windows > Windows Desktop > Staging & Provisioning.
The first staging account listed is used for command line enrollment and Drop Ship Provisioning (Offline). The second staging account on this page is specific to Drop Ship Provisioning (Online). In this test, using the staging account UPN and password, repeat everything you did in Test 1 except that in Step 7 adjust the command line to use the first staging user listed and change AssigntoLoggedInUser to Y as shown here:
msiexec.exe /i c:\temp\AirWatchAgent.msi /quiet ENROLL=Y IMAGE=N SERVER=YourDSServerName.awmdm.com LGName=YourOG USERNAME=STAGINGUSER PASSWORD=STAGINGUSERPASSWORD DEVICEOWNERSHIPTYPE=CD ASSIGNTOLOGGEDINUSER=Y DOWNLOADWSBUNDLE=false /log c:\temp\airwatch.log
This test may seem redundant, but trust me, there are plenty of things that can still break at this point.
If all goes well, the change you are expected to see in the UEM Console is that the Device is now assigned to the AD End User that logged in.
If something breaks at this point check if the Intelligent Hub is able to complete the switch from staging user to the end user. Logs to assist with this portion of troubleshooting include:
At the end of Test 2, the computer has been renamed, the domain has been joined, the user has logged in and UEM has assigned the device to the end user. We need to add Workspace ONE Tunnel into the mix but before we do there is one more test that is recommended.
Test 3: Leave your home / office / network (or involve a co-worker who is remote)
For the next set of tests you need to have a Windows 10 device that is not on your network so that you can test the Tunnel configuration, and ultimately the entire process. There are all kinds of ways to fake this experience, for example let your Windows 10 device connect to your cell phone (turn off Wifi), or a mobile hotspot, or a separate VLAN that can’t talk to your DC, but the most simple method is to grab a test laptop and leave the office.
Before you leave, have with you an automated Windows 10 OS USB key that works with the test laptop you are bringing. Formatting the computer is a better approach to testing than trying to uninstall things and hope that goes cleanly. My favorite automated OS install method is the USB key method Brooks Peppin includes on his blog. You can read about it here to make your own.
Also before you leave, since Test 3 is a replica of Test 1, go through the entire Test 1 again with your physical device. They key here is that you want the machine domain joined, and you need to successfully log into the Windows device as a domain user at least once while still on your own network so that the Windows OS will cache the login credentials. Trust me, this will make sense when you try it remotely.
Oh one more thing before you leave: don’t forget your primary work laptop because you’ll need a second device to troubleshoot with.
Test 4 – Workspace ONE Tunnel – you are off your network correct?
This test is all about making sure a functional Windows 10 OS with Workspace ONE Tunnel can use Workspace ONE UEM to establish VPN connectivity back to the Domain Controller.
- For this test the test machine must have completed the domain join operation from Test 1 successfully and you should have logged into the computer at least once as the domain user to generate the Windows OS cached credentials.
- Wifi is the expected network connectivity option so keep in mind that unless you are in OOBE you will not be prompted for Wifi when you turn on the computer in this new location until AFTER the Windows login prompt. So boot the laptop, login as the domain user (cached credentials) and do what you need to do to join the Wifi. If the Wifi connection can be remembered by your device, set Windows to automatically connect at login. The point here is that you need to be able to reboot the device and have the Windows OS automatically reconnect the Wifi Connection.
- In Part 1 of this blog, the Per App VPN configuration is set up to establish a VPN connection back to your network whenever you attempt to map a network drive. Map a network drive using a DNS name of a file share on your network. If this works, you’ve validated that the Tunnel app is able to establish the tunnel. If it does not, it’s time to start digging into log files.
- The VMware Tunnel logs are located at c:\VMware\AirWatch\Logs\var\log\vmware\tunnel\vpnd\
- Start with the tunnel_init.log which should show if the Tunnel is configured and initialization
- Next check tunnel.log
- The VMware Tunnel logs are located at c:\VMware\AirWatch\Logs\var\log\vmware\tunnel\vpnd\
- The next test is to try and logon to the computer as a different domain user. If your Wifi credentials are saved, log out of Windows 10 and try to login to the computer as a different AD account. Workspace ONE UEM Tunnel will establish the pre-login VPN connection back to the Domain Controller which will allow you to login to the computer as a new user.
- If these tests are failing, consult the Workspace ONE UEM Tunnel app and the Diagnostics setting. These logs should provide you with the reason why the Tunnel is unable to establish connectivity. If you can troubleshoot while you are remote
Test Z: Workspace ONE Provisioning Tool
There’s a pretty good chance you are using the Domain Join as part of Workspace ONE Drop Ship Provisioning or AutoPilot Hybrid Domain Join scenarios, so you need to make sure that everything works in Windows 10 Audit Mode. VMware has the Workspace ONE Provisioning Tool available for this test case. The file can be downloaded from https://my.workspaceone.com . The testing process will vary based on your decision to use Drop Ship Provisioning Online or Offline but in both use cases, running the Provisioning Tool in Audit Mode will provide a good illustration of the domain join process. The keys to this test are a) Did Workspace ONE Tunnel install successfully and b)
To be continued…