I got this working as I needed to install BizTalk Adapter to my BizTalk development environment which I have on workgroup.
HKLM\system\currentcontrolset\control\computername\activecomputername
Computername REG_SZ BIZTALKDEMO.domain.net
I changed the value of the computername to include domain name. I started the setup.exe and it started. Clicked to the point where BizTalk adapter is selected. Paused my installation.
Got back to the registry value and removed “.domain.net” from computername.
In dos prompt I connected to the AOS server (which is in domain) with domain password. NET USER \AOS /user:domain\administrator
Now the connection is established between AOS and workgroup server.
Now continued with clicking next. Everything got installed.
Though, not sure how good it will work 
I had/have also the USERDNSDOMAIN system environment variable set, not sure if it is needed. Someone who is struggling with this could test.