Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Subsystem Linux #37

Closed
selfagency opened this issue Jun 12, 2020 · 41 comments
Closed

Windows Subsystem Linux #37

selfagency opened this issue Jun 12, 2020 · 41 comments

Comments

@selfagency
Copy link

Any way to make this work under Windows Subsystem Linux? When I get to the final installation step before downloading the container, I get the error:

modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.104-microsoft-standard/modules.dep.bin'
modprobe: FATAL: Module kvm not found in directory /lib/modules/4.19.104-microsoft-standard
@SuperSandro2000
Copy link

WSL has no full kernel. You can try WSL2 but don't get your hopes up to heigh.

@selfagency
Copy link
Author

selfagency commented Jun 12, 2020 via email

@sickcodes
Copy link
Owner

Looks possible https://twitter.com/unixterminal/status/1272236470074564610?s=21

@selfagency
Copy link
Author

sweet, he wrote it up https://boxofcables.dev/accelerated-kvm-guests-on-wsl-2/

@taylor-shift
Copy link

taylor-shift commented Jun 17, 2020

(I am updating this post as I find out more about what steps the article was missing)

The article is good but it's missing a lot of key pieces on how to get X11 working in Windows, which leaves the user to have to figure it out themselves and wonder why it won't work...

I had to wrestle very hard to get this project not working but close to working on WSL2. There's a lot of subtleties in the way you have to configure the env vars for X11 for it to work with an Xserver running on Windows as well as dealing with firewall rules. The article fails to mention these with the exception of how to set the DISPLAY variable. It is part of the setup for a guide specifically on WSL2, so I wouldn't say it's out of scope.

For more information, check this SO post:
https://stackoverflow.com/questions/61110603/how-to-set-up-working-x11-forwarding-on-wsl2/61110604#61110604

X11 works, the kvm-ok output looks good, so does catting to see if nested virt is enabled, docker service is running + libvirtd, kvm_intel is loaded and configured with the options in the article. I'm at the last step where you run the docker command to launch the container. My only issue now is I'm getting a Could not access the KVM module whenever I do, even though my user is added to the KVM group and permissions are set correctly or should be. It's not x11 because I can run xterm a couple of other x11 utils.

It also fails to mention that most distros on the Windows store are going to setup for WSL1 Not 2, and that you’ll have to “convert” them to WSL2 before you’re even allowed to los a custom kernel or enable nested kvm.

If anyone has any suggestions or something I should post to get a better answer, I would love some. If I figure this out, I'll def write a script for it. Well...you'd have to have two. One bash, one batch that will each do the necessary steps in Win and nix. What would be the legalities of including a precompiled kernel image with the KVM driver? It would reduce a lot of friction for people who have no idea what a kernel even and increase adoption

@taylor-shift
Copy link

taylor-shift commented Jun 17, 2020

Also, if you follow his guide, any of the commands in the README to run the container fail because we're attaching extra chars to the DISPLAY env var

@sickcodes
Copy link
Owner

Regarding legal: the user has to accept or reject apples terms during install.

@sickcodes
Copy link
Owner

@luisroel91 awesome stuff btw, feel free to create a pull request if you want to get it rolling.

Or if you have the bash output you can just summarize it here ill add it to the dockerfile

I think having all scripts and text inside the dockerfile is awesome, it makes them truly portable. E.g paste into stdin and done

@selfagency
Copy link
Author

can barely contain my excitement tbh

@taylor-shift
Copy link

@luisroel91 awesome stuff btw, feel free to create a pull request if you want to get it rolling.

Or if you have the bash output you can just summarize it here ill add it to the dockerfile

I think having all scripts and text inside the dockerfile is awesome, it makes them truly portable. E.g paste into stdin and done

Apologies for my lack of clarity. Part of the process of getting this working is grabbing the Linux kernel sources specifically altered to run on WSL2, altering config by adding the kvm_intel module + virtio + linux guest support and then recompiling the kernel. What I meant to ask was, what would be the legality of including a precompiled MS Linux kernel and distributing that instead of having the user recompile when they might not even have an idea what a kernel is.

Will do. I have to get passed the kvm access denied (despite adding user to proper groups) I am fully committed to getting this working. I go to a school that's mac based but due to the quarantine, a lot of us don't have access to macs....guess what OS the school's grading system runs its unit tests on :|

@sickcodes
Copy link
Owner

@luisroel91 awesome stuff btw, feel free to create a pull request if you want to get it rolling.

Or if you have the bash output you can just summarize it here ill add it to the dockerfile

I think having all scripts and text inside the dockerfile is awesome, it makes them truly portable. E.g paste into stdin and done

Apologies for my lack of clarity. Part of the process of getting this working is grabbing the Linux kernel sources specifically altered to run on WSL2, altering config by adding the kvm_intel module + virtio + linux guest support and then recompiling the kernel. What I meant to ask was, what would be the legality of including a precompiled MS Linux kernel and distributing that instead of having the user recompile when they might not even have an idea what a kernel is.

Will do. I have to get passed the kvm access denied (despite adding user to proper groups) I am fully committed to getting this working. I go to a school that's mac based but due to the quarantine, a lot of us don't have access to macs....guess what OS the school's grading system runs its unit tests on :|

Awesome :)

Precomplied sounds fine, Linux is open source, you already posted a link to this instructions right? I’m gonna boot into Windows 10 hopefully today and have a go with it

@taylor-shift
Copy link

I am in the process of writing up instructions from 0 up to the point I got (which is trying to run the docker command and getting KVM access denied). The Hayden Bowles article details how to recompile the kernel, yeah

@sickcodes
Copy link
Owner

Keep us posted @luisroel91 :) Or create a new branch if you want or are halfway thru

@jamestalbot3
Copy link

jamestalbot3 commented Aug 7, 2020

@luisroel91 did you ever get it working? i'm at the same point, using the Hayden Bowles kernel. Still getting this error:

Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied

Edit: Apparently I'm an idiot and I missed a command under the troubleshooting section. I'm up and running with no issue (besides slow performance due to old hardware).

@ghasemikasra39
Copy link

Any update regarding running this container on windows 10?

@sickcodes
Copy link
Owner

@luisroel91 did you ever get it working? i'm at the same point, using the Hayden Bowles kernel. Still getting this error:

Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied

Edit: Apparently I'm an idiot and I missed a command under the troubleshooting section. I'm up and running with no issue (besides slow performance due to old hardware).

Hey @jamestalbot3 how slow was it running? WSL2 may have had some performance updates in the last few months with respect to kvm

@jamestalbot3
Copy link

It varied a lot, just wasn't too consistent. I didn't keep using it because I couldn't get imessage and facetime working otherwise I'd upload a recording for you.

@sickcodes
Copy link
Owner

This will fix iMessage: https://github.com/sickcodes/Docker-OSX#serial-numbers

@sickcodes
Copy link
Owner

Looks like @steeve got it working on Hyper-V

https://twitter.com/steeve/status/1297618048946442243?s=20

EgIQihfXkAIRfvS

@sickcodes
Copy link
Owner

Another successful WSL2 implementation by @iyadahmed

https://twitter.com/cgonfire/status/1373479561195892736?s=21

image

@Neurrone
Copy link

Any chance that the steps mentioned in that article on making this work with WSL2 could be integrated into this project?

@ghost
Copy link

ghost commented Mar 22, 2021

Unfortunately as of now, WSL/2 nested virtualization is not very straightforward, while Microsoft actually enabled KVM on recent WSL2 kernels, and they also provided a config for enabling nested virtualization, it doesn't actually work without a workaround

microsoft/WSL#4193

Here's the workaround:
There should be no need to build a custom kernel if you are running a recent WSL2 kernel
https://gist.github.com/steffengy/62a0b5baa124830a4b0fe4334ccc2606

So a way to go was to disable Hyper-V, Windows Hypervisor Platform, etc.. and rely on virtualization provided by VMWare, VirtualBox, QEMU, etc..., you can also install Intel HAXM if you like, the downside is you lose WSL

But anyways the rest is straight forward you need to make sure you enabled virtualization in BIOS, you also need to setup a Linux VM on your favorite software, you also need to turn on nested virtualization in the software

An equally valid way is to use Hyper-V nested virtualization

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

I did the first way, performance is indistinguishable from running a Mac VM directly on Windows or Linux

@sickcodes
Copy link
Owner

Alternatively, you can use an Arch Linux VM and use the commands inside of the Dockerfile to run the same as Docker-OSX but on an Arch host rather than Ubuntu

@ghasemikasra39
Copy link

ghasemikasra39 commented Apr 3, 2021

I installed VirtualBox on my windows and then installed Ubuntu 20.04.2.0 LTS on that.
Then I ran the following command inside the terminal:

sudo docker run --device /dev/kvm --device /dev/snd -v /tmp/X11-unix:/tmp/X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:latest

This is the output I got:

Unable to find image 'sickcodes/docker-osx:latest' locally
latest: Pulling from sickcodes/docker-osx
cf21842e2731: Pull complete 
183cfc865da6: Pull complete 
28ed4b642654: Pull complete 
cffa77372631: Pull complete 
e21c1e0b9d73: Pull complete 
3261ef84de9c: Pull complete 
a27ed0692209: Pull complete 
ce41c626546a: Pull complete 
9b53d6226ccd: Pull complete 
441d664b85f4: Pull complete 
54be6f111230: Pull complete 
dbb7d5734e05: Pull complete 
e668ab6c8fb4: Pull complete 
2f742cabedf0: Pull complete 
e4f3ecb022de: Pull complete 
61967ba1eaaf: Pull complete 
30aeea50d16f: Pull complete 
57f7fbf783e9: Pull complete 
Digest: sha256:d83f324533985e8a21ebbd1a5126d134e41848c5e460afcfed268325bff58a27
Status: Downloaded newer image for sickcodes/docker-osx:latest
docker: Error response from daemon: error gathering device information while adding custom device "/dev/kvm": no such file or directory.
ERRO[0129] error waiting for container: context canceled 

Am I missing something?

Update:
I used this and solved the issue above.

Now when I run the same command:

sudo docker run --device /dev/kvm --device /dev/snd -v /tmp/X11-unix:/tmp/X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:latest

I get this error:

[sudo] password for kasra: 
touch: cannot touch '': No such file or directory
chown: cannot access '': No such file or directory
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 
++ id -u
++ id -g
+ sudo chown 1000:1000 /dev/kvm
++ id -u
++ id -g
+ sudo chown -R 1000:1000 /dev/snd
+ [[ 3 = max ]]
+ [[ 3 = half ]]
++ id -u
++ id -g
+ sudo chown -R 1000:1000 /dev/snd
+ exec qemu-system-x86_64 -m 3000 -cpu Penryn,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -machine q35,accel=kvm:tcg -smp 4,cores=4 -usb -device usb-kbd -device usb-tablet -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -drive if=pflash,format=raw,readonly,file=/home/arch/OSX-KVM/OVMF_CODE.fd -drive if=pflash,format=raw,file=/home/arch/OSX-KVM/OVMF_VARS-1024x768.fd -smbios type=2 -audiodev alsa,id=hda -device ich9-intel-hda -device hda-duplex,audiodev=hda -device ich9-ahci,id=sata -drive id=OpenCoreBoot,if=none,snapshot=on,format=qcow2,file=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2 -device ide-hd,bus=sata.2,drive=OpenCoreBoot -device ide-hd,bus=sata.3,drive=InstallMedia -drive id=InstallMedia,if=none,file=/home/arch/OSX-KVM/BaseSystem.img,format=qcow2 -drive id=MacHDD,if=none,file=/home/arch/OSX-KVM/mac_hdd_ng.img,format=qcow2 -device ide-hd,bus=sata.4,drive=MacHDD -netdev user,id=net0,hostfwd=tcp::10022-:22,hostfwd=tcp::5900-:5900, -device vmxnet3,netdev=net0,id=net0,mac=52:54:00:09:49:17 -monitor stdio -vga vmware
Unable to init server: Could not connect: Connection refused
QEMU 5.2.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]
gtk initialization failed

I also tried this command suggested on the home page:

docker run -it --device /dev/kvm -p 50922:10022 -v /tmp/.X11-unix:/tmp/.X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:latest

I get this output, and this time the vm crashes:
Capture

@ghost
Copy link

ghost commented Apr 3, 2021

I installed VirtualBox on my windows and then installed Ubuntu 20.04.2.0 LTS on that.

Then I ran the following command inside the terminal:

sudo docker run --device /dev/kvm --device /dev/snd -v /tmp/X11-unix:/tmp/X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:latest

This is the output I got:

Unable to find image 'sickcodes/docker-osx:latest' locally

latest: Pulling from sickcodes/docker-osx

cf21842e2731: Pull complete 

183cfc865da6: Pull complete 

28ed4b642654: Pull complete 

cffa77372631: Pull complete 

e21c1e0b9d73: Pull complete 

3261ef84de9c: Pull complete 

a27ed0692209: Pull complete 

ce41c626546a: Pull complete 

9b53d6226ccd: Pull complete 

441d664b85f4: Pull complete 

54be6f111230: Pull complete 

dbb7d5734e05: Pull complete 

e668ab6c8fb4: Pull complete 

2f742cabedf0: Pull complete 

e4f3ecb022de: Pull complete 

61967ba1eaaf: Pull complete 

30aeea50d16f: Pull complete 

57f7fbf783e9: Pull complete 

Digest: sha256:d83f324533985e8a21ebbd1a5126d134e41848c5e460afcfed268325bff58a27

Status: Downloaded newer image for sickcodes/docker-osx:latest

docker: Error response from daemon: error gathering device information while adding custom device "/dev/kvm": no such file or directory.

ERRO[0129] error waiting for container: context canceled 

Am I missing something?

Update:

I used this and solved the issue above.

Now when I run the same command:

sudo docker run --device /dev/kvm --device /dev/snd -v /tmp/X11-unix:/tmp/X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:latest

I get this error:

[sudo] password for kasra: 

touch: cannot touch '': No such file or directory

chown: cannot access '': No such file or directory

ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 

++ id -u

++ id -g

+ sudo chown 1000:1000 /dev/kvm

++ id -u

++ id -g

+ sudo chown -R 1000:1000 /dev/snd

+ [[ 3 = max ]]

+ [[ 3 = half ]]

++ id -u

++ id -g

+ sudo chown -R 1000:1000 /dev/snd

+ exec qemu-system-x86_64 -m 3000 -cpu Penryn,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -machine q35,accel=kvm:tcg -smp 4,cores=4 -usb -device usb-kbd -device usb-tablet -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -drive if=pflash,format=raw,readonly,file=/home/arch/OSX-KVM/OVMF_CODE.fd -drive if=pflash,format=raw,file=/home/arch/OSX-KVM/OVMF_VARS-1024x768.fd -smbios type=2 -audiodev alsa,id=hda -device ich9-intel-hda -device hda-duplex,audiodev=hda -device ich9-ahci,id=sata -drive id=OpenCoreBoot,if=none,snapshot=on,format=qcow2,file=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2 -device ide-hd,bus=sata.2,drive=OpenCoreBoot -device ide-hd,bus=sata.3,drive=InstallMedia -drive id=InstallMedia,if=none,file=/home/arch/OSX-KVM/BaseSystem.img,format=qcow2 -drive id=MacHDD,if=none,file=/home/arch/OSX-KVM/mac_hdd_ng.img,format=qcow2 -device ide-hd,bus=sata.4,drive=MacHDD -netdev user,id=net0,hostfwd=tcp::10022-:22,hostfwd=tcp::5900-:5900, -device vmxnet3,netdev=net0,id=net0,mac=52:54:00:09:49:17 -monitor stdio -vga vmware

Unable to init server: Could not connect: Connection refused

QEMU 5.2.0 monitor - type 'help' for more information

(qemu) qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]

qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]

qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]

qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaveopt [bit 0]

gtk initialization failed


I also tried this command suggested on the home page:

docker run -it --device /dev/kvm -p 50922:10022 -v /tmp/.X11-unix:/tmp/.X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:latest

I get this output, and this time the vm crashes:

Capture

Edit: Hmm the problem is related to audio check your vm settings and your guest configuration

Not sure where exactly is the problem but make sure to enable Nested Virtualization
image

@ghasemikasra39
Copy link

@iyadahmed
Capture
This option was enabled for me. I saw an image a few posts above hat someone ran this successfully on VMware, mint. Does it matter which virtualization tool/distribution of Linux we should uses?

@ghost
Copy link

ghost commented Apr 3, 2021

@iyadahmed

Capture

This option was enabled for me. I saw an image a few posts above hat someone ran this successfully on VMware, mint. Does it matter which virtualization tool/distribution of Linux we should uses?

Yeah it was on Mint under VMWare, Mint was straight forward, make sure to update Mint though, before trying to enable KVM and install docker

to get nested virtualisation working properly you should disable all features related to Hyper-V and WSL

then make sure to enable all Virtualisation features in BIOS settings

then setup a virtual machine with Linux and get KVM working

@ghasemikasra39
Copy link

@iyadahmed
I tried Linux Mint on VirtualBox, getting exactly the same error as Ubuntu. I can confirm that both WSL and Hyper-v is disabled in my windows. I will try Mint on VMware as my last try.

@ghost
Copy link

ghost commented Apr 4, 2021

@iyadahmed

I tried Linux Mint on VirtualBox, getting exactly the same error as Ubuntu. I can confirm that both WSL and Hyper-v is disabled in my windows. I will try Mint on VMware as my last try.

Btw can you give system info?, the very first error says that cpu doesn't support eax

@ghasemikasra39
Copy link

@iyadahmed
same Error on VMWare.
Yes sure, which info do you need?

@ghost
Copy link

ghost commented Apr 4, 2021

@iyadahmed

same Error on VMWare.

Yes sure, which info do you need?

Just usual, CPU, Host system what build number and so

@ghasemikasra39
Copy link

@iyadahmed
Update:
I used this and ran:

# UBUNTU DEBIAN
sudo apt install x11-xserver-utils

# then run
xhost +

now when I run sudo docker run --privileged -v /tmp/.X11-unix:/tmp/.X11-unix sickcodes/docker-osx , I get this window:

image

But running the commands in my previous posts still gave me the same errors.

@ghasemikasra39
Copy link

@iyadahmed
same Error on VMWare.
Yes sure, which info do you need?

Just usual, CPU, Host system what build number and so

image

@ghasemikasra39
Copy link

@iyadahmed

Update:
I could make all the commands working by adding sudo at the beginning and using the option --privileged.
I tried both Big Sur and Catalina but both show this page:

Capture1

Then I continue with the second option. But in both cases I get the message This dist is locked:

Capture3
Capture2

Is it the expected behaviour? I mean, am I supposed to re-install to be able to use it?

@ghost
Copy link

ghost commented Apr 6, 2021

@iyadahmed

Update:
I could make all the commands working by adding sudo at the beginning and using the option --privileged.
I tried both Big Sur and Catalina but both show this page:

Capture1

Then I continue with the second option. But in both cases I get the message This dist is locked:

Capture3
Capture2

Is it the expected behaviour? I mean, am I supposed to re-install to be able to use it?

EDIT: AFAIK, the error this is not expected behaviouri f you use the auto build
Awesome,
to reuse the docker image see this https://github.com/sickcodes/Docker-OSX#start-the-same-container-later-persistent-disk
no need to reinstall it keeps your installation

@lokinmodar
Copy link

I was able to run this just fine:
docker run -it \ --device /dev/kvm \ -p 50922:10022 \ -v /tmp/.X11-unix:/tmp/.X11-unix \ -e "DISPLAY=${DISPLAY:-:0.0}" \ sickcodes/docker-osx:latest (i was even able to update Catalina to its latest version)

But this docker run -it \ --device /dev/kvm \ -p 50922:10022 \ -v /tmp/.X11-unix:/tmp/.X11-unix \ -e "DISPLAY=${DISPLAY:-:0.0}" \ sickcodes/docker-osx:big-sur

is giving me a headache. Here are the steps i follow:
1 - Run the command and wait for QEMU output.
2 - Launch macOS installer
3 - Disk utility -> erase -> exit Disk utility
4 - macOS installer -> reinstall macOS big-sur
5 - choose the drive I just erased
6 - wait for installation to finish
7 - installation ends, system reboots as expected and then:

image

Where is the disk it installed the system? If i launch the installation again it just retuns to openCore screen... Is there something I am missing?

@sickcodes
Copy link
Owner

sickcodes commented May 1, 2021

Base System is the one :)

Installer

@lokinmodar
Copy link

lokinmodar commented Jun 4, 2021

Base System is the one :)

I had some issues with my system and could not test this. After doing it all again and using the additional options e GENERATE_UNIQUE=true -e MASTER_PLIST_URL=https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist I still can't boot to big sur even when entering Base System. It shows me the menu to Reinstall MacOS and Disk Utillity as if it isn't installed...

With Catalina this does not happen. It shows the 200GB disk and I can boot the system...

@sickcodes
Copy link
Owner

Let me pull latest and check if anything's wrong

@1ycx
Copy link

1ycx commented Jun 16, 2021

wow, that's a lot of information. If Is* there a compiled blog of sorts for WSLg ?

@selfagency
Copy link
Author

closing this for staleness and i wanna clean up my issues tab

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants