NEO Z83-4: Cannot get to boot from USB Stick

I try to install a Linux on this box, but do not get very far.

F7 or F8 do not seem to make any difference during boot up, I always end up in Windows (keyboard is a perfectly fine generic USB type). I can enter the BIOS though, where I can change boot order. So I change the first entry from Windows Boot Loader to USB Key and upon reboot - I get the UEFI shell. If I exit it, (by typing exit), the USB's boot menu seems to load (instantly with Ubuntu, after quite a while with Manjaro), and it is reactive, however when I chose something (boot Ubuntu) the system seems to die and stays like that, until I unplug it (Power button not reacting).

I am on BIOS ver 2.17.1249.

Any ideas on what I could try?
Thanks @notrui2,

I did that already, but thanks for the heads up!
My question stays the same though ...
Most of these threads seem to part from the fact that it is obvious how to start the installation of a different OS, which is not that is happening here.
Is there maybe somewhere some authoritative documentation about how one should go about this? I know about F7 only from these forum, but I assume, somewhere these things have been written down ...