Tuesday, 29 October 2019

Setting up a headless linux Blender 2.8 Render Node computer for Farmer Joe Render Farm

Prelude

I'm pleased to say I will be releasing an add-on for Farmer Joe Render Farm which will work with a newly patched version of Farmer Joe for Blender 2.80 and above. This tutorial is me documenting my efforts to create a stable linux host for the render farm so that I can do it again if I need to, and also so those of you, who, like me, have little interest in Linux, can build a box that does what you want and move on to other things.

Farmerjoe is now available here https://github.com/Laurencitow/Farmer-Joe-Render-Farm

Introduction

Spurred on by a recent video showing how much faster linux can be rendering blender files I forgot previous attempts at using linux as a render node OS and had another go.

I must admit after a week of mucking about, getting this to work, I now wholeheartedly appreciate Microsoft Windows: it's stability, forgivingness and ability to heal itself. It's much, much easier to do stuff on windows when you don't know what you are doing than Linux.

That said, here we go. I was setting out to create an install of linux that would:
  • run VNC server so I could control the render node from a main, Windows computer.
  • have fully functioning Nvidia drivers for running blender (AMD? you're on your own - sorry)
  • have auto login
  • be able to share a folder from the render computer that windows can easily see and use
  • be able to see and use a folder shared from windows ... easily
  • run Farmer Joe at login
  • and finally be strong and stable 😁
The computer name and login will be the same, and the passwords will be simple and the same for all passwords on all render computers(root, user, vnc, share etc).So the system is not at all secure.

PLEASE ENSURE THE SECURITY YOU USE SUITS YOUR NEEDS. WHAT IS SHOWN HERE IS A GUIDE ONLY AND I AM NOT LIABLE FOR ANY LOSSES YOU MIGHT SUFFER IF YOUR SECURITY IS INADEQUATE.. sorry, but these things have to be said..


You will need a Dummy HDMI dongle to fake a screen attached (look on eBay)(assuming your graphics card has an HDMI output of course).


"1.4 DDC EDID Dummy Plug Display Emulator VGA 1080 Virtual Display Adapter HDMI"

If you are using a Windows PC to control your render nodes I recommend TightVNC viewer.

For Linux, I tried Ubuntu, Mint, Manjaro, MX and Debian Linux. Got a long way with Manjaro but found it really flakey and unreliable and in the end transferred what I had learnt to Debian: renowned for it's stability.

Debian can exist along side windows but you have to do extra steps if you have UEFI enabled on windows... not covered here so google it. Also there are problems with windows updating afterwards.

I hope I have got all the steps right in this tutorial...

*** ALSO I AM NOT RESPONSIBLE FOR ANY LOSS OF DATA YOU MIGHT SUFFER***
so be careful...

Installing Debian

Download from here https://www.debian.org/CD/http-ftp/#stable and use Rufus to make a bootable USB drive. This tutorial is made with Debian 10.1.0 and used the debian-10.1.0-amd64-netinst.iso file.

Boot from the USB drive on your render computer and follow the instructions. Choose Graphical Install.

You should have chosen your computer/username and simple password.

First thing you need to get right is partitioning the disk. If its the whole disk you are using then it's easy but if you want to delete a partition and use that the menu system is a bit quirky. Using the whole disk use Guided - Use Entire Disk.. Otherwise choose Manual
  • delete the partition
  • select the free space you have created
  • select 'Automatically partition free space' and then 'All files in one partition'
  • 'Finish partitioning and write changes to disk' and the next screen you have to confirm this.
  • no to a proxy (unless you need it)
  • yes to a grub drive and select the drive you system is on if there's more than one disk
During the installation you get a chance to use a more local repository (or server where the debian software updates are stored) Sometimes these local repositories are not as good as the main ones and you can get problems.

When it come software selection page:


  • select XFCE
  • keep Standard System Utilities selected
  • Deselect EVERYTHING ELSE
XFCE is the GUI interface - it's not the best or the prettiest but it does work the way we are going to use VNC.

Your computer name is the same as your login name and all the passwords are simple and the same (at least that's what I've done)

The rest should be straight forward.

First boot and installing things

Choose the 'default panel' option in the dialogue that appears.

We are going to install things two ways: the Synaptic Package Manager and the terminal.

From the Applications button go to Settings/ Synaptic Package Manager  (need password)



Under Settings/Repositories. In the top two ticked selections, in the 'Section(s)' section make sure it says 'main', add 'contrib' and 'non-free'.


Press the Reload button


This allows us do download the proprietary drivers

When done click OK
(if you have problems here it might be worth changing repository)

Click search on the right hand side and put in 'Nvidia Driver'. Scroll down the search results until you come to Nvidia Driver. Right click and select 'Mark for Installation'. (AMD? you're on your own - sorry)



You'll see another screen showing you all the other stuff you are going to install. Say yes. then click apply. It will take a while... its a big file to download and process.

You need to add Cuda files for Cycles (which I forgot to do earlier as I hadn't tested cycles.)


Search for 'cuda'. You need the nvidia-cuda-dev and nvidia-cuda-toolkit. The other items in green will be included with these.


Next while we are in Synaptic Package Manager search for 'TigerVNC'.
 You need these packages.


Finally search for 'Numix' and add the Numix icons. The default restart icon looks like a medieval diagram of a stomach. Numix is more intelligable.


When done open XFCE Terminal from Applications.
Type 'vncserver'
You will be asked for a password. Type the password you are using for everything and confirm it.
It will ask if you want a 'view only' password... I would say 'No' or 'n'

Before we go any further you might want to add 'Restart' option to the 'Action Buttons' (top right hand side) and move the 'Action Buttons' to the left hand side and add the Numix icons (Settings/Appearance/Icons). 

 Right click on the right hand most 'button' marked with your username and select properties. Change to Session Menu to Action Buttons. Untick everything except shutdown, Logout and tick restart. We would get rid of logout you will need it to finish off later. You can move some separators to make it look nice.

Right click on your new Action Buttons and select Panel then Preferences and then Items. You can move things about on the top bar. Make sure your Action Buttons are now on the left hand part of the screen in case of screen shrinkage. If your VNC window is smaller than your actual window you won't loose the restart and shutdown buttons if they are here.


It's a bit clunky and I'll leave you to figure it out but you might end up with something like this.

For the icons go to Applications/Settings/Appearance and choose the Numix icons. Now your restart button doesn't look so stomachy.


You might want to add the handy CPU monitor too!



Important - switch off all power settings.. if the screen goes blank.. it will stay blank
Applications/Settings/Power Manager. When power button pressed to Shutdown (may be useful oneday). Make sure system and display are set to 'Never'.

Installing networking stuff

With terminal still open we can install networking stuff.
For some reason the 'SUDO' command does not do what I'm used to in Debian so we use 'su root'.
Typing 'su root' in the terminal is the same as opening the command prompt as an administrator in Windows. After you've typed it and put in the password you are root administrator.

Installing Samba (Networking):
type -      
apt-get install samba
and type 'y' for yes if prompted.
When asked about WINS select yes.
when done type -
apt-get install samba-client
This looks after linux shares.
apt-get install cifs-utils
For mapping windows shares 


While we are here lets add Gparted (partitioning software) because it's useful:
apt-get install gparted

Finally we are going to set the render computer to auto logon.
Still as su root, type:
mousepad /usr/share/lightdm/lightdm.conf.d/01_debian.conf

in the terminal there will be loads of warnings because they don't like you using mousepad but we don't care. Mousepad text editor will open. There should be some configuration stuff in there already (if there isn't you've made a typo in the above line). Add these lines to the bottom where USERNAME is your login and computer name.

[SeatDefaults]
autologin-user=USERNAME
autologin-user-timeout=0

 Save and exit mousepad and type 'exit' in terminal to exit root.

Launch Synaptic Package Manager again and search for GVFS (this does the searching the windows network niceness)
scroll down - you need to add gvfs-backends, gvfs-fuse and gvfs-bin



Close Synaptic Package Manager

I think now is a great time to reboot. And hopefully your computer should log straight in.


When it boots up I would load this tutoral on the webrowser in your render node so that you can easily copy some of the scripts that follow.

Configuring VNC


Unlike Windows, VNC on linux does not necessarily give you what you see on the screen. Perish the thought! You can have many different VNC connections to many different virtual screens. What is difficult is getting it to show you whats on the actual screen and as we are using Blender this is important that we do. (Just trust me here). So we use a much maligned TigerVNC feature: the X0vncserver. And this shows you the actual screen and (with Debian anyway) behaves well with the Nvidia drivers.

At the moment you will be plugged into a monitor and you probably won't want your VNC window to be as big as the monitors default resolution. X0vncserver will probably crash if you change resolutions with it running so we have to set this first. It's complicated by the fact that your HDMI dummy dongle you will be using to replace the monitor has more resolutions available but more of that later.

We can set the display resolution in Applications/Settings/Display. Let's choose 1440x900

This is the command we will be using for VNC

x0vncserver -display :0 -geometry 1440x900 -rfbport 5900 -passwordfile /home/USERNAME/.vnc/passwd

I have found it necessary to have the resolution in the command otherwise things go wrong. So if you change the resolution again you will have to change it in the x0vncserver command. I have found that when using the Dummy HDMI dongle X0vncserver is less likely to crash if you change resolution.

You can configure x0vncserver as a service but we are going to use the Applications/Settings/'Session and Startup' app to do it because its easier. Go to 'Application Autostart'. Create a new entry called VNC and paste the above command into the command field.

u

If you reboot the render node x0vncserver should be running at startup and you should be able to connect to it from you PC. You will need the password. When you are connected  you can click the disk icon in the top left hand corner of the window and save the connection information to the desktop. It will then ask you to save the password too!

Networking

Having already installed all we need its time to configure.

Shares are shared using the Samba smb.conf file which lives in /etc/samba.

Drives are mounted in the fstab file which is located in the /etc/ directory.
To add a little more detail here, and especially in Debian where extra hard disks are not automatically mounted, we create a folder in the filing system and mount the drive as if it appears in that folder. So in windows we sort of mount a drive to a drive letter, in linux we mount it to a folder name. And the same is true if we are mapping a shared drive from an existing computer.

Mounting drives with fstab

It is important to mount a drive before you share it for obvious reasons. To mount a hard drive within your computer, or another partition on the system drive we need to know it's UUID.

First run Gparted which we installed earlier.
Applications/System/Gparted. Each disk and partition have a code. The first disk is usually SDA and the second SDB and each partition a number so the third partition on the second disk is SDB3.

Note down which one you want to mount.

In terminal run:

ls -l /dev/disk/by-uuid/

The blue number/letter codes are the UUIDs of the partitions on the disk. Identify the one you want from the SDxx code in yellow at the end of each line and copy it (just the blue) to the clip board. For me sda2.


Type 'su root' to become an administrator and type 'mousepad /etc/fstab' . There should be enteries in there already or you have made a mistake.

Make a comment to indicate what you are adding. The Hash key does not work for some reason so you will have to copy it from another line.


I am going to add this line:

UUID=01D57AE18561B120   /media/data    ntfs    defaults    0    0

 where /media/data is a folder I have created in the media directory to map the drive. ntfs is the file system. Linux supports many. Defaults are the options and the two zeros are 'dump' and 'pass' what ever that is. The spaces are tabs.

Under the latest special linux rules. Permanent drives are mapped to /Mnt/ and removable drives are mapped to /Media/.

(If we want to make a folder to map to - say /media/renders we will need to do in terminal as root with mkdir /media/renders ). Any folders/files outside the users home directory are root permissions only and require 'su root'. If you want to see the whole filing system go to the File Manager and look at File System.


If we want to add a windows share from another computer as a perminantly mounted drive then..

//SERVER_IP/SHARENAME   /media/MOUNTDIRECTORY  cifs    auto,nofail,x-system.device-timeout=10,username=USERNAMEOFSERVER,password=PASSWORDOFSERVER,uid=1000,iocharset=uft8   0 0

Spaces inbetween are tabs.

If the share isn't present at boot time then the computer will take a little longer to boot

You can use a hostname as long as the computer can resolve it, you should use static ip addresses really though DHCP computers tend to keep their ip addresses. The UUID here should be unique if you add another share and you can get a UUID generator program.

The username and password here are for the windows box you are sharing from so you may not wish to include the password depending on your situation. There are other more secure ways of storing passwords to shares not covered here.

Save the fstab file. and exit 'su root'. If you have made a mistake then... BOOOOOOMM!!! you'll be pluging that monitor back in again.

Reboot and check to see if the share is mapped to the mount position.

If the share has not been mapped try   mount -a   in terminal as root (this reloads the fstab file). If you get an error you might have to start from scratch again as there is obviously a problem with networking.

If it does map with is command it could be that your computers hardware is slower to get the network stuff up and running and not ready when fstab is run. In this case we need to do something else.

If the FSTAB mapping fails

We can not use mount -a as it only gives us read only permissions.. so

Firstly we're going to create a handy place for some scripts. In the users home directory create a folder named 'scripts' using the File Manager and in it, right click, and create and empty file called startup.sh

then in terminal in as root type chmod +x /home/USERNAME/scripts/startup.sh and this will make the file executable.

edit the file in Mousepad as follows:

!/bin/sh
mount -t cifs -o username=USERNAME,password=PASSWORD,uid=1000,iocharset=utf8, file_mode=0777,dir_mode=0777 //SERVERADDRESS/SHARENAME /media/MOUNTNAME

and save. Now in terminal as root type
crontab -e and if it gives you a choice select 'nano'
cursor down to the bottom of the file and type

@reboot sleep 10 && /home/USERNAME/scripts/startup.sh

type 'Ctrl X' and they 'y' and return to save.

the number after sleep is how long after boot it will try and map the share. I found 5 was too short so settled for 10. You may need to experiment leaving a margin for error.


Sharing things with smb.conf

I wasted many hours trying to come up with a simple way of sharing things from Linux. In the end I used this smb.conf from user Adriano P on Serverfault.com who offered this example.

[global]
workgroup = WORKGROUP
server string = Samba Server %v
netbios name = COMPUTERNAME
security = user
map to guest = bad user
dns proxy = no
#============= Share Definitions =================
[NAME_OF_SHARE]
force user = USERNAME
path = /home/USERNAME/renders
browsable =yes
writable = yes
guest ok = yes
read only = no


This entry in /etc/samba/smb.conf will share the 'renders' folder in the user's home directory WITHOUT asking for any username or password. There were some 'tutts' from other users about security in the thread but this is what I want. If you want security there are plenty of examples on the web. You can add as many shares as you like by copying and renaming the [NAME_OF_SHARE] section.

To set up this file. 'su root' in terminal and type 'mousepad /etc/samba/smb.conf'. There will be an existing file which I suppose you should back up. Then after deleting all the entries from the old file, type or paste the above changing the words in Capitals to your own values and save as smb.conf.

On reboot you should be broadcasting your shares and you should be able to see them in windows.

  Misc

If you want you can remove libreoffice because we don't need office on here.

apt-get remove --purge libreoffice*
apt-get clean
apt-get autoremove

FarmerJoe

To be a render 'node' in a Farmer Joe render farm you will normally use the .sh file as the .pl is there if you want to develope the software further.
 
If you want to run the Farmer Joe perl script use Synaptic Package Manager to install libyaml-tiny-perl and libio-compress-perl.

If you want to compile your own .sh from the perl script you will need, in terminal root, apt install libpar-packer-perl


You will have mounted the share that Farmer Joe resides on as something like /media/renders and the FarmerJoe.sh file or perl script (.pl) will be in the root.

We want to run this, at boot, after the share has been mounted and in a terminal window.

The only way I have found to do this is as follows.

We are going to make another .sh file in our 'scripts' folder (more details in the 'If the FSTAB mapping fails' section). If you haven't made the 'scripts' folder make one in /home/USERNAME/ and in it make an empty file 'fj.sh'.

In terminal as root they chmod +x /home/USERNAME/scripts/fj.sh to make it executable.

Edit the file in Mousepad and add the lines:

!/bin/bash
sleep 15
xterm -e /media/MOUNTNAME/farmerjoexx.sh

or .pl if its a pl script. Here sleep should be a bigger value than the one set to map the share if used. 15 should be fine.

For the master computer where you are running a master, node and webserver you can use the line:

xfce4-terminal -T Master -e "/media/renders/Farmerjoe.pl --master" --tab -T Node -e "/media/renders/Farmerjoe.pl" --tab -T Webserver -e "/media/renders/Farmerjoe.pl --appserver" & 

This will put them in three tabs in same window. xfce4-terminal has a bigger font by default

You can open a web browser on the Farmerjoe webpage with xdg-open http://IPADDRESS:2007

Finally, in 'Applications/Settings/Session and Startup' add a new entry in 'Application Autostart', as we did for VNC, called 'Farmer Joe' and give it the command line:

/home/USERNAME/scripts/fj.sh

  Reventing sessions from being restored

Normally it does not bother us if windows reopen as they were when we log in, but with FarmerJoe we can end up with dead windows from before with new windows from out startup file.

XFCE makes damn sure your session comes back when you reboot so it's hard work stopping it. You have to do three things.

First press logout
and uncheck 'save session for future logins'.

Then in 'Settings/Session and Startup' make sure 'Automatically save session on logout' is unchecked.

then, still in Session and Startup go to the 'Session' tab


and with all other windows closed press clear saved sessions and close 'Sessions and Startup' and cross your fingers :)



And reboot..... and good luck - if the Farmer Joe drive is available and mapped you should be running as a NODE.

 

Finally


If you run into problems I would seriously consider starting again from scratch. It will be faster in the long run.

Conclusion

I was very disheartened in the beginning. I think this is my third attempt at doing this. What appears to be an easy thing to accomplish turns out to be very difficult if you don't have a good knowledge of Linux. And what I have created seems to work well and be robust.












Tuesday, 30 April 2019

USB-C multi adapter problems with the ASUS UX390

I have gorgeous ASUS UX390 laptop which has only one USB-C port and I have had numerous problems getting third party USB-C adapters to work. And now I have hit upon a solution - although it is not a great solution if you want to change adapters a lot.



This one arrived yesterday, a 7 in 1 adapter with 2 USB 2 ports, 1 USBC port, two card slots, an HDMI port and a USB-C charging port.. It looks nice. The box it came in actually has a model number HYD-9822T.

I plugged it in and discovered that every thing worked but the USB ports. The USB2 ports did nothing. The USB-C port made the red LED on a mouse glow but other wise did nothing and no new USB stuff appeared in device manager. The HDMI, card slots and charging port did work.

I mucked about for a long time with drivers and then decided to look if the laptop BIOS needed updating. I rebooted into the BIOS and BEHOLD! the USB ports (even a flash drive i'd left in) were there. I saved and exited and rebooted and everything worked. And continued to do so. 

So there you go..

Thursday, 17 January 2019

How to get motion vector blur in Blender eeVee

(This tutorial was correct for the 2019-01-16 Win 64 build of Blender 2.8)


This is a bit of a dirty workround and was inspired by YOGYOG's alpha tutorials for eeVee.


This workround works by rendering in both Cycles and eeVee and using the vector and depth passes from cycles in the compositor to add motion blur to the eeVee render.


This may sound like a long process but if you turn everything down to the minimum in Cycles the cycles part of the render does not take too long. (though if anyone knows how to make it faster I'd be very glad to hear from you.


So STAGE ONE


Having created your scene in eeVee  you need to duplicate it


 and use 'Link Objects'




We will now have two scenes in the Outliner, which we can rename 'eeVee' and 'Cycles'


STAGE TWO


Select the 'Cycles' scene and we need to make the cycles pass as fast as possible and include the Vector Pass.



Turn everything down to as low as possible even if you don't understand what it does to make Cycles render a quick as possible.

 Select the Vector pass and the depth pass and turn everything else off or low.


STAGE THREE


In the Compositor, using render layers, select the Image from the eeVee scene and the Depth and Vector from the into the Vector Blur node.

And Voila!!


Vector motion blur


Current problems.

1) you must have the Cycles scene selected in Blender or you will just get the eeVee render without motion blur for some reason. (I guess the cycles doesn't render)

2) Blender crashes if you try to change the dimensions of one of the scenes and try and render it.




First project using Blender eeVee

I'm very proud to say I did the CG for this impressive and intriguing installation. I used the Blender eeVee engine while it was still in alpha and had remarkably few problems which is a testament to the fine work of the developers. If you happen to be in Copenhagen between 19th January and 12th March  2019 then do go and see this latest work by Søren Thilo Funder!

THE WATCHERS OF MALHEUR (TWEET TWEET)

 Søren Thilo Funder

 Link to Den Frie Gallery