https://lg.io/feed.xml

Updated Dec 17, 2016: Removed mention of the Azure Preview (since it’s now GA), linked to easier method to disable monitors, updated driver link

Playing Overwatch at 60FPS, 2560x1600, everything on Epic quality, and streaming from the cloud – not too shabby!

It’s no secret that I love the concept of not just streaming AAA game titles from the cloud, but playing them live from any computer – especially on the underpowered laptops I usually use for work. I’ve done it before using Amazon’s EC2 (and written a full article for how to do it), but this time, things are a little different. Microsoft’s Azure was first to give access to NVIDIA’s new M60 GPUs, completely new beasts that really set a whole new bar for framerate and image quality. They’re based on the newer Maxwell architecture, versus the Kepler cards we’ve used in the past. Hopefully one day we’ll get the fancy new Pascal cards :)

Before going through this article, I strongly recommend you at least skim my EC2 Gaming article from before so you can grasp some of the concepts we’ll be doing here. Basically it’ll come down to this: we’re going to launch an Azure GPU instance, configure it for ultra-low latency streaming, and actually properly play Overwatch, a first-person shooter, from a server over a thousand miles away!

And yes, it seems I always need to repeat myself when writing these articles: the latency is just fine, the resolution is amazing, it can be very cost-effective (if you don’t forget the machine on), and all very practical for those of you obsessed about minimalism (like me). If you’re the hardcore gamer-type who’s knee-jerk reaction regardless of anything is “omg i dont trust anything that doesnt have a Zalman watercooled CPU fan or has a ‘hardware-offloaded’ networking card” or if you’re in some sort of sunk-costs situation of already having spent loads of money on a gaming rig, I strongly recommend not reading this article – it’ll only further infuriate you. :)

Note that in this article I assume you’re on a computer similar to mine (a Macbook laptop running MacOS). That’s not a requirement to do this stuff though. You could be on Linux or Windows as a client, though some of the client-side tooling changes around a bit. You’ll figure it out!

It’s not the cheapest thing out there, but if you’ll just be playing here and there it can be. Like in the past, the majority of the bill will probably come from bandwidth usage from the server (especially if youre doing 30+ MBit/s). Also remember, there’s $0 upfront cost here. This contrasts dramatically to the thousands of dollars you’d end up paying for a similarly spec-ed gaming rig.

• NV6 Server: $1.35/hr
• Bandwidth at 10MBit/s: $0.41/hr
• HD storage: $0.003/hr

Total at 10MBit/s: $1.76/hr
Total at 30Mbit/s: $2.58/hr (recommended tho)

Note: playing via the cloud doesn’t make you suck any less at gaming ;)

1. Go to the Azure portal and use the following instructions to create a new NV6 type machine (has NVIDIA’s M60 GPU). The K80 machines won’t work for this since they don’t virtualize the display adapter we need.
   1. Enter the Azure Portal
   2. On the left side select ‘Virtual machines’ and click ‘Add’
      
   3. Select ‘Windows Server’ then ‘Windows Server 2016 Datacenter’
   4. When prompted for the deployment model, select ‘Resource Manager’ from the dropdown and click the ‘Create’ button
   5. Enter a name and some credentials for the machine. Make sure though that the ‘VM disk type’ is ‘HDD’ and the Location is ‘South Central US’ (this is the only location they support right now)
      
   6. When prompted to pick the size (type) of machine select ‘View all’ up top and click on the ‘NV6’ machine type. Try not to panic about the cost, we’ll only be using it for a few hours while gaming, not 24/7. :)
      
   7. On the Settings screen, most defaults are fine, but do change the Network Security Group to ‘None’ and turn off Diagnostics
      
   8. Confirm everything on the Summary screen and the instance will launch. Note that it takes a few minutes until your machine is Running and it’ll have an IP address.
      
2. Install Microsoft Remote Desktop on your Mac if you haven’t already. Set up the machine with the username/password you specified when creating the instance and the IP address listed on Azure. Additionally:
   • Select ‘Connect to admin session’ on the machine’s properties
   • Unselect ‘Start session in full screen’
   • Select a resolution of 1024x768
   • For ‘Sound’ select ‘Don’t play sound’ (it’s unnecessary)
3. Once connected, you’ll need to create a new user account that isn’t the account you specified earlier. This is necessary for some driver changes and auto-login steps you’ll be doing later.
   1. Right click on the Start button and select ‘Control Panel’
   2. Then select ‘User Accounts’ and then ‘User Accounts’ again, then ‘Manage another account’
   3. Then click to ‘Add a user account’ and create a new user. This will be the accout you’ll use going forward.
   4. Once created, click on it on the list
   5. Click on ‘Change the account type’, select ‘Administrator’, and confirm
   6. Close the window and disconnect from the session
      
4. Set up this new user on Microsoft Remote Desktop on the client side and re-login with the new account. You won’t need the old account anymore, so feel free to remove settings about it from Remote Desktop.

1. Run Windows Update on it
   • Click the Start button and select ‘Settings’, then select ‘Update & Security’ and run Windows Update there
   • It can take a while for this to complete, and even might appear to be stuck at a certain percentage, but all is ok, just keep waiting
   • Once it completes, restart the machine if necessary. It’ll probably take several minutes until the machine will be back up and running.
2. Turn on ICMP Ping requests with the Windows Firewall. It’s useful for debugging.
   • Click the Start button and type ‘Windows Fire’ and select ‘Windows firewall with Advanced Security’
   • Select ‘Inbound rules’ on the left side and enable the rule ‘File and Printer Sharing (Echo Request - ICMPv4-In)’
     
   • You should now be able to ping your server machine from your laptop to figure out what the roundtrip time is
3. Disable Windows Defender
   • Click the Start button and type in ‘Windows Defender’
   • Click Settings and turn off ‘Real-time protection’
4. Turn off auto Defragmenting drives (yes this is a real problem that’ll happen in the middle of your games)
   • Click the start button and type in ‘Defragment’ and select ‘Defragment and Optimize drives’
   • Click on ‘Change settings’ for ‘Optimization schedule’
   • Uncheck ‘Run on a schedule’
5. Turn on performance flags, plus turn off the boot timeout (since we’re on a server)
   • Right click the Start button and select ‘System’, then select ‘Advanced system settings’ on the left side
   • In the Performance section, click on ‘Settings…’
   • In the Visual Effects tab, select ‘Adjust for best performance’
   • In the Advanced tab, select ‘Programs’ and click Ok
   • Back to the ‘System Properties’ dialog, click the ‘Settings…’ button in the ‘Startup and Recovery’ section
   • Uncheck ‘Time to display list of operating systems’ since you never get to see that anyways
   • Finally, select ‘(none)’ for the ‘Write debugging information’ dropdown
6. Remove unnecessary scheduled tasks
   • Right click the start button and select ‘Computer Management’
   • Expand ‘Task Scheduler’ on the left side
   • Now for each of the following in ‘Task Scheduler Library > Microsoft > Windows’:
     • Disable all tasks in ‘Chkdsk’
     • Disable all tasks in ‘Diagnosis’
     • Disable all tasks in ‘DiskCleanup’
     • Disable all tasks in ‘Maintenance’
     • Disable all tasks in ‘SystemRestore’
     • Disable all tasks in ‘Windows Defender’
7. Disable File and Printer sharing and SMB client on the network
   • Right click the start menu and select ‘Network Connections’
   • Double click on ‘Ethernet’. Try not to be distracted by the alleged ‘40.0 Gbps’ connection the machine has. :)
   • Click on Properties and uncheck ‘Client for Microsoft Networks’ and ‘File and Printer Sharing for Microsoft Networks’
8. Disable unnecessary services
   • Click the Start button and type in ‘Services’
   • For each of the following services, double click on it and select ‘Startup Type > Disabled’ and Stop the service
     • Server
     • Print spooler
9. Turn on auto-login into Windows. This is useful to make it faster to start playing games after rebooting.
   • Press the Start button and type in ‘netplwiz’
   • Click on your username
   • Uncheck ‘Users must enter a user name and password to use this computer’
   • Then click the OK button and enter your user’s password
     
10. Enable file extension showing and system/hidden file showing
    • Open up the File Explorer and click ‘View’ up top
    • Put a check in ‘File name extensions’, geez, how is hiding extensions ok?

1. Make the desktop background black (or a non-picture) to minorly speed up Remote Desktop sessions
   • Right click the desktop and select ‘Personalize’
   • For ‘Background’ select ‘Solid color’
   • Then pick the black color below (or whatever color your heart desires)
2. Turn off notifications of things you already told it you don’t want on
   • Right click the Start button, select ‘Control Panel’, click ‘System and Security’, and finally click ‘Security and Maintenance’
   • On the left side select ‘Change Security and Maintenance settings’
   • Uncheck all the things that make you feel superior to other people and don’t need to be reminded about
3. Turn on always showing icons in the system tray
   • Right click on the taskbar and select ‘Properties’, then select ‘Notification area’ > ‘Customize…’
   • Click on ‘Select which icons appear on the taskbar’
   • Turn on ‘Always show all icons in the notification area’
4. Turn off combining taskbar icons and some other taskbar cleanups
   • Right click on the taskbar and uncheck ‘Show Task View button’ and ‘Show touch keyboard button’
   • Right click on the taskbar and select ‘Properties’, then select ‘Taskbar buttons’ > ‘Never combine’
5. Fix the time zone because you’re not some crazy sysadmin
   • Right click the time in the systray and select ‘Adjust date/time’
   • Select ‘Time zone’ to be where you are
6. Make the Server Manager not start up every time the computer does
   • Open the Server Manager
   • Go to the ‘Manage’ menu and select ‘Server Manager Properties’
   • Put a check in ‘Do not start Server Manager automatically at logon’

1. You’ll notice that if you pull up the Device Manager that the driver will be missing for the M60 video card.
   

2. Get the latest NVIDIA Tesla M60 drivers from here. Sometimes there are other versions of the drivers mentioned on Azure forum, so go there if you have incompatibility problems.

3. Do the regular Express install and reboot when completed.
   

4. Disable the default display adapter in Windows or else games will choose the wrong one
   • Right click the Start button and select ‘Device Manager’
     
   • Expand ‘Display adapters’, right click on ‘Microsoft Hyper-V Video’ and select ‘Disable’. Though notice the fancy M60 card on there now too!
     
5. You’re using the Remote Desktop service in Windows to administer everything. The problem is that whenever you disconnect from the server, Remote Desktop will lock the screen (and you’ll need to CTRL-ALT-DEL to be able to use things). This is no good if we’ll want games to be running.
   • On the desktop, right click and create a new shortcut and have it run the command tscon 1 /dest:console and name it ‘Disconnect’. NOTE: sometimes tscon 1 isn’t the correct one, try tscon 2. you’ll know that it works if you get disconnected when running this after the remaining steps below.
     
   • Right click on it, select Properties and click the ‘Advanced’ button
   • Put a checkmark in the ‘Run as administrator’ option
     
   • In the next step and going forward, we’ll be using this link when needing to log out of Remote Desktop but keep the unlocked desktop available for other applications
6. As this video card introduces possibility of another monitor being attached to the server, some games will get confused which one to use. Follow the instructions on this post (thanks /r/openstack!) to disable all monitors but the NVIDIA one.

1. Enable audio in Windows Server (the service is off by default)
   • Click the Start button and type in ‘Services’
   • Double click on ‘Windows Audio’ and select ‘Automatic’ for ‘Startup type’
2. There is no soundcard on the VM. A free product, VB-CABLE handles this quite well.
   • Download it from here
   • Extract the zip and run the ‘VBCABLE_Setup_x64.exe’ installer as Administrator
   • After installing it, Windows will still say there’s no installed audio device, but don’t worry, Steam will pick it up. You can also see it in the Device Manager.

1. Azure (like EC2 and others) is still very fresh with IPv6, and it’s current implementation is not enough for use here. We’ll need to force disable it in Windows otherwise some software might try to do a IPv6-over-IPv4 tunnel which ruins everything (Zerotier for examples tries to do this).
   • Open up an Administrator PowerShell (click Start, right click on ‘Windows PowerShell’ and ‘Run as Administrator’)

   • Then run the following:

       Set-Net6to4Configuration -State disabled
       Set-NetTeredoConfiguration -Type disabled
       Set-NetIsatapConfiguration -State disabled
     

2. For Steam In-Home Streaming to work properly, you’ll need to set up a VPN. I strongly recommend ZeroTier for this since it’s the best at ensuring a peer-to-peer connection between the machines and not re-routing through some other server who knows where. Oh and don’t worry, for what you’ll be using it for, it’s free. They’re also a super ethical company and opensource large amounts of their core software.
   • So, go to Zerotier’s website and create an account there and create a network
   • When configuring the network, there are a few settings necessary:
     • I’d recommend setting ‘Access Control’ to ‘None’. It makes configuration easier, just don’t distribute the network id.
     • For ‘IPv4 Auto-Assign’ select to ‘Auto-Assign from Range’ and pick an IP range for the VPN. You’ll notice the Managed Routes above will be updated.
     • For ‘Ethernet Frame Types’ unselect IPv6, it’s unnecessary since Steam does not support it right now. Make sure IPv4 is on though.
     • Write down that Network ID
       
   • Now, go install ZeroTier on the server. The Windows download link is here. When installing, select to approve everything, including the network adapter.
   • At the end, it’ll come up with the ZeroTier One window. Put your Network ID at the bottom and click Join. Click Yes on any Windows prompts.
     
   • Repeat the same installation of ZeroTier again except on your laptop. The Mac download link is here. Have it join the same network.
     
   • On the ZeroTier website under your network, you should now be able to see the IP addresses of the two machines.
     
   • To test the tunnel works, have one ping the other by its ‘Managed IP’. The ping times should be around the same as if you pinged the Physical IP of the other machine. Ideally your ping times are way lower than the ones in this screenshot… I’m looking forward to when the Azure GPU machines are closer to where I live!
     
3. To optimize the network traffic and packet sizes Steam In-Home Streaming uses you might need to adjust MTU on the server-side. I figured out this number by trial and error and using WireShark on the client side to see when what is supposed to be one streaming packet ended up with one big packet and one tiny one.

   • Open up an Administrator PowerShell and run the following on the server. ‘Ethernet 2’ here is the name of the ZeroTier adapter.

       netsh interface ipv4 show subinterfaces
       netsh interface ipv4 set subinterface "Ethernet 2" mtu=1410 store=persistent
     

4. I’d recommend rebooting now, just because of all the quirky network changes. Note that ZeroTier will always reconnect on both systems, so if you ever need to make changes, use ZeroTier One on both systems to configure network changes.

1. Install Steam (yes, even though we eventually want to play Overwatch, we need Steam’s In-Home Streaming to work)
   • Click on the Internet Explorer button on the task bar and go to ‘https://steampowered.com’ and download and install it from there
   • Configure the server Steam to:
     • Save your password and auto-login (if you have Steam Guard on, you’ll need to put in the code they email you)
     • Account > Beta participation, and select ‘Steam Beta Update’. Then restart Steam.
     • Friends > Automatically sign into Friends when I start Steam
     • In-Game > In-game FPS counter > Top-left (optional, but I like it since this is the raw FPS on the machine)
     • In-Home Streaming > Enable streaming
     • In-Home Streaming > Advanced Host Options, and Check only the following:
       • ‘Adjust resolution to improve performance’
       • ‘Enable hardware encoding’
       • ‘Enable hardware encoding on NVIDIA GPU’
       • ‘Prioritize network traffic’
       • Note that everything else should be unchecked. I’ve messed with NvFBC, but what Steam does for full-screen capture seems to be superior (most of the time). Of course, you can mess with it later if you’re trying to debug a game. If you try to use NvFBC, please see my previous EC2 Gaming article for instructions on how to get that set up (you need to run NVFBCEnable).
         
     • Interface > Favorite window > Library
     • Interface > Notify me about additions or changes […], uncheck it
2. Now lets install Overwatch:
   1. Download and install Blizzard’s upsell-galore Desktop App for Battle.net launcher thingy
   2. Log into it with your Battle.net account
   3. Select Overwatch on the left side and install it
   4. Now wait until it finishes downloading/installing (takes a few minutes, but surprisingly fast!)

3. Overwatch in particular has some issues quitting properly after being launched from Steam. Fortunately a tool, bnetlauncher was built to help properly start/stop the game. Basically, keep the official Battle.net client running, but in the next step use this launcher. Feel free to put it anywhere, including on your Desktop.

4. To stream games we use Steam’s In-Home Streaming functionality since it’s the most mature game-streaming tech out there (that I’m aware of). The good news is that you can add non-Steam games (like Overwatch/bnetlauncher) and it works great too.
   • In Steam, click ‘Add A Game…’ (in the bottom left corner of the window) and select ‘Add a Non-Steam Game…’
   • Overwatch should be on the list, but don’t add it that way. Instead, use the ‘Browse…’ button and then select the ‘bnetlauncher’ app from the previous step. Then click the ‘Add Selected Programs’ button.
   • After its added, in the Games Library, right click on ‘bnetlauncher’ and select Properties. Set the name to be ‘Overwatch’
   • As part of the Target, add ‘ Pro’ after the double-quotes
     
5. On your computer (in Steam), configure it as such:
   • In-Home Streaming > Enable streaming
   • In-Home Streaming > Client options > Beautiful
   • In-Home Streaming > Client options > Advanced client options > Limit bandwidth to > 30 Mbits/s (do NOT set unlimited, it does not work)
   • In-Home Streaming > Client options > Advanced client options > Limit resolution to > Display resolution
   • In-Home Streaming > Client options > Advanced client options > Enable hardware decoding
   • In-Home Streaming > Client options > Advanced client options > Display performance information
     

1. Make sure the Steam client is running on your laptop
2. On ther server, fully exit Steam and re-open it. After a few seconds, you should see the In-Home Streaming popup saying the two machines see eachother.
3. Use the ‘Disconnect’ shortcut from earlier to close Remote Desktop
4. On the Steam client, click the Play button and hopefully the game will load!
5. In Overwatch, go to Options, and set the graphics quality to Epic, hit the Apply button and restart the game.
6. Happy gaming!!

• Don’t forget to Stop the Azure instance when you’re done playing. It’ll be very expensive if you don’t. You can stop the VM without losing your config. If you want to delete everything, make sure to delete the Resource Group in Azure

So pretty :)

Steam In-Home Streaming details after pressing F6 (bottom-left of screen)

• Here you can see all sorts of interesting stats: we’re doing 2560x1600 at 60fps via NVENC
• Latency is around 82ms. Once Azure spreads this tech around to all their datacenters, things should really go down in terms of latency.
• We’re doing a solid 21MBits, though in action it’ll easily hit 30MBits

Steam In-Home Streaming also displays a rolling latency graph (bottom-right of screen)

• Dark blue line is time it took to encode the H264 on the server side
• Light blue line is time it took to transfer to the client side
• Red line is decoding time and display latency

Overwatch’s FPS debugging output (top-left of screen)

Steam’s non-In-Home-Streaming FPS counter (top-right of screen)

Thanks for reading! Hopefully this helps out. My day-to-day job is at Envoy, a startup in San Francisco building pretty cool products for rethinking typically outdated processes in the workplace. We’re hiring fun engineers with a passion for building re-thought tech. Check out our jobs page! And definitely mention if you’re into streaming games :)

Shoutout to @Karan_Batta for helping me get started with the GPU machines there!

Updated Nov 27, 2018: Updated credentials for new unifi versions (uses a new username)
Updated Jul 31, 2016: Moved away from letsencrypt-auto and switched to certbot, updated the auto-renewal script, and changed the suggested cron time to weekly. Also made mention that mFi series has been discontinued. Finally, fixed the install instructions for Unifi Video.

Wow – I got myself a free signed SSL cert for my WiFi controller!

Lets Encrypt recently was released and is definitely super interesting. They basically issue SSL Certificates for free. SSL Certs typically would cost hundreds of dollars per domain and even more for Wildcard certificates. It’s insane, it’s essentially an entire industry predecated around artificial pricing for something that is essentially zero cost to generate and maintain. Not to mention holding back security and encryption on the web since not just anyone can afford hundreds of dollars a year for a cert. This entire industry is holding back progress at a massive scale, so we’re going to fix that :)

With Lets Encrypt, this is all free now. As cost is no longer a problem, we can encrypt other communication like router config landing pages and other services. No need for self-signed certificates that your browser freaks out about when navigating to. Now we can have real certs!

As I’m a big fan of Ubiquiti products, I’m going to show some examples in this article for how to use Lets Encrypt to generate certificates that are compatible with the mFi automation stuff, Ubiquiti’s Unifi wifi controller and their Unifi Video series for surveillance. Ubiquiti, as they’re an enterprise company, [imo wrongly] expects companies to want to host the backing controller software for these devices on-site. We’re going to host them on EC2 though, so we don’t need to manage servers or have people tripping over power cables. Since we’re on the internet though, we need proper SSL to prevent the NSA and all their shenanigans.

Ubiquiti’s mFi, Unifi wireless and Unifi Video micro camera. They all need a hosted controller.

Note: This article will be specific to configuring Ubiquiti’s services, but the Lets Encrypt instructions are the same regardless of what kind of service you might want.

Note: Also heads up that the mFi line of products has currently been discontinued by Ubiquiti. Instructions here should still work though.

So, lets get started!

Lets Encrypt is somewhat unusual in the way it works. Essentially yes, they give out free certificates, but they need to be renewed every 3 months. Not sure why this is, but my guess is it has something to do with that they’re free. As such, on whatever server you’re using to host your service, you’ll need to have a cronjob that runs Lets Encrypt on that server. Otherwise your cert will expire, and you’re going to have a bad time.

Go to AWS EC2 and create an instance. For Ubiquiti products, I’ve found that even one t2.micro machine can run all three of the servers we’ll be deal with in this artice. If configured right, you might even be able to stay in the AWS Free Tier

• Type: t2.micro
• OS: Ubuntu Server 18.04 LTS
• Storage: ~30GB (maybe more if you’ll be doing a lot of video recording)
• Ports to open: At least port 443 for the Lets Encrypt verification, but depending on the Ubiquiti service (all TCP unless otherwise specified):
  • mFi: 6080, 6443
  • Unifi: 8081, 8080, 8443, 8880, 8843, 3478 (UDP)
  • Unifi Video: 6666, 7080, 7443, 7445, 7446, 7447
• Remember to log into your server using ssh ubuntu@IP-ADDRESS

You’ll need to add Ubiquiti’s repositories so you can use apt-get to easily install the right services.

• mFi:

  echo 'deb http://dl.ubnt.com/mfi/distros/deb/ubuntu ubuntu ubiquiti' | sudo tee -a /etc/apt/sources.list.d/100-ubnt.list
  sudo apt-key adv --keyserver keyserver.ubuntu.com --recv C0A52C50
  sudo apt-get update
  sudo apt-get install mfi
  

• Unifi:

  # Unifi has been a pain to install, so people have created a script to install everything, see:
  # https://community.ubnt.com/t5/UniFi-Wireless/UniFi-Installation-Scripts-UniFi-Easy-Update-Scripts-Ubuntu-18/td-p/2375150
  

• Unifi Video:

  # visit https://community.ubnt.com/t5/UniFi-Video-Blog/bg-p/blog_airVision for the latest version instructions.
  # here's version 3.3, though there may be a newer version by now:
  wget http://dl.ubnt.com/firmwares/unifi-video/3.3.0/unifi-video_3.3.0~Debian7_amd64.deb
  sudo dpkg -i unifi-video_3.3.0~Debian7_amd64.deb
  

After the installation of the packages you want, you should be able to go to the https endpoint to see the page. It’ll be: https://<your-aws-ip>:6443 for mFi, https://<your-aws-ip>:8443 for Unifi, and https://<your-aws-ip>:7443 for Unifi Video.

Problem is, you’re using a self-signed certificate, so your web browser will complain. Next, we’re going to use Lets Encrypt to get a real certificate.

Self-signed cert’s not so hot. ;(

Lets install Lets Encrypt now. Reminder that this needs to be done on this server, not your local machine. We’ll be using certbot and essentially the instructions there.

wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto
./certbot-auto

That last line will configure certbot and also install some dependencies.

Now, using certbot, we generate the signed certificate. So lets run the wizard:

./certbot-auto certonly

Select option 2 (to use a temporary webserver), then enter your email (so you get alerts if things go wrong), agree to the agreement, then finally type in your domain name (along with the subdomain). If everything went well you should get a Congratulations message.

The Ubiquiti services are Java-based and they use the Java Keystore as a way of storing the private keys and certificates. We first need to generate a PKCS #12 certificate from the raw ones we just received:

sudo openssl pkcs12 -export -inkey /etc/letsencrypt/live/mysubdomain.mydomain.com/privkey.pem -in /etc/letsencrypt/live/mysubdomain.mydomain.com/fullchain.pem -out /home/ubuntu/cert.p12 -name unifi -password pass:temppass

Again, don’t forget to replace mysubdomain.mydomain.com with your domain name. Everything else can remain as-is.

Now for each service you’ll need to load the PKCS #12 certificate into its own keystore.

• mFi:

  sudo keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/mfi/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias unifi -noprompt
  

• Unifi:

  sudo keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/unifi/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias unifi -noprompt
  

• Unifi Video:

  sudo keytool -importkeystore -deststorepass ubiquiti -destkeypass ubiquiti -destkeystore /var/lib/unifi-video/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias unifi -noprompt
  

Basically all that’s different is the keystore location of the service, and the password Ubiquiti uses to protect it.

Finally, delete the PKCS #12 files (since they’ve already been imported), and restart the services (as appropriate)

sudo rm /home/ubuntu/cert.p12
sudo /etc/init.d/mfi restart
sudo /etc/init.d/unifi restart
sudo /etc/init.d/unifi-video restart

That’s basically it! You should go to those same urls as before and you’ll now hopefully have your browser not complaining. :)

The browser likes it!

As mentioned before, Lets Encrypt certificates only last 3 months. As such, we’ll need to get this machine to attempt to renew the certificates probably weekly and then place the new certs back into services. It’s essentially doing parts C and D on a scheduled job using cron. Weekly can seem like a lot, but it’ll fail fast if no renewal is necessary.

Create a new file /home/ubuntu/renew_lets_encrypt_cert.sh and customize it according to what you used in Parts C and D. No sudo needed since cron will run it automatically as a super user. Use full paths to files. Here’s an example:

# Get the certificate from LetsEncrypt
/home/ubuntu/certbot-auto renew --quiet --no-self-upgrade

# Convert cert to PKCS #12 format
openssl pkcs12 -export -inkey /etc/letsencrypt/live/mysubdomain.mydomain.com/privkey.pem -in /etc/letsencrypt/live/mysubdomain.mydomain.com/fullchain.pem -out /home/ubuntu/cert.p12 -name unifi -password pass:temppass

# Load it into the java keystore that UBNT understands
keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/mfi/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias unifi -noprompt
keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /var/lib/unifi/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias unifi -noprompt
keytool -importkeystore -deststorepass ubiquiti -destkeypass ubiquiti -destkeystore /var/lib/unifi-video/keystore -srckeystore /home/ubuntu/cert.p12 -srcstoretype PKCS12 -srcstorepass temppass -alias unifi -noprompt

# Clean up and use new cert
rm /home/ubuntu/cert.p12
/etc/init.d/mfi restart
/etc/init.d/unifi restart
/etc/init.d/unifi-video restart

Make sure to make this executable:

sudo chmod +x /home/ubuntu/renew_lets_encrypt_cert.sh

Lets start modifying the crontab file with sudo crontab -e and put the following at the bottom:

1 1 * * 1 /home/ubuntu/renew_lets_encrypt_cert.sh

This will schedule the certificate renewal every week on Monday at 1:01am

And now you’re really done! You have a free SSL certificate by Lets Encrypt being automatically renewed and assigned to the different services on a monthly basis.

Thanks for reading!

I really wouldn’t mind moving from Dropbox to S3, Google Cloud Storage, Azure Storage, or any other provider

Recently at work I’ve been using the fancy new 4K iMac, yet when I’m at home or out-and-about, I use my equivalently-sexy MacBook. I do the same stuff on both computers though, it just so happens that one is way faster and has a bigger screen, and the other is slower though portable and much more practical for my lifestyle. They both need access to the same files.

Typically to solve this problem you’d use Dropbox or a similar service. But one thing I strongly dislike about specifically Dropbox is that it really wants you to sync everything to every computer. To me this seems extremely wasteful as I might not want the storage overhead of syncing my personal photos onto my work computer. Ideally files could be uploaded to a service, but only downloaded on demand. I understand Dropbox has Selective Sync, but that’s not what I want. I should be able to mount all my stuff onto a Raspberry Pi for example.

What got me curious though is: which of the major cloud services out there actually focused on performance? I have 100mbit+ symmetric connections at both home and work (thx MonkeyBrains and Mimosa! pic here), so I figured that I should optimize for low-local-storage requirements yet be ok with high-bandwidth usage.

 Amazon AWS S3
 Google Cloud Storage
 Microsoft Windows Azure Blob Storage

I would have wanted to also test Backblaze’s B2 thingy, but I haven’t been invited into their Beta program yet. ;(

For all three services I want to test 3 real-world things:

1. Large file: I created a 100mb file using mkfile 100m ~/Desktop/100m.
2. Small files: I found a git repo on my machine and copied the .git directory.
3. Many instructions: After the above git directory was synced, I removed all the files from the cloud provider.

Also, the point of this is that it’ll be used from the consumer side. So this was all run on my own laptop and to the service’s storage datacenter which was closest to San Francisco, CA.

• Google Cloud Storage (gcsfuse)

  gcsfuse --limit-bytes-per-sec "-1" --limit-ops-per-sec "-1" --debug_gcs lg-storage cloud
  time cp ~/Desktop/100m ~/cloud/100m
  time cp -r ~/proj/myrepo/.git/ ~/cloud/myrepogit
  time rm -rf ~/cloud/myrepogit
  

• Amazon AWS S3 (Transmit app)

  For AWS I used Panic’s Transmit app to mount the disk, along with their Transmit Disk patch. I would have tried to use s3fs-fuse to mount via Fuse, but since I’m running El Capitan on my laptop, the new signed-only kext changes have completely broken my ability to use it. I used the same cp and rm commands as for Google Cloud Storage.

• Microsoft Windows Azure Blob Storage

  No tools exist to mount this as a disk on my mac :(

Using the sync SDKs, although easier, bring the downside that we’re now syncing all files/folders, very contrary to the whole premise of this exercise. That said, it could be useful for comparisons.

• Google Cloud Storage (gsutil)

  time gsutil -m rsync -d -r ~/Desktop gs://lg-storage
  time gsutil -m rsync -d -r ~/proj/myrepo/.git gs://lg-storage/myrepogit
  time gsutil -m rsync -d -r ~/empty gs://lg-storage/myrepogit
  

• Amazon AWS S3 (aws-cli)

  time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete ~/Desktop s3://lg-storage
  time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete ~/proj/myrepo/.git s3://lg-storage/myrepogit
  time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete ~/empty/ s3://lg-storage/myrepogit
  

• Microsoft Windows Azure Blob Storage (blobxfer.py)

  time blobxfer --storageaccountkey "..." lgfilestorage lgfilestorage ~/Desktop/100m
  time blobxfer --storageaccountkey "..." lgfilestorage lgfilestorage ~/proj/myrepo/.git
  

  Unfortunately, blobxfer.py provides no way to delete files from Blob Storage. I’ve created an issue for this here.

There are a variety of things to note in these results:

• Each request to any of the services is super slow. When debugging, I noticed an average time of 100ms-300ms just to get the metadata of a file that exists there. When mounting, a lot of these stat-like requests are done, slowing things down dramatically. Weird that the latency is so high – you’d think it should only be a few miliseconds at most.

• Using any kind of Disk Mounting is a bad idea. FWIW, I expected this, but I didn’t expect it to be that bad. I guess it makes sense though, the storage provider has no idea which files you’re going to access so it can’t parallelize anything for you. When using the sync SDK method, they usually launch tons of parallel requests since then it knows all the files in a folder that need to be transferred.

• Azure’s tooling for non-Windows is extremely poor. Right now there’s literally one guy, Fred, who’s building out tooling for us non-Windows folks. Because of the immaturity of what’s actually available, I couldn’t run many tests, including file deletion from Azure and also any form of mounting of Azure Storage to my local filesystem. I feel like this could be really hurting Microsoft’s adoption amongst people and businesses that use real operating systems for their automations.

• Using the services’ sync SDKs was surprisingly quite fast. They’re all on even footing (unlike the disk mounting method), with Azure being the fastest on average. It was quite noticeable actually.

These results aren’t good for what I originally set out to do. Yes it’s great I have a possibly-faster-than-Dropbox solution, but I don’t want to sync everything everywhere. The disk mounting method, though possible, is still extremely slow and makes it almost impractical for what I am trying to do.

Perhaps a tool that someone should build is something that uses the sync SDK method to transfer files, but something else for directory listings. When it appears that the user is touching a file in a folder (say a git repo), the tool would pull the whole repo locally in parallel. Or maybe only if you’re accessing 5+ files in a folder will it download the whole thing. I think there’s space for some sort of smart caching tool that also auto-deletes infrequently used files depending on free disk space (or a specified quota). That way what’s stored on my computer is minimal, but I still have access to everything, everywhere (assuming I have a good internet connection).

As for the data collected, when comparing the three providers, Azure seems to be the fastest, but hardest to use. Google is quite fast too, with better tooling. And AWS S3 is also quite fast, but probably with the best tooling (there’s even iOS apps to browse your S3 Buckets!).

Overall, FML, I don’t know what to do. Please email me, trivex@gmail.com, if you have ideas.

Playing Witcher 3, a GPU-intensive game on a 2015 fanless Macbook

I’ve written about using EC2 as a gaming rig in the past. After spending some time and getting all sorts of feedback from many people, I’m re-writing the article from before, except with all the latest and greatest optimizations to really make things better. Now we’re using things like NvFBC for graphics card H.264 encoding, using the built-in SSD for better hard drive performance, plus getting rid of things like VNC. I’ve also made the OpenVPN instructions easier to follow.

This is the perfect solution for you fanatics that love to play AAA games but are stuck with one of the new fan-less Macbooks (or similarly slow machines). This is a pretty awesome alternative to building out a new gaming PC. Just make sure you have a good internet connection (ideally 30mbit+ plus <50ms ping to the closest Amazon datacenter). This article will assume you’re on a Mac client, though it should work on Linux or Windows with some minor changes in the client tools.

TLDR: I have made an AMI for you to use such that you don’t need to go through this lengthy step-by-step. Note you become less of a badass if you use the AMI instead of doing it yourself ;). See instructions below.

Believe it or not, it’s actually not that expensive to play games this way. Although you could potentially save moneys by streaming all your games, cost savings isn’t really the primary purpose. Craziness is, of course. :)

• $0.11/hr Spot instance of a g2.2xlarge
• $0.41/hr Streaming 10mbit at $0.09/GB
• $0.01/hr EBS storage for 35GB OS drive ($3.50/mo)

You’re looking at $0.53/hr to play games this way. Not too bad. That’s around 1850 hours of gaming for the cost of a $1000 gaming PC. Note that prices vary for different datacenters.

1. On AWS, create a new EC2 instance. Use defaults everywhere except as mentioned below:
   • Base image should be Microsoft Windows Server 2012 R2 Base (since Windows still has all the best games)
     
   • Use a g2.2xlarge instance (to get an NVIDIA GRID K520 graphics card). Though a larger instance does exist in some regions, I have been unsuccessful in taking advantage of the multiple vGPUs w/ SLI. Plus it’s four times the cost.
   • Use a Spot instance, it’s significantly cheaper (fractions the regular cost) than regular instances. Note cost will vary depending on region. I usually bid a penny more.
   • For the storage step, leave everything at the defaults. This will provision a 35GB EBS drive where your OS will live, and a 65GB SSD-backed instance-store (which is super fast and where the games will go). This instance-store will be available as a Z:\ drive.
   • For the Security Group, I’d recommend creating one that has 3 rules: one that allows All TCP, one that allows All UDP and one that allows All ICMP. Source should be from Anywhere for all 3. Yes, its not maximum security, but with the VPNs you’ll be setting up, it’ll be very convenient.
   • Finally, for the Key Pair, create a new one since you’ll need one for Windows (to retrieve the Administrator password later)
     
     

2. Once your machine has spun up, get the Windows password using your private key. Connect via Microsoft Remote Desktop and add the details in there. Also make sure to select Connect to admin session to avoid GPU detection troubles. Note that your first connection might have a black screen for about a minute as it creates your user profile.

3. Before we go too crazy:
   • Disable the IE Enhanced Security Configuration (so you can use IE)
   • Enable auto-login
   • Disable the windows firewall
   • Enable showing filename extensions
     
     

4. Download and install version 347.88 of the GeForce GTX TITAN X driver package for Windows 8.1. Only the GeForce package contains the latest drivers for the GRID cards. If you get an error when installing the drivers that says it couldn’t detect a GeForce card, you’re not in Remote Desktop as an admin session. Reboot when asked. Note that the latest version of the drivers sometimes cause Windows not to be able to restart.

5. The GRID cards have an optimization Steam can use which can offload the H.264 video encoding to the GPU. We need to enable this though. Download NvFBCEnable from here and run the following (using a Command Prompt): NvFBCEnable.exe -enable -noreset. Reboot again.

6. In order to make games actually use the video card, you’ll need to completely remove the default display driver. Open up Device Manager, and a) disable the Microsoft Basic Display Adapter, b) uninstall it and c) run the following in a Command Prompt. Run each command undividually. Reboot afterwards.

    takeown /f C:\Windows\System32\Drivers\BasicDisplay.sys
    cacls C:\Windows\System32\Drivers\BasicDisplay.sys /G Administrator:F
    del C:\Windows\System32\Drivers\BasicDisplay.sys
   

7. Start the Windows Audio Service as per the instructions here. As we’re also on an EC2 machine, there is no soundcard, so install Razer Surround to get a virtual soundcard, AND you get fancy 5.1 simulation! Note that there’s no need to create/login to a Razer ID account.

8. Download OpenVPN from here. Select the 64-bit Vista installer and when installing make sure to select to select to install all components. After installing, open a Command Prompt and run the following:

    cd C:\Program Files\OpenVPN\easy-rsa
    init-config
    vars
    clean-all
    build-ca			(leave all answers blank)
    build-key-server server	(leave all answers blank except Common Name "server", yes to Sign and yes to Commit)
    build-key client		(leave all answers blank except Common Name "client", yes to Sign and yes to Commit)
    build-dh
    robocopy keys ../config ca.crt dh1024.pem server.crt server.key
   

   Then:

   1. Download my server config and place it in the C:\Program Files\OpenVPN\config directory.
   2. Use the Microsoft Remote Desktop file sharing feature to download the following files from the C:\Program Files\OpenVPN\easy-rsa\keys directory: ca.crt, client.crt, and client.key onto your client computer.
   3. Combine those files along with my client config, up.sh and down.sh. The up and down are used to forward multicast from your client to the server. Note you’ll need WireShark installed on the mac (with pcap support) in order to make multicast more reliable with the up/down scripts.
   4. Edit the client.ovpn file to have your server’s IP in it.
   5. Install TunnelBlick on your Mac. Rename the folder with all the files from above (on your client) to have a .tblk extension and double click on it. TunnelBlick will install the VPN.
   6. Finally, start the OpenVPN service on the server (you should also set it to start Automatically), and connect to it from the client. Don’t bother with the OpenVPN GUI stuff.

   
   phewf That was difficult, though you’re pretty badass for getting it done! Note alternatively you can use ZeroTier (make sure to enable IP addressing on their website w/ an IP range) and not do any of the above OpenVPN craziness. ;) Also alternatively to ZeroTier is Hamachi.

9. Create a new file, C:\startup.bat which contains md Z:\SteamLibrary. The idea is that when the computer boots fresh, it will ensure that the Z drive is initialized properly for Steam to use as a game storage drive. Add this script via gpedit.msc to your startup. See instructions here.
   

10. Install Steam and set the following settings:
    • Make it remember your username/password so it can auto-login every time
    • In the Steam preferences, create and add Z:\SteamLibrary to your Downloads > Steam Library Folders.
    • I recommend you turn off Automatic Sign-in of Friends (since this server will always be logged in) in Friends, and turn off the promo dialog in Interface (at the bottom).
    • Enable hardware encoding at In-Home Streaming > Advanced Host Options > Enable Hardware Encoding
      

    
    On your mac, make sure you have Steam installed, but change In-Home Streaming > Enable Hardware Decoding. Similar settings to above might also be applicable. One other tip, I recommend setting the bandwidth limit to 20 or 30mbit specifically, ‘Automatic’ isn’t good at guessing.
    

11. Once all is set up, run the following to log out of the Remote Desktop session and not lock the screen (so games can start): tscon %sessionname% /dest:console. I suggest creating a shortcut on the desktop for this.

1. Make sure the image you created above is ready. I recommend the gaming-up.sh and gaming-down.sh scripts mentioned below to load/save state via an AMI.

2. With TunnelBlick on your client, connect to the VPN and start Steam on your client. It should detect the remote machine.
   
   

3. Select a game to install (make sure to install to the Z drive), and after installing and click the Stream button!
   Server
   
   
   Client
   
   
   Client streaming Deus Ex: Human Revolution
   
   
   Closer view of stats
   
   

• Because these machines have a lot of RAM, i’d suggest setting the Pagefile to something small like 16MB. See how here. The smaller your C:\ drive, the faster the AMI creation will be.
• Often times games will crash when trying to start. It’s usually because they’re missing certain libraries. Make sure to install .NET 3.5, XInput/Xaudio libraries, and the Media Foundation feature package (from Server Manager). Also force run Windows Update and apply everything (including Optional packages).
• I wouldn’t suggest attempting to write scripts to backup your Z:\ drive to C:\ when shutting down your machine. The games download quite quickly on a fresh boot from Steam. The C:\ drive and EBS is quite slow.
• To make it easy to start/stop the gaming instance I’ve made gaming-up.sh and gaming-down.sh. gaming-down.sh will terminate the instance after creating an AMI, and gaming-up.sh will restore this AMI. You’ll need jq installed. Thanks to Matt Marino, here are instructions for running this on Windows)
• Some games don’t have Steam Cloud. I’d recommend installing Dropbox and syncing the My Documents directory with it. That way you won’t lose your save game files between terminations.

There are two ways to see how your streaming performance is doing.

1. The first is have the Display performance information option enabled in your client’s Steam In-Home Streaming settings. Then when in-game, press F6 (Fn+F6 on a Mac) and information will be displayed at the bottom of the screen.
   
   
   
   • Make sure that the Encoder is always NVFBC. If it’s not this will significantly slow things down since the H.264 encoding of the video will be done on the CPU (slower than the hardware H.264 encoding on the GRID GPU). If you see any form of x264 here, it’s using CPU encoding.
   • Same goes for making sure you’re not doing software decoding. VideoToolbox is good if that’s what you see.
   • The Incoming bitrate will be high, so make sure nobody else is using your internet!
   • Packet Loss needs to be extremely low. Often times MTU problems will bring this up in the double digits, making the game unplayable. Do a Google search for how to fix MTU problems.
   • The graph on the right side is important. The colors basically mean:
     • Dark Blue: amount of time to generate/encode the frame. If this is past 10ms, turn down your game resolution and settings.
     • Light Blue: amount of time to transfer the frame over the network. This will be the crazy one if you don’t have a spectacular connection or one of your roommates decides to start Bittorrent, etc. Try to keep this one as low to the Dark Blue line as possible, but much of it is out of your control.
     • Red: amount of time to decode and display the H.264 video. You can’t do much here except keep the resolution down and make sure hardware decoding is on.
       
       

2. The second, more detailed way to look at streaming performance is to press F8 while gaming. Note this will likely crash your Mac Client in the process. An example of the output (found at C:\Program Files (x86)\Steam\logs\streaming_log.txt on the server):

    {
    	"GameNameID"             "The Witcher 3: Wild Hunt"
    	"TimeSubmitted"          "1435438519"
    	"ResolutionX"            "1280"
    	"ResolutionY"            "800"
    	"CaptureDescriptionID"   "Desktop NVFBC H264"
    	"DecoderDescriptionID"   "VideoToolbox hardware decoding"
    	"BandwidthLimit"         "15000"
    	"FramerateLimit"         "0"
    	"SlowGamePercent"        "0"
    	"SlowCapturePercent"     "0"
    	"SlowConvertPercent"     "0"
    	"SlowEncodePercent"      "0"
    	"SlowNetworkPercent"     "0"
    	"SlowDecodePercent"      "0"
    	"SlowDisplayPercent"     "0"
    	"AvgClientBitrate"       "21.21160888671875"
    	"StdDevClientBitrate"    "28.340831756591797"
    	"AvgServerBitrate"       "10105.5224609375"
    	"StdDevServerBitrate"    "0"
    	"AvgLinkBandwidth"       "104074.671875"
    	"AvgPingMS"              "8.4620447158813477"
    	"StdDevPingMS"           "1.4712700843811035"
    	"AvgCaptureMS"           "4.6132941246032715"
    	"StdDevCaptureMS"        "2.260094165802002"
    	"AvgConvertMS"           "0"
    	"StdDevConvertMS"        "0"
    	"AvgEncodeMS"            "4.6132822036743164"
    	"StdDevEncodeMS"         "2.2601768970489502"
    	"AvgNetworkMS"           "6.5326347351074219"
    	"StdDevNetworkMS"        "2.3294456005096436"
    	"AvgDecodeMS"            "2.4401805400848389"
    	"StdDevDecodeMS"         "3.9411675930023193"
    	"AvgDisplayMS"           "6.3233218193054199"
    	"StdDevDisplayMS"        "6.5956048965454102"
    	"AvgFrameMS"             "27.07377815246582"
    	"StdDevFrameMS"          "14.234905242919922"
    	"AvgFPS"                 "60.287784576416016"
    	"StdDevFPS"              "9.2482481002807617"
    	"BigPicture"             "0"
    	"KeyboardMouseInput"     "1"
    	"GameControllerInput"    "0"
    	"SteamControllerInput"   "0"
    }
   

   See more information about this file in the Steam In-Home Streaming Steam Group.

• If when you start streaming a game, Steam says the “Screen is locked”, you’ll need to make sure you close your Remote Desktop session with tscon %sessionname% /dest:console.
• If you can only see part of the game view, it’s likely it launched as a window and it’s being improperly cropped by Steam. Make sure your game is in fullscreen mode (usually done in the game’s options).
• If the game is extremely choppy, check the Packet Loss percentage by pressing F6. If it’s any higher than 1% or 2% (especially if it’s around 50%), you’re likely having an MTU problem. Try using ZeroTier or Hamachi as the VPN instead of OpenVPN.
• If the streaming is really glitchy and has bad compression, on the client Steam change the bitrate from Unlimited/Automatic to 20mbit or 30mbit.
• If the computers can’t see each other, on your Steam client, go to the InHome Streaming settings and disable and enable streaming. That will send the UDP Multicast packet which should be forwarded over the VPN and get the server to reveal itself. Also, check your VPN connection in general.
• If when you start Steam on your Mac you get a Streaming error, follow the instructions here to fix the executable.
• If the game just suddenly hangs, Cmd+Tab out of the game, and go back to Steam on the host. Press the “Stream” button to start the game again. This will terminate the current streaming session, and restart one that uses the same remote process. This should resume your game. (Thanks Jérémie Lumbroso!)

Lets face it, following all of the stuff above is a long, tedious process. Though it’s actually quite interesting how everything works, I’m sure you just want to get on the latest GTA pronto. As such I’ve made an AMI with everything above, including the optimizations.

1. On AWS, create a new EC2 instance. Use the instructions on the first step, except select the ec2gaming Community AMI. Don’t worry about the Key Pair. FYI the AMIs are:

   ami-017dbf6a (us-east) ami-8735c5c3 (us-west-1) ami-dfefeeef (us-west-2) ami-20175557 (eu-west-1) ami-e47842f9 (eu-central-1) ami-60cd6260 (ap-northeast-1) ami-8c5b5bde (ap-southeast-1) ami-4d9eda77 (ap-southeast-2)

2. Follow step 2 except the password for the instance is rRmbgYum8g. Once you log in using Microsoft Remote Desktop, you’ll be asked to change the Administrator password. Change it to something. If you’re on Windows, you’ll need to use a Mac or Linux or a mobile client to reset the password since there’s a bug in the Windows Remote Desktop client.

3. Install TunnelBlick on your Mac. Download the VPN configuration from here and unzip it. In the client.ovpn file, change YOUR_HOSTNAME_HERE to your instance’s IP/hostname. Rename this folder to ec2gaming.tblk and double click on it to import. Connect to the VPN with username Administrator and the password you set in the previous step.

4. Set up Steam as above, though it’s already installed. Just login with your account credentials and configure it accordingly.

5. You should be good to go! Use the logout shortcut on the Desktop to log out, and then follow the standard Gaming Time section above.

Huge thanks for helping me with this goes out to: @crisg, @martinmroz, Jeff K. from AWS Support, Daniel Unterberger, Clive Blackledge, Matt Marino, Jérémie Lumbroso, and Alexander Sandström

It’s a bit of a sensationalistic title, but hear me out. And sorry for the boring-ish example in this article. This is designed to only excite people who really get it. :)

AWS Lambda lets you run NodeJS code or any binary (or code of any language) as a fast starting and stopping EC2 instance. You can use it to, for example, quickly resize an image as a background worker. What I’ll quickly show you here though is that you could even possibly use it as a full-on back-end for a single-page app.

What’s even more awesome? It basically scales by parallelizing infinitely. No need to ever spin up more EC2 instances or Dynos or any of that. Capacity planning? No need!

So, lets build a quick static webpage (that you can host anywhere, including file://)

All this page does is connect to Lambda and send sup1 as the value to key1 through simple JSON.

Here’s the lambda code (which you literally can do on the AWS Console):

This is even simpler. The handler callback is called by Lambda whenever this Lambda Function is called. context.succeed is used to return a value to the caller.

And lets run it:

FYI, this is crazy. We just did a backend call with full-on backend logic without setting up any kind of instances. No worrying about starting a new not-even-free-anymore Heroku Dyno. No choosing machine speed and memory and hard drive sizes. It just works, effortlessly, and scales enormously.

One thing you might note is that we’re exposing our AWS public and secret keys. This is actually not that big of a deal. To make this secure, we actually created a new AWS User and with a Policy specifically preventing everything except the one Lambda Function:

Oh and I mentioned that it scales “infinitely”, yeah there’s a 100-parallel-requests maximum right now, but I’d imagine this will be removed by AWS as the service matures. Meanwhile, you can ask them for more.

I guess recent AWS changes makes my lambda-job project kinda useless. It’s all built in now.

Very cool stuff. Hat tip to Amazon for constantly doing new amazing things.

thanks to @siong1987 for the quick review

NOTE: This article has been superseded by my more recent one here

How to use EC2 GPU machines + Steam In-Home Streaming + a VPN to play AAA titles on a shitty laptop

You might have tried a service like the now defunct OnLive. Though personally I’ve played and beat many AAA games on the service, it unfortunately a) had a very limited selection and b) is now gone. I also have a bunch of games on Steam that I’ve played using my eGPU. With the new Macbook though, I won’t be able to continue my low-end-laptop but high-end gaming extravaganza since there’s no Thunderbolt. So why am I not concerned? Steam recently introduced In-Home Streaming, which basically creates a mini-OnLive in your own home with all the same Steam games I played with my eGPU. But… let’s do it over the Internet!

Playing games this way is actually quite economical – especially when comparing to purchasing a full-on gaming rig. Here are the costs you’ll need to consider:

• GPU Instance runs about $0.11/hr (on a Spot instance, regularly around $0.70/hr)
• Data transfer will be around $0.09/GB, and at a sustained ~10mbit, itll cost you $0.41/hr (4.5GB/hr)
• Updated Apr 25: Hard drive (EBS General Purpose SSD) storage of 100GB is $12.00/mo, or a bit under $0.02/hr

This comes out to around $0.54/hr (Updated Apr 25), not bad, for the cost of a $1000 gaming pc, you get ~1900 hours on much higher-end hardware!

This is all fun and games, but you need to make sure of two things:

1. You are within 40ms to the closest AWS datacenter (test here) and has GPU instances (Updated Apr 25 brought this up from 20ms, many people reporting it’s just fine at slightly higher latencies)
2. You have at least a 10mbit connection and it’s unmetered

1. On AWS, create a new EC2 instance. Use these settings:
   • Base image should be Microsoft Windows Server 2012 R2 Base (since Windows still has all the best games)
     
   • Use a g2.2xlarge instance (to get an NVIDIA GRID K520 graphics card). Updated Apr 19: There is no point using any larger instances since all they do is just give you more GPUs you can’t use.
     
   • Use a Spot instance, it’s significantly cheaper (1/7th the regular cost) than regular instances
   • For storage, I recommend at least 100GB (Updated Apr 19: was 60GB) (so you can install lots of fancy games)
   • Also for storage if you’re using spot instances, make sure your primary disk doesn’t get deleted on termination
   • For the Security Group, i’d recommend just adding type All traffic
   • Finally, for the key pair, create a new one since you’ll need one for Windows (to retrieve the password)

2. Once your spot instance is assigned, use Microsoft Remote Desktop to connect to it (get it here on a Mac). The username is Administrator and the password you’ll need to retrieve from the EC2 Console. Once inside, make sure to install TightVNC server and use Screen Sharing (on a Mac, Updated Apt 19: or alternatively Screens which has better clipboard handling) to connect to the server. VNC is necessary so that the server uses the proper video card for rendering.

3. Install the NVIDIA K520 drivers from the Nvidia website
   

4. In order to make it actually use the video card, you’ll need to completely remove the default driver. Open up Device Manager, and a) disable the Microsoft Basic Display Adapter, b) uninstall it and c) delete the driver file C:\Windows\System32\Drivers\BasicDisplay.sys (see instructions for how to delete protected drivers here). Reboot and VNC back in.
   

5. Start the Windows Audio Service as per the instructions here. Also, since you’re on EC2, those machines do not virtualize a sound card. So install VB-Cable so you can get sound. Updated Apr 25: Alternatively, you can install the more-commercial Razer Surround to simulate 5.1 – it’s pretty cool.

6. Updated Apr 19: Install OpenVPN via the instructions here. Make sure to use the TAP interface so Steam’s multicast discovery gets forwarded. Here is my server config and client config. I personally use TunnelBlick on my Mac as the client.
   Updated May 7: You can alternatively use ZeroTier for both the server and client which is easier to set up, and still free, but less configurable. Remember to create a network on their website and enabling IP addressing.

7. Install Steam and get yourself on the Beta channel (available in the preferences). Also, start downloading whatever games you’ll want to stream. Oh and on your own Steam installation, make sure to turn on Hardware Decoding in the Steam settings, and I also recommend turning on Display Performance Information.
   

8. Start gamin!
   

While playing, make sure to hit F6 to see the latency graph. Anything above 50ms will make the delay somewhat noticable, though I’ve played with delays up to 100ms. It just takes some getting used to and before you know it you won’t even know you’re streaming your games from a computer far far away.

One other thing you should do is hit F8 while playing (note that sometimes this will cause the client to crash, but the file will still get written). F6 will do a dump of stats in the C:\Program Files (x86)\Steam\logs\streaming_log.txt file on the server. Open it up to see detailed latency timings. Here’s an example of the interesting lines:

"AvgPingMS"       "11.066625595092773"

"AvgCaptureMS"    "4.555628776550293"
"AvgConvertMS"    "0.023172963410615921"
"AvgEncodeMS"     "5.5545558929443359"
"AvgNetworkMS"    "7.0888233184814453"
"AvgDecodeMS"     "3.7478375434875488"
"AvgDisplayMS"    "6.3670969009399414"

"AvgFrameMS"      "27.798770904541016"
"AvgFPS"          "57.622333526611328"

Unfortunately Steam doesnt support pulling the video from the H264 encoder on the GRID’s NvFBC (which would reduce AvgEncodeMS a bunch). If you were running a GTX video card locally this is one thing that’d make it faster than using EC2 (in addition to largely decreasing NetworkMS).

See more information about this file in the Steam In-Home Streaming Steam Group.

• If you have the VPN running and you can’t get your client computer to see the server Steam, usually restarting Steam on the server will get the client to see it again. It’s a bit of a pain since you’ll need to VNC into the computer to restart things.
  Updated Apr 26: You can use wireshark and netcat to pipe broadcast pings from a mac to the EC2 machine.

  tshark -T fields -e data -l 'udp and dst port 27036' | script -q /dev/null xxd -r -p | nc -b tap0 -u 10.8.0.1 27036 > /dev/null
  

  See my up.sh and down.sh for automating this with the OpenVPN connection. You need to do this because the mac Steam client only broadcasts to one interface, i.e. not your VPN tunnel.

• If the game freezes, the way to get it out of it’s broken state is to Microsoft Remote Desktop in, close things, then go back in via VNC to restore Steam, etc.

• Updated Apr 19: If your Steam client freezes after logging in, restart the Steam server.

• Updated Apr 19: If you only see a black screen though you do hear sound, it usually means you need adjust the MTU of the VPN. See the OpenVPN discussion. Also, see the Steam In-Home Streaming discussion about this for more ideas. Updated Apr 26: I’ve also been told that switching OpenVPN to TCP might fix things, though that seems more of a workaround since now you don’t get the benefits of UDP.

If you have a) a fast internet connection and b) you’re near an AWS datacenter with GPU instances, in my opinion, this is actually quite practical. Not only performance-wise, but it’s also quite economical.

Also, RIP OnLive. Loved those guys.

Happy gaming!

Updated Jan 17, 2015: Moved the dynamic DNS away from a scheduled task to the new custom- service method
Updated Apr 3, 2016: Added dynamic DNS instructions for iwantmyname
Updated Jul 24, 2016: Added NAT-PMP for UPnP, up-to-date dynamic DNS methods, IPv6 instructions, and EdgeRouter-X mention

I’ve gotten a new, inexplicable, love for Ubiquiti. They fit in my favorite category of companies: they make high quality products that cost nothing compared to the old-boys-club equivalents. Oh and these products look spectacular. I’m actually quite surprised how UBNT is able to do all the things they do… I couldn’t support them more.

After finding out about them from MonkeyBrains, I looked through their product line to be quite surprised at the feature sets, yet still low cost. Being a huge fan of networking equipment, I decided to buy their cheapest router (at the time), the EdgeMAX EdgeRouter Lite. A 3 port, gigabit-capable router, that can really only be configured by commandline. Today, they have an even cheaper version of it, for $50! The EdgeRouter-X.

For this article, we’re going to configure the EdgeRouter Lite for home-use. Instructions should be similar if not identical for the EdgeRouter-X.

A Challenger Appears! and it’s only $100!

First off, like I mentioned, this thing is only really configurable via commandline. It uses a forked version of the opensourced edition of Vyatta 6.3 (now maintained as VyOS post their acquisition by Brocade). So to learn about how this thing works, you’ll need to read pages upon pages of documentation from the old Vyatta docs keeping in mind the Ubnt has modified a bunch of stuff too. A lot of help is available on their community page. Oddly enough, they don’t document anything they do, so you’re stuck reading release notes and harassing the employees that troll the forums.

A great way to get started is to read the EdgeOS CLI Primer and the Basic System [PDF] docs.

When you buy and receive yours, plug into Port 0, assign yourself a Static IP, and use the https Web UI to upload the latest version of their firmware (which is 1.6 as of this writing). This will get you up to speed with all the latest Web UI (which is still heavily limited). Use the Wizards on the 1.6 Web UI to get an initial configuration that can actually route something. Then lets start customizing it.

Log in via SSH with ubnt/ubnt.

As of 1.4, Ubiquiti added the non-Vayatta config of port-forward. Whereas in the past you’d need to manually create NAT mappings and firewall rules (as per here), the EdgeRouter now has made it significantly easier with even automatic firewall rules.

port-forward {
  auto-firewall enable
  hairpin-nat enable
  lan-interface eth1
  rule 1 {
    description "synology webui http"
    forward-to {
      address 192.168.0.10
      port 5000
    }
    original-port 9876
    protocol tcp
  }
  rule 2 {
    description "router ssh"
    forward-to {
      address 192.168.0.1
      port 22
    }
    original-port 8792
    protocol tcp
  }
  wan-interface eth0
}

As a reminder, to actually set settings on the router, switch to configuration mode, configure. Then use commands like set port-forward auto-firewall enable or set port-forward rule 1 forward-to address 192.168.0.10 to actually set the settings. Then use commit and save to make the config live and save it to /config.

In this example, we have the WAN on eth0 and LAN on eth1. We enable auto-firewall which really makes our life easy by dealing with firewall rules for LAN targets (more on that later). hairpin-nat makes it so that if an app on your LAN uses your public IP address as the remote host, the router will turn the packet right around without going out to your ISP.

There are two rules, rule 1 which we have exposed the Synology DiskStation Manager WebUI (great device btw) to the outside. Connections from TCP port 9876 on our WAN interface will be forwarded to 192.168.0.10 on port 5000. Firewall rules will automatically be created to allow incoming connections to the LAN for this port because of the auto-firewall instruction.

rule 2 allows me to access the router’s SSH for config from the outside. Same deal as rule 1 except unfortunately when routing to the router itself, it appeared as though I needed to create the firewall rule manually.

firewall {
  name WAN_LOCAL {
    [...]
    rule 4 {
      action accept
      description "port forwarding manual - router ssh"
      destination {
          address 192.168.0.1
          port 22
      }
      log disable
      protocol tcp
    }
    [...]
  }
}

Reminder that WAN_LOCAL is for packets between the outside and destined for the router versus WAN_IN which is for packets destined for the LAN. Normally home-grade routers don’t really distinguish these two apart, but for advanced configuration’s sake, this is the case here.

This firewall rule basically allows traffic that should be going to 192.168.0.1, the router’s internal IP, to access port 22, the internal port for ssh.

It’s always useful to have a couple machines on the network use DHCP, yet always get the same IP addresses assigned to them. Fortunately doing so is quite easy.

service {
  dhcp-server {
    [...]
    hostfile-update enable
    shared-network-name LAN {
      [...]
      authoritative enable
      subnet 192.168.0.0/24 {
        static-mapping driveway-camera {
            ip-address 192.168.0.20
            mac-address 00:02:d1:12:d3:e9
        }
        static-mapping synology {
            ip-address 192.168.0.10
            mac-address 00:11:32:41:1e:25
        }
      }
    }
  }
}

It’s mostly obvious what this does though the hostfile-update enable section is useful so you can access these mappings by name from the DNS server and other routing rules.

There are security problems with UPnP in general, but because of stuff like BitTorrent, World of Warcraft and many other applications, we need it to be easy to open ports for faster peer-to-peer.

service {
  upnp2 {
    listen-on eth1
    wan eth0
    nat-pmp enable
    secure-mode enable
  }
}

We’re using the upnp2 to do this. It’s the more up to date UPnP server in comparison to the legacy upnp that was available on the Vayatta system. upnp2 is more compatible with the latest applications too. Ubiquiti also included the ability to configure NAT-PMP, Apple’s way of doing UPnP (for things like Back To My Mac). secure-mode here just ensures that a computer cannot open a port for another.

It’s always useful to update a dynamic dns provider like DynDNS whenever your public IP changes. I personally own lg.io and I want it to be updated. The router actually has some convenient dynamic dns abilities depending on your provider. Below is an example using the custom- syntax for a No-IP entry and my iwantmyname updater too:

dynamic {
  interface eth0 {
    service custom-oursfapt {
      host-name oursfapt.workisboring.com
      login AWESOMEUSERNAME
      password AWESOMEPASSWORD
      protocol noip
    }
    service custom-iwantmyname {
      host-name myawesomehostname.lg.io
      login abc@def.com
      options script=/basicauth/ddns
      password awesomepassword
      protocol dyndns2
      server iwantmyname.com
    }
  }
}

In order to use a non-standard hostname (or service), you need to use the custom- prefix to the service name. Additionally, the protocol needs to be one that ddclient supports (look for the my %services = ( line).

The future is now, and you should start using IPv6. The router certainly doesn’t make this easy, and there are like 50 flavors of configuring IPv6 for different ISPs. The following is what I use with my ISP MonkeyBrains.

First up, like for IPv4, we’ll need to add firewall rules to let IPv6 traffic pass-through the router:

firewall {
  [...]
  ipv6-name IPv6_WAN_IN {
    default-action drop
    description "IPv6 packet from the internet to LAN"
    rule 1 {
      action accept
      description "Allow established sessions"
      state {
        established enable
        related enable
      }
    }
    rule 5 {
      action accept
      description "Allow ICMPv6"
      log disable
      protocol icmpv6
    }
    rule 10 {
      action drop
      description "Drop invalid connections"
      state {
        invalid enable
      }
    }
 }
 ipv6-name IPv6_WAN_LOCAL {
    default-action drop
    description "IPv6 WAN to Local"
    rule 5 {
      action accept
      description "Allow established sessions"
      state {
        established enable
        related enable
      }
    }
    rule 10 {
      action drop
      description "Drop invalid connections"
      state {
        invalid enable
      }
    }
    rule 15 {
      action accept
      protocol ipv6-icmp
    }
    rule 30 {
      action accept
      description "Allow dhcpv6"
      destination {
        port 546
      }
      protocol udp
      source {
        port 547
      }
    }
  }
  ipv6-receive-redirects disable
  ipv6-src-route disable
}

Next, you’ll need to get an IPv6 address for your router from your ISP. Typically they have DHCPv6 to do this for you, but not always. Fortunately for me MonkeyBrains does. Due to the way DHCPv6 works, we then use Prefix Delegation to route requests to the proper clients and we’ll use SLAAC for internal discovery.

interfaces {
  [...]
  ethernet eth0 {
    [...]
    dhcpv6-pd {
      pd 0 {
        interface eth1 {
          service slaac
        }
        prefix-length 64
      }
      rapid-commit enable
    }
  }
}

Oh and don’t forget to assign the IPv6 firewall rules to the internet interface:

interfaces {
  [...]
  ethernet eth0 {
    [...]
    in {
      ipv6-name IPv6_WAN_IN
      [...]
    }
    out {
      ipv6-name IPv6_WAN_LOCAL
      [...]
    }
  }
}

And that should be it! Sometimes it can take a while for IPs to get set up, but once it is, you should be good to go. A good way of testing is to SSH into the router and issue the command: ping6 google.com. Then try it on your own computer.

So here’s the fun part. The great part of an enterprise router is that it can do some pretty crazy things. What I like it for? Well, my parents want to use services like Netflix, Hulu, etc. But because of all sorts of anti-consumer business tactics on the publishers’ parts, these services don’t have a full catalogue available in Canada.

Using the EdgeRouter, you can do things like route all traffic from one LAN client (like an Apple TV) to always go through a VPN (like an OpenVPN). Fancy! Lets do it.

interfaces {
  [...]
  openvpn vtun0 {
    config-file /config/auth/pia/USEast.ovpn
  }
}

I use Private Internet Access. They’re kinda sketch in their branding, but they have a great product and have access to OpenVPN too, which I wanted. What sucks though – OpenVPN is not hardware accelerated on the EdgeRouter. This isn’t great for throughput. We’ll only be able to do about 7mbit/s. Fortunately that’s sufficient for my parents’ needs though. The EdgeRouter does support hardware accelerated IPSec, but there aren’t any VPN providers out there that I know of that allow you to tunnel IPSec through them.

Here’s the config I use with PIA (in /config/auth/pia/USEast.ovpn):

client
dev vtun
proto udp
remote us-east.privateinternetaccess.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /config/auth/pia/ca.crt
tls-client
remote-cert-tls server
auth-user-pass /config/auth/pia/pw.txt
comp-lzo
verb 1
reneg-sec 0
crl-verify /config/auth/pia/crl.pem
route-nopull

The big deal is the route-nopull which makes it so that PIA doesnt set the default gateway and route all your traffic to the VPN. Normally you’d use a VPN provider on your computer to route everything, but in this case we only want selective routing. Which we’ll set up next.

The way we can modify packets to go through a different interface is to use the firewall. Lets create a firewall modify config to take all traffic from one IP address and force it over the OpenVPN tunnel we created above.

firewall {
  modify SOURCE_ROUTE {
    rule 10 {
      description "traffic from a lan ip goes through vpn"
      modify {
        table 1
      }
      source {
        address 192.168.0.250/32
      }
    }
  }
}

protocols {
  static {
    table 1 {
      interface-route 0.0.0.0/0 {
        next-hop-interface vtun0 {
        }
      }
    }
  }
}

interfaces {
  ethernet eth1 {
    [...]
    firewall {
      in {
        modify SOURCE_ROUTE
      }
    }
  }
}

table 1 is just a static routes table. By using 0.0.0.0/0 we’re basically matching everything on this routing table to go to the VPN. What’s neat here is the interface-route and next-hop-interface which makes it much easier to specify which tunnel to take, even when you could be assigned a dynamic ip.

Finally, data has been sent out through the VPN now, but we need to be able to receive data back too. For that, much like on the WAN, we need to masquerade NAT:

service {
  nat {
    rule 5001 {
      description "masquerade for VTUN0"
      log disable
      outbound-interface vtun0
      type masquerade
    }
  }
}

And that’s it! Now everything from 192.168.0.250 will always be tunneled through the OpenVPN – and the Apple TV will have absolutely no idea what’s going on.

I love this router. The policy-based routing example above only begins to scratch the surface of how powerful it can really be. Yes, it’s complicated to set up, but it’s not that bad. The config is straightforward, there’s just a bunch of it. Ubiquiti is working to make this stuff configurable via the Web UI, but for now I’ll take advantage of its complexity for my own exclusive club of awesomeness.

This winter vacation, I had a bunch of spare time at home with my parents. It turns out that my parents call to Romania a lot – especially during the holiday season. Since Romania is far away from here in Canada, if they were to use the local phone provider, Bell, they’d be destroyed with long distance charges (c’mon guys, it’s 2015!). Usually you’d go out and get calling cards, etc, but in the past, when i was still living with them, what I did to get around the high fees of Bell and the unreliability and inconvenience of calling cards is I set up Asterisk.

Asterisk is a self-hosted PBX that can connect to SIP/IAX trunks provided by VoIP companies (I personally used to use Unlimitel, they’re great). Problem with Asterisk is that it needs maintenance. It needs to run on a server, it’s very complicated, patches sometimes break backwards compatibility, etc etc. I needed something less involved that if I were to get hit by a bus and go silent (otherwise known as starting a startup), they’d be hopeless to figure it out.

Phone tech has gotten significantly more advanced in the recent years. Though some companies like Bell still party like it’s 1960, companies like Twilio have come along and made things a lot more powerful, easier and cost efficient.

Today’s we’re going to set up a minimally hosted service which accepts Twilio calls on a local number and then forwards them on via VoiceTrading’s SIP server.

Basically here’s the flow of what I wanted to do:

1. Parents dial a phone number
2. Twilio accepts call
3. Parents type in a secret code
4. Twilio waits for a caller to type new phone number
5. Call gets forwarded via SIP to VoiceTrading
6. VoiceTrading calls the number and connects the call
7. If the call errors at all, tell the caller

Luckily for us TwiML can help do exactly this. It helps accept calls, play text-to-speech messages, accept input, redirect via SIP, etc. The unfortunate part is it does not allow for any logic. For example, if you prompt the caller to press 1 for abc and 2 for def, you need a serverside script to process the input. I feel like if they just added a tag as part of TwiML which let you change flow depending on input, we wouldn’t even need a serverside component. But oh well, can’t have em all…

One thing I know I strongly dislike maintaining is backend code. I spent almost 3 years at Twitter during crazy downtime days helping deal with it. I try these days to avoid building services that require me to SSH into anything, or run any kind of daemon or process that might crash. I basically want an even less involved platform-as-a-service than something like Heroku. No thx to applying security patches, upgrading versions of stuff, etc.

So, for the first time in almost 10 years, I busted out my DreamHost account which provides, wait for it, PHP CGI. I really wanted to avoid PHP because I’ve become so much better at much better languages, but honestly, in this case it fits the bill perfectly. With a strong focus on simplicity, here’s the script I ended up building:

Man, it’s killing me that I can’t just do this all in one XML and not even need a serverside component! Either way, DreamHost is super low maintenance. You upload the script and well, you’re done. Never need to touch it again or reboot dynos, or any of that.

Basically Twilio re-calls the same script with each input the user does. We pass GET parameters to specify which part of the flow the caller’s part of. Super simple, easy to read and understand.

Twilio kind of hides this functionality, but it’s not too bad.

1. Log into your Twilio account and make sure you have a phone number created that you’ll receive calls from.
2. Go to Dev Tools then TwiML Apps, then click Create TwiML App.
3. Set the Friendly Name to something, and the Voice Request URL to where you placed your script.
4. You can skip the Messaging Request URL since we’re not doing any kind of text message forwarding.
5. Save your settings
6. Now go to the Numbers page and click on your phone number.
7. Click on Configure with Application on the Voice section and select your script.
8. Click Save and you’re done!

Should you have any problems, Twilio has great developer tools and debugging capability to see exactly what you sent it and what went wrong. They even have a web dialer so you don’t even need to pick up the phone.

Twilio has been a dream to work with. Not only do we use them for Envoy at work, but I’m happy I now have an excuse to use them for myself too.

I have just built a maintenance-free system and saved a ton of time and headaches for the future. Cool stuff. Who’d have thought PHP could have been so useful? ;)

Note: This is a repost of my TechInferno Forum article from July 28th, 2013. Methods have changed and are somewhat easier these days. Make sure to check out that forum for more info.

TLDR: By buying around $250 in commonly available parts, plus a video card, you can make the graphics of your 11” Macbook Air from 5X to 7X faster. Demo video at end of post. Step-by-step, here’s how to exactly do it. Warning: not for the faint of heart!

Hey everyone!

This is my third article here on this forum, though it’s the first that the process can be done by anyone with off-the-shelf parts. No more discontinued exotic parts like the $180 BPlus TH05 are required. All you need is a macbook air, a graphics card, a power supply, Windows 7, and ~$250 to buy some adapters and software online. All these parts are readily available for anyone.

Like usual, I really want to thank nando4 for his help in doing all this. He’s the mastermind behind the technicals, I just like writing articles and making stuff easier for everyone. He’s super dedicated and eGPUs wouldn’t be anywhere near where they are today if it wasn’t for him! Thanks!

So what are we doing? We’re going to make a Macbook Air accept an external video card via Thunderbolt! Yes, you might have read in the news that real commercial solutions are just around the corner. We’ve been promised by these companies over-and-over again, with youtube videos, hands-on reviews, press releases, etc, but nobody is releasing anything. It’s been like this for over a year. Intel even openly admits its bias against GPU usage where it’s listed as unsupported in their Thunderbolt Certification Application. Talking to one of their thunderbolt guys, here’s what that “Not Supported” means:

The “Not supported” means that Intel won’t neither certify your product nor deliver, at the moment, any Technology License for this kind of usage. As you know, this Technology License is required to develop a Thunderbolt device in the market and Certification is a must have to market any Thunderbolt product.

So with the bad news out of the way, the good news is that you can still do it yourself – just a bit less elegantly. We’ll be using the Sonnet Thunderbolt to ExpressCard adapter, together with the BPlus PE4L ExpressCard to PCI-Express adapter. This PE4L adapter also includes a Delayed PCI-Reset jumper, making Windows 7 + Internal LCD rendering possible on the Macbook. Also, it’s not that bad. As you’ll see by the benchmarks later in the article, yes you’re only running at expresscard 5Gbps x1 2.0 PCI bus speed (as opposed to 16X 2.0 on a proper PC and only half of Thunderbolt’s 10Gbps), but its WAY WAY better than the internal integrated graphics of the laptop, plus you can still max out tons of games. The full PC bus speed is super rarely used anyways, so it’s not like you’ll get 1/16th the performance.

As part of this tutorial, we’ll be using Windows 7 BIOS (installed the regular Bootcamp way). Things are possible in Windows 8 as well, but the instructions differ, and I’ve also had troubles getting Internal LCD rendering working on Windows 8. Yeah I’m not a fan of using legacy Windows versions either, but whatever, every game works on both OSes for now anyways. Oh and we’re using Windows because games only exist for it, and I can’t get the setup to work on OSX (haven’t tried too much though).

Alright, lets get started!

• Mid-2013 11” Macbook Air
• 1.7 GHz Intel Core i7-4650U (basically the most maxed out 11” mba)
• 8GB 1600 MHz DDR3
• Intel HD Graphics 5000 1024 MB
• 512GB Apple SSD

• Sonnet Echo ExpressCard Pro. I purchased mine for $134 at B&H Photo Video. This adapter turns 10Gbps Thunderbolt to 5Gbps ExpressCard, which is needed for the PE4L later. It’s probably one of the more expensive parts in your setup because of Intel’s arbitrage on Thunderbolt-related parts. Note that Sonnet also sells a faster 10Gbps Thunderbolt->PCIExpress box (~US$310 Sonnet Echo Express SE) which might seem like a great idea, but that’s all sorts of problems with it, including an underpowered power supply, no PCI Delay switch (making it not easily work with Windows) and dismantling it to be able to use full length and double width video cards.

• $70 BPlus PE4L V2.1 ExpressCard to PCI-Express adapter. You want the PE4L-EC060A package that includes the SWEX adapter to power on your power supply. If wanting a neater enclosure solution then purchase a $170 BPlus PE4H V3.2 instead, noting that your chosen video card will require the pci-e power connectors on the side of the card rather than the top. If you’re curious, BPlus used to offer a US$180 TH05 (which included the TB cable), which was a direct Thunderbolt to PCI-Express, but Intel shut it down and the entire BPlus Thunderbolt division in Jan 2013 per TH05 Recall Notice.

• A Thunderbolt cable. You can get this at any Apple Store or online. I’d recommend getting a 2m cable since you’ll probably want to have your GPU not directly beside your laptop.

• 450W power supply capable of running the video card. $24AR-shipped Diablotek PH450 offers 12V/30A (360W) or Corsair CX430 offers 12V/32A (384W) of peak power, enough for ALL current video cards. If getting a basic ATX PSU then carefully read the first rail data on it, eg: 12V1:18A means 12*18=216W of peak power. That wouldn’t be enough to drive my GTX570 that can draw up to 298W peak power. Look at your video card’s spec sheet to see the peak wattage only it uses (not the suggested value that often includes motherboard + hard drives, etc). Honestly though, I’d recommend going for the 450W or even 500W power supplies available for around $20 at your local money-laundering stolen-stuff electronics store. If you get a power supply that doesn’t output enough or doesn’t like power spikes, it’ll basically make your computer blue screen a lot mid-gaming. Ask me how I know.

• $25 DIY eGPU Setup 1.X, developed by nando. Yes, you’re paying for software, get over it. Nando did spectacular work to get Windows 7 Bootcamp to be able to properly accept the external videocard without giving an “Error 12” code. You want the latest 1.20 version incorporating new Macbook features that’s not advertised on that linked page as yet.

• A video card. I have the NVidia GTX 570, which is an awesome balance of great performance and price. You can use basically any video card you want, including AMD ones. Note on AMD cards, internal LCD rendering won’t be possible without using something like Lucidlogix Virtu (not covered in this article). Also, don’t go too crazy and order a NVidia Titan. Yes it’s a great card, but you won’t see the value for money since you are limited to a slower PCI bus. I’d recommend sticking in the 5xx or 6xx series of NVidia GTX cards.

• 2013 11” Macbook Air. This is the laptop I have, but these instructions should be identical for the 13” Air. Additionally, the only step that will be different for every other kind of Macbook is the contents of the PCI.BAT file later.

• A USB memory key that’s at least 4GB for Bootcamp to install windows plus it’s drivers.

• Other software: Windows 7 ISO (from MSDN / MSDNAA / etc). Don’t steal software.

1. On your mac use Boot Camp Assistant to prep a USB key with Windows 7 64-bit

2. Still with Boot Camp Assistant, partition your main drive and start the Windows 7 installation process. I recommend around at least a 60GB partition. Games are big these days. If you just don’t have the space, you can do what I do and turn off Hibernation Mode and the Virtual Memory Page File to save hard drive space (about 17GB together). Disabling Virtual Memory is kind of a bad idea, but I’ve never had troubles doing it – this laptop’s 8GB of ram is usually enough for anything.

3. Once partitioning is done, your computer will auto reboot and the Windows 7 installer will start. Install Windows 7.

4. The Windows 7 installer will install the Boot Camp drivers at the end of it.

5. Go to Intel’s site and download the latest Intel HD 5000 drivers for Windows 7 64-bit. It’s the same download for the HD 4000 drivers.

6. Launch Windows Update and apply all required/optional patches. Reboot as required. Repeat this step until nothing’s left. This will take a long time, deal with it.

7. There’s some clock weirdness when switching between Windows and OSX, so add the key (via regedit) HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation\RealTimeIsUniversal with DWORD value 1

1. On the PE4L, set SW1 (on the right) to position 3 (6.9s, though note it’s more like 30s). Set SW2 (on the left) to position 2-3 for 2X (even though only 1X is really supported thanks to Intel requiring us to use the Sonnet Expresscard adapter).

2. Plug in the power (the white connector) into the PE4L from your power supply’s floppy disk drive connector. There is no need to plug in the Black and Red cable (this is for automatic powersupply control, which doesnt work on the Macbook). Keep the jumper on both pins on the bottom right of the board.

3. Plug in the motherboard connector to the included SWEX adapter. Make sure it is in the ON position (1-2). Also plug in any required 6-pin/8-pin connectors to your GPU. Mine required 2 6-pin plugs.

4. Plug in the PE4L to your video card. It obviously won’t cover all the pins of your video card, that’s ok. Yes, I know it looks ghetto.

5. Plug the express card end of the PE4L into the Sonnet ExpressCard Pro.

6. Make sure your laptop is shut down.

7. Plug in the thunderbolt cable to the Sonnet and then the laptop. Also, make sure no monitors are plugged into the video card.

8. Turn on the power to the power supply, the GPUs fans should start, then start your laptop. Hold the Option key. Now wait for the red light to turn off on the PE4L. Once it’s off, select Windows.

9. Go to NVIDIA’s site, download and install the latest drivers for your video card. When prompted to reboot, shut down the computer instead.

10. Turn off the eGPU’s power supply.

11. Start the eGPU’s power supply again and boot your laptop, waiting for that red light to go off again before selecting your Windows partition. You will always need to do a power cycle of the eGPU when rebooting your computer because of the way the PE4L works.

12. Once Windows has started, open the Device Manager. Select Scan For New Hardware.

13. Notice the GTX 570 (or whatever card you have) is listed with the yellow exclamation mark. If you double click on it you’ll see the dreaded Error 12. This Error 12 means that Windows wasn’t able to allocate a contiguous block of memory for the video card. Yes you probably have 8GB in your laptop, but the PCI Bus doesn’t work that way. We’re going to get rid of that error by reallocating devices in the PCI Bus.

1. Go purchase DIY eGPU Setup 1.x (link above) if you haven’t done so already. Run the self-extracting exe when booted into Windows to install to c:\eGPU.

2. Open an administrator command line and run c:\eGPU\setup-disk-image.bat. This will install everything and add a boot item so you can load the video card. Next we’ll configure DIY eGPU Setup to work with the 2013 Macbook Air and Haswell chipset.

3. Mount the virtual disk image by running e:\eGPU\eGPU-Setup-mount.bat in Administrator mode. This will mount a V: drive.

4. In notepad, create the file V:\config\pci.bat and paste the following into it. Note that this is the file that changes depending on your Macbook type. If you dont have a 2013 Macbook Air, a couple addresses might change. Post in this thread and hopefully someone will post your config.

    :: TB TH05 uses 9:0.0 bridge, Sonnet/OWC uses 9:3.0. That 9:3.0 line may need
    :: to be altered depending on what TB enclosure/adapter you use.
    ::
    :: Disable CMD, set PCIe config space
    @echo -s 0:1c.4 COMMAND=0 1d.b=50 22.w=BEB0 26.w=D3F1 > setpci.arg
    @echo -s 5:00.0 COMMAND=0 1d.b=41 20.w=B0B0 22.w=BB00 24.w=C001 26.w=D1F1 >> setpci.arg
    @echo -s 6:03.0 COMMAND=0 1d.b=31 20.w=B200 22.w=b700 24.w=c001 26.w=CDF1 >> setpci.arg
    @echo -s 6:04.0 COMMAND=0 1c.b=41 1d.b=41   20.w=BAC0 22.w=BAF0 24.w=D1C1 26.w=D1F1 >> setpci.arg
    @echo -s 7:00.0 COMMAND=0 3C.b=0 >> setpci.arg
    @echo -s 8:00.0 COMMAND=0 1c.w=2121 20.l=B300B200 24.l=C9F1C001 28.l=0 30.w=0 3c.b=10 >> setpci.arg
    @echo -s 9:03.0 COMMAND=0 1c.w=2121 20.l=B300B200 24.l=C9F1C001 28.l=0 30.w=0 3c.b=10 >> setpci.arg
   
    :: NVidia eGPU
    @echo -s a:00.0 COMMAND=0 10.l=b2000000 14.l=c0000000 1c.l=c8000000 24.l=00002F81 3c.b=10 3C.b=10 50.b=1 88.w=140 >> setpci.arg
    @echo -s a:00.1 COMMAND=0 10.l=B30FC000 3c.b=10 >> setpci.arg
   
    :: Re-enable CMD
    @echo -s 0:1c.4 COMMAND=7 -s 5:0.0 COMMAND=7 -s 6:3.0 COMMAND=7 -s 6:4.0 COMMAND=7 >> setpci.arg
    @echo -s 7:00.0 COMMAND=6 -s 8:0.0 COMMAND=7 -s 9:3.0 COMMAND=7 >> setpci.arg
    @echo -s a:00.0 COMMAND=6 -s a:0.1 COMMAND=6 >> setpci.arg
    setpci @setpci.arg
   

5. In notepad, edit the file V:\config\startup.bat and paste the following into it:

    call speedup lbacache
    call vidwait 60
    call vidinit -d %eGPU%
    call pci.bat
    call chainload mbr
   

6. Turn off your MacBook, power cycle the eGPU from the power supply and make sure the thunderbolt cable is still plugged into the MacBook. Make sure no display is plugged into the card.

7. Turn your MacBook on while holding Option. There’s no need to wait for the red light to turn off now before proceeding. Select the windows partition.

8. Windows will boot into a menu allowing you to select between Windows and eGPU setup. Select eGPU setup.

9. Once you get to the Blue first menu, press enter for Option 1. This will prep the PCI Bus. Note it might take a few seconds for the eGPU to be detected (basically until the red light goes off)

10. Once that exits, you’ll be back at that same boot menu. This time select Windows 7 and wait for it to boot.

11. Open up the Device Manager and you should see the GTX 570 again, except … without the yellow exclamation mark! Horray! You fixed Error 12! This is basically whats been preventing people for a while from getting eGPUs working on their laptops.

12. Double click on the NVidia icon in the system tray. On the left side click on “Adjust image settings with preview”.

13. I know it’s shocking, but if you see a spinning NVidia logo, your internal LCD screen is being rendered by your external GPU! If you don’t believe me, launch your favorite game and notice how there’s no way the Intel HD 5000 could render it so well. I recommend you now install your fav benchmarking software, GPU-Z, FRAPS, Steam, etc to take advantage of your laptop’s new abilities.

14. You win!

• Again, for AMD cards to render on the Internal LCD, you’ll need to use Lucidlogix Virtu.

• When doing Internal LCD mode (which you’re doing when you have no monitor plugged into the video card), PhysX might not be on. Open the NVidia control panel and switch it from CPU to Auto. When doing benchmarks, keep it on CPU though.

• Dont forget that every time you reboot, you must power cycle the eGPU!

• If someone knows how to get the eGPU to restart with a reboot of the computer, please let me know. Simply connecting the black and red cable isn’t enough on the MBP from my observations. I tried the switch in both positions and both polarities.

Cutting to the chase, benchmarks are below. It’s insanely fast because the MBA LCD is 1366x768, it’s a 15W i7-UM Haswell CPU that is on par performance wise to a high end 35W Sandy Bridge i5 CPU, plus a crazy video card. Woo! :D

• 3dmark06: 19921
• vantage: P15876 (gpu=19574)
• 3dmark11: P4900 (gpu=5210)
• 3dmark: Ice Storm: 87663, Cloud Gate: 10128, Fire Strike: 3413 see results

• 3dmark06: 17645
• vantage: P15030 (gpu=18270)
• 3dmark11: P4732 (gpu=5110)
• 3dmark: Ice Storm: 23839, Cloud Gate: 8943, Fire Strike: 3264 see results

• All numbers above are in frames per second done using FRAPS and recording 1 minute of actual gameplay

• All games were run at 1366x768. Internal and external monitor were set to this. VSync off everywhere. The Macbook Air 11” has this resolution on the LCD. Sorry that this kinda makes the numbers seem high for real-world-with-a-monitor gaming.

• “Internal” refers to numbers when rending in internal LCD mode. “External” is when I had an external monitor hooked up

• “Min settings” means that I set every setting to the lowest possible value. This should make the game run as fast as possible (but look ugly). “Max settings” is the opposite. If the game had the option for presets of “Very Low” and “Ultra”, etc, I just selected those and didnt go into advanced modes (except to turn off VSync).

• Bioshock Infinite Internal LCD numbers arent ready yet.

A decision you’ll need to make is if you want to plug a monitor into your video card or just use your laptop’s monitor. Each has it’s own pros and cons. You’ll get faster performance with an external monitor, but you’ll lose the convenience of not needing a giant monitor. This becomes relevant as people make better eGPU cases where your eGPU will be portable. Why bring a monitor to your friend’s place when your laptop already has one?

It’s actually kind of cool that you even get this choice. The way it works is by the NVidia Optimus drivers taking the video frame memory from the video card, piping it back over the Thunderbolt bridge to the Intel HD 5000 memory and overwriting Intel’s memory so that you see the eGPU’s output on the Intel LCD. Cool! If you’re curious, this is the exact tech that’s used when laptops have an NVidia internal discreet graphics chip.

It has become very clear that gaming is not only high-performance, but super practical on an 11” Macbook Air. There’s so much going against it: this hodgepodge of adapters, it has a low voltage CPU, disaster of wiring and exposed sensitive parts, crazy boot-time chainloading software, Intel killing companies producing adapters and products left right and center via legal threats, etc. but somehow, with the right parts and some patience, it works spectacularly. And is quite cheap too!

Again I want to thank nando4 for all his help in working with me tirelessly over the last few months to get this working. Also, thanks goes out to TechInferno and the community thats been built here for allowing people to help eachother so efficiently. If you have any questions/comments, please feel free to reply to this thread.

I’m also available on twitter, @lg.

Thanks everyone – have a great day! And enjoy gaming, I hear PayDay 2 is awesome too :D

If you’re a frequent flyer (like i’ve oddly enough become recently) you probably have status with your airline that gives you lounge access for when you’re flying. Also, if you’re a frequent flyer, you probably also really enjoy drinking and paying as little as possible for it. So here’s a way that you (and sometimes also your entire family) can all have unlimited party nights out — on your airline’s dollar!

1. Decide you want to party and not pay anything.
2. Go online and buy a refundable business or first class ticket with your airline to anywhere.
3. Go to the airport.
4. Check in, go past security and enter your airline’s lounge.
5. Party.
6. Use the lounge’s wifi to cancel your ticket (or call the airline to do it for you)
7. Leave.
8. Notice that you still have cash in your pocket!

Yes, it’s abusive, but there you go.