journalctl is a great tool to read and filter logs. Since it is context aware, it is much easier to use compared to just using tail on text files. It can be super helpful for example to filter output given on a specific service. But how do you combine filters with a logical OR?
Today I had the need to filter the output to kernel messages, and NetworkManager. Kernel messages is -k, and NetworkManager is -u NetworkManager in short. But the combination doesn’t reveal anything:
$ journalctl -u NetworkManager -k
-- No entries --
The reason is that the (implied) logical connection here is “AND”. Only output is shown that is both a kernel message and from NetworkManager – which doesn’t exist.
journalctl knows about the + operator to combine output – but that doesn’t help with the short options used above:
$ journalctl -u NetworkManager -k
-- No entries --
Instead, we need to use the long form, call out the systemd unit as a flag explicitly, and the transport for kernel:
$ journalctl _TRANSPORT=kernel + _SYSTEMD_UNIT=NetworkManager.service --since today
...
Dez 18 21:26:52 russel NetworkManager[1510]: <info> [1734553612.1558] dhcp4 (wlp9s0): state changed new lease, address=192.168.1.2
Dez 18 21:26:52 russel kernel: wlp9s0: Limiting TX power to 30 (30 - 0) dBm as advertised by 12:34:56:78:90:12
Of course, this can be combined with -f, --since today and other typical journalctl flags. It can even be extended by more services:
When working with larger data structures in Nushell, there are often tables that are wider than the terminal has width, resulting in some columns truncated, indicated by the three dots .... But how can we expand the dots? The answer is simple, but surprisingly, not easily found. The “Working with tables” documentation of Nushell weirdly … Continue reading "[Short Tip] Get all columns in a table in Nushell"
Show full content
When working with larger data structures in Nushell, there are often tables that are wider than the terminal has width, resulting in some columns truncated, indicated by the three dots .... But how can we expand the dots?
The answer is simple, but surprisingly, not easily found. The “Working with tables” documentation of Nushell weirdly doesn’t tell, for example. The trick is to use the command columns to get a list of all column names:
For a test I recently had to process a plain list of items that was outputted by a program. In Bash, the usual way to do so is: But how could this be done in Nushell? Just using the same command gives an error: Instead, the trick is to tell Nushell to read the input … Continue reading "[Short Tip] Processing line by line in a loop in Nushell"
Show full content
For a test I recently had to process a plain list of items that was outputted by a program. In Bash, the usual way to do so is:
while read -r line; do COMMAND $line; done
But how could this be done in Nushell? Just using the same command gives an error:
❯ flatpak list|grep system|cut -f 2|while read -r line; do flatpak info $line; done Error: nu::parser::parse_mismatch
× Parse mismatch during operation. ╭─[entry #2:1:1] 1 │ flatpak list|grep system|cut -f 2|while read -r line; do flatpak info $line; done · ─┬ · ╰── expected operator ╰────
❯ flatpak list|grep system|cut -f 2|while read line; do flatpak info $line; done Error: nu::parser::parse_mismatch
× Parse mismatch during operation. ╭─[entry #3:1:1] 1 │ flatpak list|grep system|cut -f 2|while read line; do flatpak info $line; done · ──┬─ · ╰── expected block, closure or record ╰────
Instead, the trick is to tell Nushell to read the input line by line with lines, and then process each and every item with a sub-function:
Nushell is becoming a more and more serious shell every day. One thing missing in the past was the capability to create and use Python virtual environments. This has changed: Nushell was added as another supported shell in the virtualenv package: However, there is one catch: the source command does not work when you try … Continue reading "[Short Tip] Using a Python virtual environment in Nushell"
Show full content
Nushell is becoming a more and more serious shell every day. One thing missing in the past was the capability to create and use Python virtual environments.
However, there is one catch: the source command does not work when you try to use it to switch to the new environment:
🕙(20:40:28) ~
❯ source ~/development/ansible/bin/activate.nu
Error: nu::parser::unexpected_keyword
× Statement used in pipeline.
╭─[/home/liquidat/development/ansible/bin/activate.nu:116:1]
116 │ export alias pydoc = python -m pydoc
117 │ export alias deactivate = overlay hide activate
· ───┬───
· ╰── not allowed in pipeline
╰────
help: 'overlay' keyword is not allowed in pipeline. Use 'overlay' by itself, outside of a pipeline.
Instead, you need to use the overlay command:
🕙(20:40:50) ~
❯ overlay use ~/development/ansible/bin/activate.nu
(ansible)
Afterwards, you can continue to operate in the environment like usual:
🕙(20:42:41) ~/development/ansible via 🐍 v3.11.5 (ansible)
❯ pip install ansible
Collecting ansible
Downloading ansible-8.4.0-py3-none-any.whl (47.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 MB 19.6 MB/s eta 0:00:00
[...]
[notice] A new release of pip available: 22.3.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
(ansible)
🕙(20:43:15) ~/development/ansible via 🐍 v3.11.5 (ansible) took 20s
❯
Recently I realized that one of the disks in my server had died. After the replacement, the RAID sync started – and I quickly had to learn that this was going to take days (!). But I also learned that the time it might take massively jumped up and down. Thus I thought it would … Continue reading "[Short Tip] Plot live-data in Linux terminal"
Show full content
Recently I realized that one of the disks in my server had died. After the replacement, the RAID sync started – and I quickly had to learn that this was going to take days (!). But I also learned that the time it might take massively jumped up and down.
Thus I thought it would be fun to monitor the progress of this. First, I just crated a command to watch the minutes (calculated into days) every few seconds with watch:
But since it was jumping so much I was wondering if I could live-plot the data in the terminal (remote server, after all). There are many ways to do that, even gnuplot seems to have an options for that, but I wanted something more simple. Enter: pipeplot
First I tried to use watch together with pipeplot, but it was easier to just write a short for loop around it:
After I learned how subshells can be executed within nushell I was confident that I could handle that part. But few minutes ago I run into an error I didn’t really understand: I thought the parameter was provided somehow in the wrong way, and put it into quotes: "dwebp". But it didn’t help. I tested … Continue reading "[Short Tip] Accessing tabular nushell output for non-nushell commands"
I thought the parameter was provided somehow in the wrong way, and put it into quotes: "dwebp". But it didn’t help. I tested around more with sub-shells, some of them worked while others didn’t. The error message was misleading for me, letting me think that there is a difference in how the argument is interpreted:
It took me a while until I understood what I was looking at – and to make the error message make sense: the builtin nushell command which can give back multiple results, thus returning a table. The builting nushell command echo returns a string!
Thus the right way to execute my query is to get the content of the cell of the table I am looking at via get:
Containers are meant to keep processes contained. But there are ways to gather information about the host – like the actual execution environment you are running in. Containers are pretty good at keeping everyone and everything inside their boundaries – thanks to SELinux, namespaces and so on. But they are not perfect. Thanks to a … Continue reading "Figuring out the container runtime you are in"
Show full content
Containers are meant to keep processes contained. But there are ways to gather information about the host – like the actual execution environment you are running in.
Containers are pretty good at keeping everyone and everything inside their boundaries – thanks to SELinux, namespaces and so on. But they are not perfect. Thanks to a recent Azure security flaw I was made aware of a nice trick via the /proc/ file system to figure out what container runtime the container is running in.
The idea is that the started container inherits some /proc/ entries – among the entry for /proc/self/. If we are able to load a malicious container, we can use this knowledge to execute the container host binary and by that get information about the runtime.
As an example, let’s take a Fedora container image with podman (and all it’s library dependencies) installed. We can run it and check the version of crun inside:
With this nice little feature we now have insight into the version used – and as it was shown in the detailed write-up of the Azure security problem mentioned above, this insight can be crucial to identify and use the right CVEs to start lateral movement.
Of course this requires us to be able to load containers of our choice, and to have the right libraries inside the container. This example is simplified because I knew about the target system and there was a container available with everything I needed. For a more realistic attack container image, check out whoc.
Cilium is a networking plugin for Kubernetes based on eBPF. If you want to give it a try, Minikube is a good option to get started. Background I just started with Isovalent – and since I am very much a beginner regarding everything related to Kubernetes I decided to get some hands-on experience with the … Continue reading "[Howto] Installing Cilium with Minikube on Fedora"
Show full content
Cilium is a networking plugin for Kubernetes based on eBPF. If you want to give it a try, Minikube is a good option to get started.
Background
I just started with Isovalent – and since I am very much a beginner regarding everything related to Kubernetes I decided to get some hands-on experience with the technology I am going to work with for the foreseeable future.
Isovalent’s offering is an Enterprise version of Cilium which basically manages and secures connections between containers and adds observability to it. It all runs on eBPF and thus is pretty performant. eBPF can run sandboxed programs in Linux kernel space without the need to recompile the kernel; A tiny bit like a “Kernel VM”. I always wanted to get my hands dirty with eBPF anyway, and Cilium is a very good way to approach it. But where to start? The answer is: with a small Kubernetes setup based on Minikube, a tiny Kubernetes distribution for testing and fooling around which leaves your main system almost unchanged.
Preparing the environment
Minikube runs itself in a tightly confined environment to not disturb your other systems. This abstraction is done via containers or VMs realized via so called “drivers”. Drivers are available for Docker, VMWare, KVM, Podman and others. I decided to go with the KVM driver, so the virtualization bits need to be installed:
Note in the above commands that the last command only works in Nushell and has to be slightly adjusted for Bash or Zsh.
Next we have to install Minikube itself:
❯ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-latest.x86_64.rpm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15.1M 100 15.1M 0 0 8304k 0 0:00:01 0:00:01 --:--:-- 8300k
❯ sudo rpm -Uvh minikube-latest.x86_64.rpm
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:minikube-1.22.0-0 ################################# [100%]
Also, to install and manage Cilium easily it makes sense to use the Cilium CLI. Unfortunately the CLI is currently not available as a RPM package for Fedora, so we have to install the binary and move it to /usr/local/bin:
❯ curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
[...]
❯ sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
cilium-linux-amd64.tar.gz: OK
❯ sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
Starting Minikube with CNI
We now need to start up our Kubernetes cluster, and it needs to be in a way that we can install and use Cilium in it. So we set the network configuration to CNI:
❯ minikube start --network-plugin=cni
😄 minikube v1.22.0 on Fedora 34
✨ Automatically selected the kvm2 driver. Other choices: podman, ssh
💾 Downloading driver docker-machine-driver-kvm2:
> docker-machine-driver-kvm2....: 65 B / 65 B [----------] 100.00% ? p/s 0s
> docker-machine-driver-kvm2: 11.47 MiB / 11.47 MiB 100.00% 12.50 MiB p/s
❗ With --network-plugin=cni, you will need to provide your own CNI. See --cni flag as a user-friendly alternative
💿 Downloading VM boot image ...
> minikube-v1.22.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
> minikube-v1.22.0.iso: 242.95 MiB / 242.95 MiB [ 100.00% 20.05 MiB p/s 12s
👍 Starting control plane node minikube in cluster minikube
🔥 Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.21.2 on Docker 20.10.6 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Installing Cilium into Kubernetes
Since Cilium CLI is already installed, it is fairly easy to install Cilium into the cluster itself. The installation is done into the current kubectl context, so make sure you are running in the right context for example with kubectl get nodes. Afterwards, fire up the installation:
❯ cilium install
🔮 Auto-detected Kubernetes kind: minikube
✨ Running "minikube" validation checks
✅ Detected minikube version "1.22.0"
ℹ using Cilium version "v1.10.2"
🔮 Auto-detected cluster name: minikube
🔮 Auto-detected IPAM mode: cluster-pool
🔮 Auto-detected datapath mode: tunnel
🔮 Custom datapath mode: tunnel
🔑 Generating CA...
2021/07/13 14:09:33 [INFO] generate received request
2021/07/13 14:09:33 [INFO] received CSR
2021/07/13 14:09:33 [INFO] generating key: ecdsa-256
2021/07/13 14:09:33 [INFO] encoded CSR
2021/07/13 14:09:33 [INFO] signed certificate with serial number 122640105911298337607907666763746132599853501126
🔑 Generating certificates for Hubble...
2021/07/13 14:09:33 [INFO] generate received request
2021/07/13 14:09:33 [INFO] received CSR
2021/07/13 14:09:33 [INFO] generating key: ecdsa-256
2021/07/13 14:09:33 [INFO] encoded CSR
2021/07/13 14:09:33 [INFO] signed certificate with serial number 459020519400202498147292503280351877404424824247
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed...
⌛ Waiting for Cilium to become ready before restarting unmanaged pods...
♻ Restarting unmanaged pods...
♻ Restarted unmanaged pod kube-system/coredns-558bd4d5db-8s4f6
♻ Restarted unmanaged pod kubernetes-dashboard/dashboard-metrics-scraper-7976b667d4-ctq4p
♻ Restarted unmanaged pod kubernetes-dashboard/kubernetes-dashboard-6fcdf4f6d-5wkbx
✅ Cilium was successfully installed! Run 'cilium status' to view installation health
The installation went through flawlessly. But does it really work? As mentioned in the last line of the above listing, we can check the status of Cilium easily:
We can even get one step further and check the connectivity of the cluster – after all, Cilium is all about proper networking:
❯ cilium connectivity test
ℹ Single-node environment detected, enabling single-node connectivity test
ℹ Monitor aggregation detected, will skip some flow validation steps
[...]
..
✅ All 11 tests (76 actions) successful, 0 tests skipped, 0 scenarios skipped.
As you see Cilium creates a set of pods and a service in a dedicated namespace and runs tests on them afterwards.
Interacting with the Cilium agent
Let’s have a first look at our installed Cilium environment by running a few commands on the local Cilium agent. First we have to figure out the name of the actual Cilium pod:
❯ minikube kubectl -- -n kube-system get pods -l k8s-app=cilium
NAME READY STATUS RESTARTS AGE
cilium-8hx2v 1/1 Running 0 35m
With the name of the pod we can now reach into the pod and execute the Cilium command right inside, for example querying the list of endpoints:
This list is long, detailed and only really makes sense on a wide monitor. But it already tells us a lot about the current enforcement of ingress and egress policies (here they are not enforced as of yet).
But there is more: since Cilium is eBPF based, we can go one layer deeper, and for example look at the policy related eBPF maps:
❯ minikube kubectl -- -n kube-system exec cilium-8hx2v -- cilium bpf policy get --all
Defaulted container "cilium-agent" out of: cilium-agent, ebpf-mount (init), clean-cilium-state (init)
/sys/fs/bpf/tc/globals/cilium_policy_00208:
POLICY DIRECTION LABELS (source:key[=value]) PORT/PROTO PROXY PORT BYTES PACKETS
Allow Ingress reserved:unknown ANY NONE 16959 183
Allow Ingress reserved:host ANY NONE 1098509 4452
Allow Egress reserved:unknown ANY NONE 393706 4204
[...]
Note that the policy number is related to the endpoint ID in the Cilium endpoint list above.
We now have a running Cilium setup which can be used to run tests and examples!
Next: write and enforce policies, add observability
Doing a policy enforcement test goes beyond of the scope of this blog post – but it certainly is worth a look in the future. Also with all the data already shown above it makes sense to make a deep-dive into the topic of observation in the future.
The same is true for observability: if you wonder how deep the rabbit hole really is there is Hubble which provides serious observability into the Kubernetes network, services and security, comes with a UI and can be quickly installed since it is tightly integrated with Cilium.
And if you have stories to share around eBPF, Cilium and similar topics I am finally getting an idea of what you are talking about.
As mentioned in my last post I left Red Hat – and today is my first day at Isovalent! In my new position I will be a technical marketing manager and thus working on technical content, messaging and enablement. With Cilium Enterprise Isovalent offers an eBPF based solution for Kubernetes networking, observability, and security – … Continue reading "Hello Isovalent!"
In my new position I will be a technical marketing manager and thus working on technical content, messaging and enablement. With Cilium Enterprise Isovalent offers an eBPF based solution for Kubernetes networking, observability, and security – and since I am rather new to Kubernetes, I expect a steep learning curve.
I am looking forward to the challenges ahead of me, and will drop a blog post about it once in a while =)
Over 6 years ago I joined Red Hat. For me it was a huge step – going from a small SI to a large software vendor, moving to a dedicated pre-sales role (“Solution Architect”) in a larger team, and so on. And I learned a lot. Like, A LOT. About how such a large enterprise … Continue reading "Good bye Red Hat"
Show full content
Over 6 years ago I joined Red Hat. For me it was a huge step – going from a small SI to a large software vendor, moving to a dedicated pre-sales role (“Solution Architect”) in a larger team, and so on.
And I learned a lot. Like, A LOT. About how such a large enterprise is run, how a huge software vendor operates, but with the changing landscape in customers also how customers of certain sizes and in certain industries work. At the same time I learned a lot about how the sales process of a software vendor works, what a role a pre-sales engineer can take (and what not), and how good sales teams work. I was not without success in that role.
After my few years as a solution architect I had the chance to join the Ansible product team. This meant another big change since I suddenly stopped talking to customers on a daily base, but instead talked to the larger sales organization within Red Hat. Also, moving from a German team to a mainly US team meant a lot of changes in how my daily schedules were set up – but that worked well with the kids who arrived at the same time. The new job brought had a lot of new components for me as well: Technical Marketing Manager means to shape the product message into consumable bites for people with a technical taste. Suddenly I had to wonder how I can enable other solution architects to present this to a technical savy audience – especially if these solution architects are not product experts and do have to sell multiple products anyway. I had a steep learning curve, but again the feedback was not bad, and the constantly growing team was just awesome.
But nothing is forever: over the recent months I realized that I have growth aspirations which simply don’t fit with my position anymore. Thus I had to make the hard decision to look for something else – and I found this something else outside Red Hat. My future is still within the Open Source ecosystem, deeply connected to Linux – no surprises there. But more about that in another post.
Right now I’d just like to thank Red Hat for an awesome time. And I especially would like to thank the various teams I worked in over the time with all the people in there:
The public sector Germany team, with the simply best sales person I ever met and the best middleware solution architect I worked with.
The “Ansible workshop” crew with the greatest mind in writing workshops at the top. We rocked so many conferences and summits.
The Ansible product team when I joined – being right there when a just acquired company settles into the arms of a new owner is a very interesting experience, thanks for all the support!
The BTE which formed later on – I never worked in a team like that, but I loved almost every day. It was great to grow and thrive in this team, through all transitions and re-organisations.
Without you, I wouldn’t be the person I am today – thanks for that, thanks for all the support! All the best for the future =)
