GeistHaus
log in · sign up

https://feeds.feedburner.com/TernausBlog

atom
10 posts
Polling state
Status active
Last polled May 19, 2026 04:43 UTC
Next poll May 20, 2026 01:55 UTC
Poll interval 86400s
Last-Modified Sun, 27 Jul 2025 02:30:56 GMT

Posts

Lurking Fear: El Capitan, trip report
storytrip-reportEl-CapitanLurking-Fearrock-climbingalpine-climbingmountaneeringbig-wall-climbingaid-climbing
How Vladimir Iglovikov and Anton Karnoup climbed El Capitan
Show full content

Source: https://www.bivy.com/routes/us/California/lurking-fear–5789447656308736

This text in Russian

Following the fresh tracks, I want to write a text, which could be a helpful artifact if I climb again, as technical details will easily refresh my memory.

At the same time, I would like to capture the mood so that in 50 years, I would like to share it with my grandkids.

So, this text attempts to merge the unmergeable and combine a technical report with a story of what was, wasn’t, or should have been.

The text is lengthy and filled with many details and photos. It’s the kind of text I would have wanted to read before embarking on such an adventure. (If any terms are unclear, let me know, and I’ll elaborate.)

Short version

From October 12th to 15th, 2023, Anton Karnup and I tackled the Lurking Fear route on El Capitan.

Long version

Lurking Fear is a vertical 2000-foot-tall wall in Yosemite Valley.

This was our first successful climb together and my first big wall.

Previously, we attempted a couple of Californian peaks, but the weather did not cooperate both times.

In June, along with Dmitry Sizov, we tried the North Palisade via U-Notch, and in August, with Ya Yang, we attempted Mt Tyndall.

Both times, we spent a couple of days approaching the route, and the actual climb was expected to be relatively quick and should be completed within a day. But bad weather whispered in our ears: “Next time.”

Anton and I discussed our experiences and discovered that he had extensive big-wall climbing experience, which I lacked. However, I had a strong desire to try.

Big Wall climbing is when you tackle a wall over several days, with overnight stays on ledges or a portable platform you bring along. Apart from that, you haul water, food, and sleeping gear. A unique twist to the experience is managing bathroom breaks.

We remembered the conversation, and in September, Anton suggested we try the Lurking Fear route on El Capitan.

According to the book, “Yosemite Big Walls: The Ultimate Guide,” Lurking Fear falls into the beginner category of big wall climbing.

Moreover, the allure of climbing the legendary El Capitan was tempting.

I had known about this wall long before moving to the US, and in part, my decision to pursue graduate school in California was the proximity to Yosemite.

The wall gained widespread recognition outside the climbing community with the release of the documentary “Free Solo.” In it, Alex Honnold climbs the FreeRider route, located slightly to the right of Lurking Fear. While we had everything under control and a wide margin for error, he climbed without any safety gear, demanding absolute perfection.

The documentary, though real, feels like a thriller. You know it’ll end well, but your palms sweat throughout. If you haven’t seen it, I highly recommend it.

Naturally, I couldn’t decline Anton’s offer, and we began preparations.

Previous experience

Back in St. Petersburg, as part of the Bars alpine club, I went on climbs in the Khibiny Mountains and the Caucasus, but I hadn’t been involved in mountaineering since.

In California, I did some rock climbing, both sport and trad.

My sport climbing record stands at 5.11d, and multi-pitch trad at 5.10a.

https://www.8a.nu/user/vladimir-iglovikov/sportclimbing

Over the past year, I’ve shifted my focus to Kite Surfing and weightlifting.

Anton has had many alpine ascents and big walls.

I believe I am slightly better at sport climbing, but he is stronger in trad, especially in crack climbing, which is crucial for Yosemite.

Preparation

In the text, I’ll use two climbing terms:

Free Climbing (trad climbing) — The rock serves as the point of support. You look for hand and foot holds on the rock. Occasionally, you place protection points (cams, nuts) on the wall, clip the rope into them, and continue climbing. This means the point of support (rock) and the protection point (cams) are distinct entities. The advantage is that you climb quickly. The downside is that your skill level defines your climbing limit.

Aid Climbing: Both the rock and your protection points serve as your support. In simple terms, you can grab anything within reach. For instance, you’d place a cam in a narrow crack you couldn’t hold onto, hang an aider, rest, climb, and continue the process.

The advantage is that your climbing limit is only restricted by the presence of cracks where you can place gear. If you’re placing bolts, the sky’s the limit. The downside is the significantly slower pace.

Big Walls are typically climbed in the aid style by mere mortals like us. I had only tried this style over a decade ago and not on such challenging walls.

The ascent was planned for October, the idea conceived in September, leaving us with a month.

We made two trips to Castle Rock Park.

First, we recalled how to aid climb, and I learned how to haul a bag.

The second time, we practiced assembling and disassembling the portaledge, especially while hanging.

We also did a practice climb on the Black Wall at Donner Pass. It was three pitches without an overnight stay on the wall but with hanging belays, aid climbing, hauling, gear transfers, and rope management — a complete simulation.

It also became clear that I consume a lot of water, so instead of the planned 5 gallons, we’d need 6.

All the communal gear, including:

  • The portaledge
  • Ropes
  • Haul bag
  • A pulley for hauling
  • Nuts and cams
  • Skyhooks and Camhook
  • CT Tricky device
  • Ingenious water bottles

…were provided by Anton.

(Special attention to the CT Tricky. I highly recommend it for this route. Without it, aided pitches would be quite challenging and might even become impossible, forcing a retreat.)

Skyhooks

Source: https://www.facebook.com/photo.php?fbid=10161169024047552&set=a.10153810215727552&type=3

For those unfamiliar, skyhooks are metal hooks, you attempt to latch onto rock outcrops. They aren’t considered protection points, and falling on them is inadvisable. They can easily dislodge if pulled upward or sideways. Sometimes, they’re taped or bandaged to the rock for added stability.

Cams (Camalots)

Source: https://commons.wikimedia.org/wiki/File:Camalot_C4_number_5_placed_in_a_crack.jpg

Cams expand in cracks and serve as solid protection points, unlike skyhooks or cam hooks. According to the label, a medium or large cam can bear a load of up to one and a half tons, like my Mazda 3 car.

However, that’s if the cam is placed well. A poorly placed one can be pulled out by hand or light force. Often, you won’t know how securely it’s placed, especially if you’re placing it blindly around a corner.

Furthermore, even if you’re confident in your placement, hanging on such a device 80 feet (equivalent to an 8-story building) above your last protection point (as I’ll have later) is psychologically daunting.

Basically, you need to calmly negotiate with your self-preservation instinct and come to a mutual agreement.

Bolt

I’ll frequently mention the term bolt. Bolts are considered very reliable anchors, both for falls and for hanging.

Throughout my life, I’ve never ripped a bolt out, so as long as it’s not old and rusty, I trust it fully.

Additional Purchases

I also had to purchase additional gear:

  • Approach shoes: You need ones that work both for walking and climbing. They’re not alpine boots but are a cross between climbing shoes and hiking boots. I wasn’t aware of their existence. They’re perfect for Big Walls. Bought La Sportiva TX4. I wouldn’t recommend them. They’re comfortable but fell apart after the ascent. I returned them for a refund today. However, I endorse the approach shoe concept and will purchase a new pair.
  • Harness: For Big Walls, a regular harness won’t suffice. You need one with a broad waist belt and numerous reinforced loops for gear. I bought the Metolius Safe Tech Waldo. I wouldn’t recommend it. It’s generally good, but the belt buckle comes undone, and the gear loops on the left side tend to slide to the back, which is inconvenient.
  • Mini backpack with loops: We got the Metolius Climbing Multi-Loop Big Wall Gear Sling. It’s lightweight, compact, and has a convenient attachment point for a radio. Suitable for any route type. Recommended.
  • Radios: We took the advice from the HowNot2 channel and chose the Rockie Talkie and liked them a lot. They come with an inbuilt carabiner and can be attached to the backpack strap. The rubberized coating makes them less fragile. Plus, unlike other radios with constant static and chatter from other people, these didn’t have any of that. Recommended.
  • Aiders: I hadn’t climbed with aiders before. I had a few climbs in Russia using Kurkonogi (Russian aiders). Krukonogi isn’t available in the US, and everyone uses normal aiders. Initially, I ordered from Scotts Wall Gear. They charged my card but haven’t shipped the order yet. Ultimately, I used two Yates Big Wall Ladder 6-Step Aiders. I have mixed feelings about them. They’re good, but I believe there are better options out there.
  • Headlamp: I needed a bright one since there’s a risk of ending up in the dark. With shorter daylight hours, it’s not just a risk but a plan. I chose the Petzl Actik Core 600 Lumen and liked it. It’s not necessary for casual evening outings but vital for climbing after sunset. Recommended.
  • Phone: I wanted to capture the ascent with photos. Coincidentally, the iPhone 15 Pro had just been released. I was enticed by the promise of stellar photos and bought it.
Krukonogi (Russian aiders)

Source: https://risk.ru/blog/202389

Krukonogi has two parts:

  1. Aiders that is a chain of rings.
  2. Hooks near the knees that you use in these rings or any carabiners.

Aiders part of Krukonogi — a chain of rings that could be attached to the carabiner

The hook part of the krukonogi. A hook near the knee is used on the rings from the ring ladder or carabiner.

Advantages of Krukonogi:

  • placing the hook into the ring or carabiner is faster than a foot into the aider or etirers
  • on the vertical and positive slopes, you can reach about 1 foot higher Disadvantages:

  • Shifting to free climbing and back is more challenging.
Ascent Wednesday

On the evening of October 11th, we headed towards Yosemite and camped just outside the park. The rules for staying overnight in the park are peculiar: rangers chase away those attempting to sleep inside without permission. However, a mere 100 meters outside, they don’t mind. They even recommend doing so.

Thursday

Objectives:

  1. Carry our gear to the base of the route.
  2. Climb the first three pitches.
  3. Haul up the bag.
  4. Hang the ropes down to the ground and leave them.
  5. Descend and enjoy some beer.
  6. Sleep in the car.

It’s worth mentioning that during this week, park workers were combating dry trees by extensively burning them. The surroundings weren’t just slightly smoky; they stung my eyes, and my watch indicated that my blood oxygen level (Pulse OX) had dropped.

*This isn’t morning fog; it’s intense, eye-stinging smoke.

The approach to the route is comfortable, taking just an hour from the car. We planned to do it in two trips. However, Anton’s friends, Jerome and Emil, kindly offered help, so we managed in one go.*

There are places on the approach where climbing is necessary. Fixed ropes are in place. It’s hard to see here, but there’s a rope in the center, and starting without it, especially with a backpack, is quite tricky.

That’s me. Notice the white sky on the right? Without the smoke, you’d see trees and cliffs on the other side.

At the base of the route, we met the valley legend Steve Schneider (Shipoopi). He was soloing a route about 100 meters to our left, aiming to finish by winter.

First Pitch

We overlooked a detail in the description about hooks, which forced us to pull up the missing gear.

Anton approaching the first belay point.

Second Pitch

Again, Anton is climbing on bolts.

Start of the second pitch. The rock’s like a mirror. Yet, some people free climb it here.

Third Pitch

Two options for progression. No one takes the right one; we didn’t either.

This one was mine.

The start involves bolts. I managed the first four, and then it got interesting. I couldn’t reach from the fourth to the fifth, even with an extender (CT Tricky). A very height-dependent move. After about ten minutes, I found the right combination of stepping and micro-swinging amplitude to succeed.

I got very curious about how climbers shorter than me navigate this section.

The next move was even more intriguing, described as “12d or 5.10A0.”

How it looked in practice: there were micro holds that, when touched (but not gripped), allowed some balance, enabling me to ascend higher on the aider.

A foot above my stretched hand was a hold, but this foot seemed impassable.

First idea: I tied a skyhook to my self-belay and tried to throw it precisely to latch onto the hold. Theoretically, this idea could work, but I’m not trained in skyhook tossing. So, I moved to plan B.

Second: Try balancing in climbing shoes. Anton hauled them up, and I switched shoes and tried. I did not have a third idea, and I’m glad that one worked. The climbing shoes improved my balance, letting me ascend higher. Then, in a semi-dynamic move, I switched from the aider to the minuscule hold and hooked a skyhook onto a large hold I aimed for.

After that, there’s a straightforward section of 5.9. I intended to switch to free climbing, but it didn’t pan out. Continued aiding. The aiders tangle around your feet. You have lots of gear, and, probably the most important, psychologically, switching from relaxed hanging to less confident climbing is hard.

End of the third pitch. We had to leave all our gear on these two bolts, hang the ropes to the ground, and head down.

At the same time, a group of three descended past us. They had climbed the first six pitches in a day, decided they were moving too slowly and turned back.

This surprised me. The guidebook recommends climbing seven pitches on the first day. They were on track.

According to Shipoopi, from observing the base of the route for a few months, roughly half the groups withdrew before finishing.

View from the ground of our hanging ropes. From this spot, our belay station with the bag, platform, and other gear isn’t visible; it’s 380 feet above.

We met a guy starting the route alone at night when we descended.

As we later read in the news, this was 25-year-old Oliver Tippett, another climbing legend from Britain.

He was going for an onsight (never climbed this route before) speed climb.

He finished in 17 hours and 26 minutes (compare that to our 3+ days).

After that, we enjoyed beer, pizza, and a good night’s sleep before our decisive push.

Friday

The task for the day: Ascend to a comfortable camping spot.

The guidebook suggests that those starting from the end of the third pitch (like us) climb another 7–8 pitches, and from there, a repel to a ledge where two people can fit. This ledge is known as the “Motel Zed.”

By 8 a.m., we had ascended to the start of the route. While not the end of the world, it is an area for improvement. The days are short; it’s better to start climbing at dawn, which is at 7 a.m.

Another team was already climbing, a man and a woman. She was leading and was almost at the first belay station.

To get to where we left our gear, we had to ascend 380 feet using ropes.

Anton went up first. When it was my turn, I realized a gap in my skillset. Throughout my climbing career, I always led climbs. Therefore, I never mastered the jumaring technique.

In theory, I understand and can do it, but it’s energy-consuming. This method works fine for short distances, but 380 feet on a vertical wall is exhausting. Ideally, this process should feel like a rest.

Our ascents, gear setup, and preparing the ropes for Anton to climb and for me to belay took an hour. It felt like a long time and another area for improvement.

We began the climb at 9 a.m.

Pitches 4, 5, 6

Anton led these. I heard no swearing; he combined pitches 5 and 6 into one. The climb started with bolts and skyhooks, then along a crack.

Shipoopi is working on his climb.

Valley in smoke

Seventh Pitch

I led this pitch. The rope was mostly horizontal, implying a traverse. Traversing in big walls is quite different from free climbing.

In free climbing, you lead, clipping into bolts or placing the protection. The second climber follows and removes the gear. This is exactly how I climbed my last traverse on the “Sedona Scenic Cruise” route three years prior with Galina Malovichko.

In this scenario, there are several protection points between you and the belayer. If you fall, you don’t drop far.

In big walls, it’s the opposite. There shouldn’t be any intermediate points between you and the belayer. The main rope isn’t clipped anywhere during the traverse until you start climbing upwards.

The lead climber uses two personal anchors. They clip into the first bolt with one, then reach to the second and clip it with the other. They unclip the first from the initial bolt, reach the third, and so on.

If you fall, the nearest point is the belay station, and you swing the full length of what you’ve climbed.

The start of the seventh pitch. Two bolts are between me and the anchors, but the rope is not clipped into them.

What if you can’t reach from one bolt to the next? You place the gear in a crack — a cam or a nut.

And if there’s no crack? You use a skyhook.

This is the exact situation I found myself in. I moved four bolts away from the anchors, with no intermediate points between me and the station. I couldn’t reach the fifth, so I had to place a hook and trust it wouldn’t give way.

In total, I had to traverse about 20 meters in this manner.

Very exhilarating.

Meanwhile, cheerful shouts echoed from below.

Remember how I stretched somewhere on the third pitch and wondered how those shorter than me lead that move?

The woman we saw at the start of the day reached that spot. She couldn’t reach far enough, so she tackled the section using the “pendulum” technique, swinging with a significant amplitude of about 12 feet. She attempted this more than ten times with childlike glee and shouts of “yippee,” reminiscent of childhood swings.

But these swings were 260 feet above the ground. For visualization, imagine a 26-story building and someone attached a swing at the attic level. You run along the wall with a joyful scream and try to swing as far as possible. She was quite tenacious and managed it after several attempts.

Also, on the seventh pitch, Anton dropped the belay device.

For the rest of the route, we climbed using mine, being extra cautious not only with the pulley to haul the bag (if it were dropped, we’d have to halt the ascent) but also with the belay device. Had the remaining one been dropped, we might not have turned back, but the excitement level would have notably surged.

Eighth Pitch

The start was straightforward.

I led and once again encountered a new technique, which I had to adapt to without any prior familiarity.

The second part of the pitch is an 80 feet offwidth crack about 4 inches wide.

Offwidth refers to a crack wider than a fist but narrower than a “chimney.” (A chimney is a crack you can fit into entirely and ascend by pushing against both sides.)

If you’re curious about the most “weird” type of climbing, it’s where the offwidths are.

However, we didn’t need to free climb; aid climbing changed everything.

A 4-inch cam fit well into this crack, which was good news. But we only had two of them, which was bad news.

How to climb? A technique I suddenly got acquainted with is called “Cam Jumaring.”

You place the first cam, attach an aider, ascend, place the second cam, attach another aider, climb it, remove the first cam, place it higher, and so on.

Where are the intermediate protection points? Where do you clip the rope?

Nowhere, just like traversing on bolts horizontally. Here, it’s on cams that you keep moving vertically.

In traverse, you do this for “high style” and “because it’s the way to do it.” You do it here because you have no choice, and “life makes you adapt.”

If the cam you’re hanging on pops out, you fall — and fall far.

Eighty feet is like an 8-story building. If you fall, you drop to the last protection point and then fall that distance again: 2 * 80 = 160, like falling 2 * 8 = 16 stories.

Another thrilling pitch.

Ninth Pitch

Mine again.

The pitch wasn’t tough, rated 5.9; ideally, it should be free-climbed. That means leaving the extra gear with Anton, removing the aiders, and moving forward.

But! It was getting dark, my energy was waning, and I wasn’t keen on switching from slow, comfortable aid climbing to physically and emotionally demanding free climbing.

I chose safety over speed and climbed using aid. The pitch was not difficult, with a good crack and even some holds around it.

I finished in the dark.

It’s somewhat easier at night; the height isn’t as intimidating when you can’t see the ground.

The end of the ninth pitch is the “Pillar Of Despair,” the best ledge we encountered all day, about a square meter in size.

You couldn’t lie down to sleep on it, but after a whole day on hanging belays, the sensation of a horizontal surface where you could stand with both feet was sheer joy.

The guidebook suggests not bivouacking there but climbing 1–2 pitches higher and repelling diagonally to “Motel Zed.” However, we felt that perfection is the enemy of the good.

We set up a portaledge, a new experience for me. We secured the corners, a technique I didn’t know was necessary.

We prepared food, which meant boiling water on a hanging stove, and slept.

I was curious if I would roll off in my sleep. And if so, how far over the edge would I go before waking up? We slept in harnesses tied to the rope, ensuring safety. But still, if I had slipped off, it would’ve been a night-time, mid-sleep fall.

Sometimes, I fly in my dreams; this situation would have added a unique twist.

Fortunately, that didn’t happen, and I woke up right where I fell asleep.

Against the dark backdrop, or rather below, were the lights of fires.

Saturday

The goal was to reach a large ledge named Thanksgiving Ledge at the end of the 17th pitch, meaning if we climbed six on Friday, one of which was in the dark, we needed to climb eight today.

We woke up, had breakfast, and got ready. Again, for some mysterious reason, we started climbing at 9 a.m. The sun rises at 7, after all.

Tenth Pitch

Anton led. From a distance, it looked easy, but only from a distance. The two consequent moves Anton had to make on camhooks were impressive.

Eleventh Pitch

Anton led. While climbing, we heard cheerful shouts below us and saw the next group reach the Pillar Of Despair, where we had bivouacked two pitches below.

I thought it was the couple that had been two pitches behind us, but as I found out later, it wasn’t them; that couple must have bailed.

Twelfth Pitch

I led. Another traverse, but unlike the previous one, which had bolts, this one had rusty pitons. And when the hooks ended, there was a crack where cams could be placed.

Yes, the crack was good, but it was quite an experience to hang on a single cam 10 meters away from the last protection point.

I wasn’t scared, but I’d like to use the words “nervous” and “tingling” side by side.

From anchors, I climbed up to a roof with a piton underneath. After that, I placed a cam, followed by two more pitons, and then two more cams. I’m hanging on to the third one. Everything I clipped, I removed after passing. This is precisely what “high style” aid climbing looks like on traverses.

From Anton’s perspective, it didn’t look that dramatic. I’m about 400 meters above the ground, higher than the Eiffel Tower.

It wasn’t that hard. Yes, it’s unsettling to be several meters away from the belay and hanging on 1–2 pieces of your gear. But it probably felt easier than the previous traverse on skyhook.

Thirteenth Pitch

I led again.

A third of the pitch was 5.12b, another third 5.8, and the last third 5.3.

I aid climbed the first part, a good finger-width crack, for which we had plenty of gear.

Then, I had to transition to free climbing. And again, the same story — it’s just about 25 feet, the aiders get tangled around the legs, a cam popped out of the crack prematurely. Basically, I chickened out and aid climbed, which was a small but time-consuming issue (and, again, an area of improvement for me).

While climbing the 5.8, a female team caught up with us. First, Kate sped past me, followed by Michele.

By the time I reached the belay, the first lady had already taken off, and the second one was preparing to ascend.

These strong women work for the local Search and rescue. They had a day off and decided to stretch their muscles by climbing our route in a single day.

They started climbing at 8 a.m. I didn’t ask, but most likely, they are also some legends of Yosemite Valley.

From the end of the 13th pitch, there was a feeling that life was getting back on track, and the vertical part was over, probably because there was a ledge where we could stand on both feet.

It’s the first time today we’re standing on two feet, not hanging on the wall.

Fourteenth Pitch

Anton led, and he free-climbed. He finished in less than half an hour. Compared to the 1.5–2.5 hours each of us took before, this was impressive and gave us hope that we’d complete another pitch before dark.

Fifteenth Pitch

It was getting dark.

Anton led again. While we were handling the rope, his headlamp suddenly went out. Everyone recommends bringing spare headlamps on a climb, and it’s advice you shouldn’t ignore! A backup headlamp is crucial.

The beginning of the pitch was 5.6, which he breezed through without placing any protection. Then came the 5.10c section and the sunset (7 PM). Free climbing was over, and aid climbing began. We sent up some of the gear, especially the hooks he’d left behind, to speed things up.

Again, it took 1:30 hours for this pitch.

Sixteenth Pitch

I led, and it was already dark. My headlamp was acting up and wouldn’t switch to maximum brightness, only working on a medium setting. It probably didn’t slow me down much, but it would’ve been better without this issue.

I climbed 5.6 + 5.10b, wearing climbing shoes. I free climbed the 5.6 without placing gear, and aid climbed the 5.10b. I strongly felt that this pitch could and should be free-climbed in the daylight, but in the dark, I wasn’t enthusiastic about the idea.

Seventeenth Pitch

The last pitch for Saturday, at the end of which awaited a ledge and a cave.

Anton led.

The first section was an offwidth 5.11d). Similar to what I had on the eighth pitch, where I had to move two 4” cams, one above the other.

The challenge was that this crack was slightly wider, requiring two 5” cams. We only had one.

With some magic of climbing, Anton made it up, losing a water bottle along the way (which I later picked up). And then, there was the magnificent ledge with a cave named Thanksgiving Ledge, which we had been aiming to reach.

I got to it just after midnight. So, the last three pitches took 4.5 hours.

Thanksgiving Ledge

A massive ledge with a cave, so spacious that we slept without harnesses. For this bivouac, we had brought along some beer and snacks.

Our exhaustion was evident, leading to some oversight. When I’m extremely tired, I snore. Due to the cave’s acoustics, the snoring was amplified. This meant Anton slept poorly, as sleeping inside a running tractor wasn’t ideal. At the same time, I had earplugs specifically for this situation, and we both knew about them.

By the way, this wasn’t my only oversight of the day. Up to this point, I had dropped two slings and one carabiner. I don’t recall ever losing gear before, but here, I lost three items in one day.

Sunday (the last day)

We woke up, had a snack, and realized we had significantly more food than needed. We also had ample water, so we left one gallon in addition to the water left by previous groups.

Eighteenth Pitch

Another blunder.

The end of the 17th pitch and the beginning of the 18th are in different locations. We saw two anchor bolts to the right of the cave, where the number 17 is marked on the map. The fact that there is no two bolt station in the topo didn’t alarm us. The rock not matching the description didn’t either. We didn’t think to walk around and check the surroundings. We also overlooked that pitches from a big ledge seldom have bolted anchors.

So, Anton started climbing from there. A few meters above the anchor, a foothold broke off, causing him to fall. Thankfully, he wasn’t injured, but it was surprising to encounter such an issue on such a solid and popular route.

On his second attempt, he climbed further and grabbed something, which gave way, sending several rocks tumbling down. While he didn’t fall, such moves could endanger the groups below.

We then wondered if we were climbing the right way. Turns out, the correct start was 25 feet to the left.

Descending from the wrong start cost us three cams. They are probably still there.

From the correct start, the climbing was smooth and straightforward, free climbed.

It’s worth noting that the guidebook listing all the bolts on the route, except these two, isn’t great. For those who will climb this route, be extra cautious here.

Regardless, the fact that we climbed the wrong way and thoughtlessly relied on those two bolts indicates that our overall fatigue was setting in, and our intellectual quotient was declining.

Nineteenth Pitch

I led, and it was straightforward. Halfway up the pitch, I placed a cam but didn’t clip it to the rope and lost a carabiner without noticing.

So, I climbed the entire pitch without intermediate protection and lost a carabiner. We would’ve left the cam behind if Anton hadn’t spotted it on the side.

Another oversight.

After that, it was straightforward: pack the gear, walk to the top, and hike down. Got to the top at 2 p.m.

At the center of Half Dome. Last year, I free-climbed it via the Snake Dike route, 5.7, over eight pitches.

The wall is in the background. That’s the one we climbed.

The descent from the top took 5 hours. At one point, we had to rappel using fixed ropes, which took slightly longer with one belay device compared to two. However, since we had plenty of time, this was seen not as a problem but as part of the adventure.

During the descent, we met a guy (sadly, I didn’t remember his name) who had climbed The Nose on El Capitan (32 pitches) that day, starting at 8 AM. I wished I’d taken a selfie with another living legend.

Conclusions

Surprisingly, aid climbing felt psychologically more comfortable than free climbing 5.9, despite the traverses, hook exits, night climbing, and overall height.

In aid climbing, you don’t hang onto tiny holds with the last protection point way below, trembling legs, trying to place gear, struggling to find it on your harness, losing strength, and the situation worsening with every second.

Clip in, hang, ascend the ladder, place a new piece, clip—hang, hang the ladder, and repeat a thousand times. It’s joyous, comfortable, and has no shaky legs.

On the other hand, the journey is constantly draining: hauling the bag, jumaring, moving with all the gear and two ropes.

This was my first big wall climb, and it went well.

However, after every adventure, it’s beneficial to analyze to improve future endeavors, no matter how minor the improvements could be.

I’m considering attempting The Nose in the spring. It’s also beginner-friendly. It’s 50% longer at 3000 feet, but there are no C2+ moves, and according to Anton, who’s climbed it twice, it’s more straightforward.

Covering more distance in a day; there’s room for improvement here.

Some optimizations could be attributed to climbing during a season with longer daylight hours. This time, we had exactly 12 hours.

If we break down the time spent on each pitch:

1.5–2.5 hours to lead with aid or 30 minutes of free climbing. Thirty minutes to haul the bag, the second to jumar, deal with ropes, and hand over gear. The hassle with hauling and ropes is memorable, but the primary optimization should focus on the lead climber’s speed.

Climbing Optimizations
  • Get krukonogi, potentially increasing speed by 30–50%.
  • Train to switch between free and aid climbing. I lacked in this area. When I did, it was slow, and I didn’t do it where I should have.
  • Be more mindful about the gear taken on each pitch. Knowing the gear required, I wouldn’t have taken nuts, only two skyhooks, three cam hooks, and lighter 5” and 4” BD ultralights.
  • Use only Totem cams in the [0.5–1] range, no BD, Metolius, or Aliens. Though the guidebook recommends taking three pieces of some sizes, this looks overkill. Two of each except 5” feel like good enough. I did not use half of the cams that we had. Less weight speeds your climbing, it is easier to look for gear on the harness, and passing gear to the other person is also faster.
  • All carabiners should exclusively be HMS.
  • Invest in a better big-wall harness. The one I had kept slipping, affecting gear distribution and making gear searches longer.
  • Introduce color differentiation for cam carabiners. I bought 24 color carabiners for Anton’s cams. But it is far from enough. Improve crack climbing skills. Indoor climbing gyms have numerous challenging cracks, and California is rich in crack routes. Also, reduce body weight. For this trip, I weighed 88kg (195lb). Hauling was easy, but climbing felt heavy. Surprisingly, I returned home at the same weight, even though legends say one burns 3000–5000 calories daily on a Big wall.

Overall, these changes could speed up the process by 50%.

About the Routine
  • Lighten everything possible.
  • Get a lighter platform and bag.
  • Choose lighter food, but a can of beer is a must.
  • Improve rope ascent techniques. There’s room for me for improvement. Techniques vary based on the wall angle. I need to master them all.
  • Improve rope management at stations. We were always extra cautious and coiled repeatedly, wasting time and energy.

Big wall climbs are cool. And those for beginners don’t require a high climbing level.

I highly recommend it to everyone.

https://ternaus.blog/story/trip-report/2023/10/23/El-Capitan_Lurking-Fear_Trip_Report
From Albumentations to Image Search
storyalbumentationsimage_search
Started with the feedback for Albumentations, ended with an image search..
Show full content

I wrote a Web App that performs Image search on the Open Images dataset (1.7M images).

Here is the link.

You upload an image or perform a text query, and it returns 18 similar ones.

Questions:

  • I want to use the service as a part of the pipeline. Can I get access with an API and not a web app?
  • 18 is good for a demo but not enough to enrich my training set. Can I get 1000 similar images per call?
  • We care about the privacy of the data. Can we have an image search on our data and hardware?
  • Can we get an image search on some other dataset? COCO, ImageNet, something medical?
  • We want to use the data to train models. Can the service return segmentation masks, bounding boxes, and class labels in addition to the images?
  • It works fast on 1 million images. Will it be fast on 1,000,000,000 images?
  • Right now, we perform a search using a sample image. Can we use text to describe what we want?

  • The answer to all these questions is a solid yes.

I need to admit that it is unclear how image search will work with other domains. At the moment, everything is designed to work on natural images.

To be applied to medical or satellite, I will need new models, and I do not have them in front of me. If there is interest, we can explore this option.

I have a request — if you have an idea how your product may benefit from an image search, do me a favor, and write in the comments or message on LinkedIn.

All ideas, including wild ones, are welcome.

Right now, I have in mind:

  • On public data: Company has a limited amount of data. The model is not performing well on a set of images in the test set. The company uses API to get 100 similar images for every “hard” sample, label them, and add them to the train set.
  • On your data: The same as above, but on the company’s private data.
  • On your data: Faster data labeling. For every image, you find a set of similar images. If all belong to the same class, you add a label to all 20 images in the cluster. => 20x to the data labeling performance.
  • On your data: QA for data labeling. When I checked data labeling quality at segmentation masks or bounding boxes on the image, I was unsure if it was perfect or just good. But looking at the cluster of similar labeled images made the limitations of the labels evident.

An important part is over. The rest is just for the logging.

Q: How did I get the idea to implement a service for image search?

In my free time, I participate in developing the Albumentations.AI.

Albumentations is an open-source library for image augmentations.

For more details on how the library was born and evolved, feel free to check — The Birth Of Albumentations.

At some point, we had five members in the Core Team.

A couple of years ago, the library became mature, and how we work on it changed.

Originally it was a fun project we did for ourselves, but after it became popular, it started resembling maintenance and support.

At this point, Alexander Buslaev (the library was named after him ) left the project. I started to invest much less time as well.

And then there were four of us.

In February, Russia invaded Ukraine, and Eugene Khvedchenya, who lives in Odesa, Ukraine, left the project. When Russian rockets hit buildings around him, he shifted his focus from the library to things that are more important in life.

And then there were three of us.

The shock of the war hit as hard as well. We did not look at the library for months.

Even more, we started the library because we needed it to train ML models, but in the past years, we moved to other activities.

By May of 2022, we had at GitHub:

On the one hand, the project was quite popular; on the other — it had all chances to die.

We needed someone to pick the work up.

First idea: we were thinking of finding a motivated ML Engineer who will work on the project as we did at the beginning. The problem is that motivation comes and goes, and we would need someone new at some point.

Second idea: Add a new person to the Core Team. As I wrote in The Birth Of Albumentations, we invited to the Core Team people who actively contributed to the project with high-quality Pull Requests. We contacted one such person, but he was busy working on his Open Source project.

Third idea: Find $5000 per month, and hire a person to work on a project full time.

I asked around about how to find money in such a situation.

There are two ways:

  • Donations.
  • Build a company on top of the Albumentations in an Open Core way. I.e., the library is free as it is now, but there are paid features like privacy or security that are needed in the enterprise. Another new thing was that I and startup founders think about the value of the library differently:

I:

  • Blazingly fast.
  • Great functionality.
  • Wide range of transforms for different domains.
  • Easy to extend.

Startup founders:

  • Brand
  • Community
  • The fact that the library is already integrated into thousands of companies.

For a few weeks, I interviewed people who use the library in production.

I collected the feedback (blog post with the summary — Albumentations: Feedback) and looked for pain points that could be addressed with a product.

No luck. I could not find an idea on how to use augmentations to solve significant pain.

I think the main reason is that building tooling for developers is a horrible business idea.

A better one would be to build a product for the collaboration of developers.

An even better idea is a tool for collaboration between all kinds of people, not limited to developers.

I looked at the bigger picture.

Q: Why do we need augmentations?

A: Because we have a limited set of relevant labeled data.

Q: How else can we address the issue?

In practice, if such an option exists, everyone is just labeling more data.

It is relatively common to have a lot of unlabeled data, and you need to figure out what data to label.

After you have trained a model and checked its performance on the validation set, you have a collection of validation images on which the model is not performing well. Let’s call them hard samples.

The assumption is that if we add more data similar to these hard images and retrain the model, the model will improve its performance.

“Similar” is not a well-defined term. It could be:

  1. Way the model performs. Image A and B are similar because model XXX is not performing well on both.
  2. Color distribution. I had a situation when the train set was collected in summer, but in fall, it became darker, and model performance decreased.
  3. Semantics: Images A and B are similar because they have the same set of objects. For example, a raccoon.
  4. Any other distance metric. What type of similarity to use depends on the problem.

The first case is called Active Learning, and almost all ML Engineers implement something like it at some point in their career. By the way, augmentations add a lot of value here.

Right now, I would like to figure out how to adapt image search to add value to people.

For those who got to this point, I ask for a favor. Could you please go to the App and play with it? If you get an idea of how it could be valuable for you or someone else, please message me. All wild ideas are welcome.

I know that it is the technology in the search for the product which is anti-canonical, and that is why I am looking for pain :)

P.S. Russian version of the text.

https://ternaus.blog/story/2022/08/10/From-Albumentations-To-Image-search
Albumentations: Feedback
storyalbumentations
Feedback for the Albumentations Library.
Show full content

Warning: This text is dry, and its primary purpose is logging. It would be mainly helpful to those who already use the library.

I am one of the core contributors to the Open Source library Albumentations.

The library is used to augment images and is typically used to train neural networks.

To train neural networks, you need a lot of labeled data.

There are two ways to address this:

  • Collect and label new data. It is expensive and slow but gives solid value.
  • Augmentations. There is typically a value, but you do not know how much and how to choose them correctly. But you get them for free and right now.

In practice, you use both methods.

We have worked on the library for more than four years. Last year I wrote a long post on how the library was born and how we promoted it. [ The birth of Albumentations ]

Current traction:

There are a few different reasons why the library took off. Functionality and good promotion are one thing, but the most important is performance.

From the authors of the Augly, developed at Meta at CVPR 2022.

We developed the library as a typical open source — the core team in different countries. When someone wanted to add a feature, he did this without planning, syncs, or OKRs.

After all these years of development, I decided to perform Customer Research and ask people how they use the library and what is missing.

I talked to 20+ people, and that is what I got:

Q: In what domains is the library used?
  • All kinds of medical imagery.
  • Document recognition.
  • Satellite imagery.
  • Autonomous vehicles.
  • Robotics.
  • Defect detection in the factories.
  • Face recognition and antispoofing.
  • Photo editing and enhancing.
  • Road traffic monitoring.
  • Beauty industry
  • Retail
  • Model testing and robustness to the data drift.
  • Video processing from sports events.
  • Mem generation.

If people need to train models on images or videos, they use Albumentations.

Q: What types of problems do people solve with Albumentations?
  • Classification
  • Object Detection
  • Semantic and instance segmentation
  • Action recognition
  • Pose estimation
  • Super Resolution
  • Image inpainting
  • Matting

Initially, we developed the library for classification and semantic segmentation. We added the rest of the functionality later. And it looks like it was worth it :)

Q: What is lacking?
  • Images with examples to the documentation.
  • Instruction on how to add new transforms.
  • GPU support. There are Image Augmentation libraries that work on GPU, say DALI or Kornia, but they lack much of the functionality that Albumentatons have.
  • 1D and 3D support. Video processing (XYT) and real 3D, as in medical imagery (XYZ) become more common. There are libraries such as Monai and Volumentations, but they are far from perfect.
  • Export the augmentation pipeline from python to other languages such as C++ or Swift.
Q: What new transforms could be added?
  • Add random text to images. Similar to what TextRecognitionGenertor does.
  • AugMix, CutMix, MixUp.
  • CutAndPaste — you have an object, and you need to add it to various background images filling the boundaries with inpainting or Poisson Blending.
  • Add a mask or glasses to the face. As I understand, you find facial key points and use them.
  • More ImageCompression transforms. We now have Webp and Jpeg formats, but we could add heic and jpeg2000.
  • Mosaic Transform — you combine a few images into one big one. It is used in Yolo V4.
  • In all dropout transforms, do not zero out but convert target pixels to the grayscale.
  • Transform regions only inside of the bounding boxes.
  • Transform regions inside of the specified mask. Use case — you need pants of various colors. You segment them beforehand and apply color transforms where the mask is nonzero.
Q: What non-standard transforms work well?
  • In the tasks where you have an imbalance of the people with different skin color, say a lot of middle-aged white people, but not enough young black FDA transform works well.
  • FDA, RandomSunFlare, and RandomFog in document recognition.
  • ImageCompression in all situations where you have an old cell phone or bad camera images.
Q: How do people choose what augmentations to use?

There are different ways, and there is no one best practice so far:

In general, there is a feeling that the task of choosing a proper augmentation pipeline is not solved yet. It is a business opportunity to create a tool that takes a training dataset and returns a strong augmentation pipeline.

But there are a few problems:

No one knows how to choose such a pipeline. There is research in the domain of AutoAugment, which is a subset of even more popular AutoML, but it is in its early stages. Augmentations depend not only on the dataset and task but also on the model. The heavier model you use, the stronger augmentations you can apply. Augmentations add to the accuracy and generalization ability of the model. The smaller the dataset — the more value they add. The problem is that it is unclear beforehand how much value will be added if any. Returning to the library, we have good documentation and functionality, but much could be done.

If you want to contribute to Open Source and have never done it before — you are welcome. We have a lot of simple tasks for you :)

In general, we would like someone to work full-time on the project, but it looks like we need to set up a donation system. It seems like with the current traction, we may succeed in this :)

P.S. Russian version of the text

https://ternaus.blog/story/2022/08/02/Albumentations-Feedback
Unified Sports Classification System
storysport
Unified Sports Classification System.
Show full content

When I came to the U.S., there were many small things that I needed to get accustomed to.

One of these things was the lack of a unified sports classification system.

I tried different physical activities from childhood — running, swimming, soccer, rock climbing, weight lifting, and many others.

In all the communities related to each activity, you could hear things like: “Did you hear? Vladimir finally became CMS, and I need a few more years to get there”.

This classification was developed in the Soviet Union a long ago but is so successful that even now, when the Soviet Union is a history, it is still popular in Ukraine, Russia, Kazakhstan, Belarus, and all other former Soviet Union Republics. People in these countries could have quite the opposite political and different views, but you can ask any person from these countries and, for sure, they at least have heard about it.

Requirements for every sport are developed by ministries of sport in various countries or by Federations that do not have government affiliation. A lot of research is put into them and a whole set of PhDs is defended every year on the way :)

There is no such system in other parts of the world, say in the U.S., where I live now.

What is the main idea?

We introduce a set of titles to every sport or physical activity: MSIC, MS, CMS, first/second/third class sportsman. Some sports may also have a junior version of I/II/III grades.

To get the title, you need to complete a set of requirements. They are different for men and women and may also depend on the person’s weight.

https://sportscategory.info/

You may also think about it as a way to “compare apples to oranges.” For example, CMS in swimming Galina put more effort into her training than third-class sportsman Vladimir, a third-class sportsman in Olympic Weight lifting.

There is some gamification element in this as well. In computer games, you collect badges like: “Killed five black dragons,” while in real life, you can collect “was able to satisfy requirements for XXX grade in YYY sport.”

Typically, almost everyone can become a second-class sportsman in every discipline. It could be challenging for higher grades if you do not have proper genetics or do not want to use pharmacology.

I had a discussion about this classification with my English speaking friends so many times that finally, I decided to translate the requirements for:

You can find them at https://sportscategory.info/

I want to clarify that satisfying the requirements for a title is not the same as getting the title, and you get the title only in the official competitions.

But for some sports, you can try to satisfy official requirements by yourself without any bureaucracy. You will not have a paper that proves how cool you are, but who cares? Make a video of your records, and it will be more than enough :)

(I do not know how to embed from Instagram, so here is just a link to the video for my 245lb benchpress )

My “badges” :)

  • I Class Sportsman in Mountaineering. (I did it 13 years ago)
  • II Class Sportsman in Powerlifting (I can do it now)

  • Currently, I am working on becoming a third-class sportsman in Benchpress and weighted pull-ups.

P.S. Could you please write in the comments requirements what sports and physical activities I should add to the website and, what is more interesting, requirements for what titles and in which sports do can you do?

https://ternaus.blog/story/2022/06/06/USCS
The birth of Albumentations
storyalbumentationsopen-source
About the birth and growth of the albumentations.
Show full content
Table of Contents

I would like to talk about Albumentations, an open-source library for image Augmentations. An explanation of how it was born and evolved over time. The text’s primary purpose is to depict its creation, as I would like to have a story to enjoy in 20 years.

I will mainly talk about the library’s origins and cover the process of iterative improvements in fewer details. The text could be more concise, but I prefer to keep unnecessary information for historical purposes.

Albumentations is another python library for image augmentations. It is widely used in Deep Learning tasks, especially with PyTorch.

  • Q: What are image augmentations?
  • A: Image augmentation is a way to generate a new image from the one you already have. Typically, it is something simple as rotations or reflections but could be more involved, like adding rain or applying style transfer.

  • Q: Why do we need this?
  • A: Neural networks trained on larger datasets generalize better.

One way to increase the size of the dataset is to collect and label more relevant data. It works well but is expensive and could take some time.

Another way is to apply augmentations. They are not as valuable as adding new data, but you get them for free.

Two methods are not mutually exclusive, and in practice, you combine them.

With augmentations, you extend the pipeline of:

Read image from the disk => feed to the neural network.

with an extra step:

read image from disk => randomly augment it => feed to the neural network

In most computer vision pipelines, the first two steps happen on CPU and the last on GPU.

June 2017 Kaggle organized the machine learning competition Planet: Understanding The Amazon from Space. Participants needed to create a model that assigns tags to images: forest, river, road, etc.

The task was rather dull, very similar to the MNIST classification. Discussing it at ODS.AI, a few experienced Kaggle competitors and I decided to join the challenge.

The problem was not exciting but did not require much work. End2end pipeline could be implemented in one evening. After that, you needed to experiment with hyperparameters and augmentations, which could be done passively.

Augmentation is an excellent regularization. The more augmentations I added - the better my networks behave.

We finished 7th out of 936 teams. We did not get any money, but titles of Kaggle Grandmasters became closer :)

Regarding image augmentations, three things were important:

  • The mainstream opinion was that the bottleneck in training was GPU. The more powerful GPUs you have - the faster is network training.
  • In the DSTL Satellite Imagery Feature Detection challenge at Kaggle, my team finished 2nd, and I bought a home computer with four powerful GPUs (for 2017).
  • For augmentations, we used the open-source library ImgAug. It has a lot of powerful transforms, but it is not optimized for fast execution.

The bottleneck in the Amazon Challenge was the CPU, while GPU utilization was far from 100%.

All team members, including Alexander Buslaev, started building custom augmentation pipelines, trying to make them flexible but fast.

Four months later, in fall 2017, I left a job at TrueAccord, accepted an offer from Lyft (blog post on how I was searching for the job), and took a break.

At the same time, Kaggle hosted Carvana Image Masking Challenge. The task was binary segmentation. I used the pipeline from a previous competition and got to the top 10 on the public leaderboard. Another team from the top 10: Alexander Buslaev and Artem Sanakoyeu, proposed to join the efforts. We merged and finished first, by luck beating the legendary anonymous Bestfitting by 0.000001. (blog post with the solution).

During the challenge, we shared code. I checked out Alexander’s augmentation pipeline. It was elegant and fast, and I liked it.

I had my own and used it to win a few ML challenges, but I was tired of extending and debugging it.

I copy-pasted his augmentations, and for the next six months, we improved them independently. In winter 2018, I won another challenge and achieved the title of Kaggle Grandmaster, while Alexander won SpaceNet and Urban 3d at Topcoder. He used the prize money to buy an apartment :)

April 2018 The team ods.ai: Alexander Buslaev, Selim Sefebekov, and Victor Durnov won Data Science Bowl 2018, finishing 1 out of 3634 teams and were relaxing after this exhausting marathon.

After getting the Kaggle Grandmaster title, I stopped caring about machine learning competitions.

But there was an issue that I wanted to address. I did not have papers about Deep Learning during the job search, which made my life harder.

I wanted to fill my Google Scholar with many deep learning publications and no longer worry about that issue.

I found that CVPR 2018 had a Deepglobe workshop, which had a competition track with three challenges. For every challenge, participants needed to submit a prediction on the test set and a short paper describing the solution for every challenge for all three.

And I found it ten days before the deadline. Ten days for ML competition is too few, and we had not one but three. It did not scare us. Academic ML competitions rarely attract strong competitors, as academia focuses on novelty, not accuracy. Nearly every winning solution in ML competitions is a combination of good existing ideas. Publishing papers and winning competitions are two different sports.

I convinced Alexander Buslaev, Selim Sefebekov and Alexey Shvets to give it a try. In ten days, we trained three models, prepared three submissions, and wrote three papers. Our standing: (2nd, 2nd, 3rd (Paper 1, paper 2, paper 3)

In graduate school, I was frustrated that the Quantum Monte Carlo code that our and other research groups used was floating between scientists in tarballs via emails. Every scientist independently made improvements and fixed bugs, and all 100500 versions lived their own life.

The situation with augmentations started to look similar. I asked Alexander to publish his pipeline publicly to GitHub to avoid it. Even more - I wanted to release the code before the CVPR. I hypothesized that it could attract more people to look at the code, not just people - experts in computer Vision, as CVPR is the top world conference in this field. I proposed to add a link to the GitHub to all three posters that we needed to present.

I wanted to publish the code and make improvements later, and Alexander wanted first to rewrite the code and make it production quality before the publication. It looked like it would take us another couple of years.

At that moment, Alex Parinov joined the initiative. He understood the importance of the infrastructure for efficient problem-solving in ML. He actively participated in Kaggle competitions and contributed to open source. He said that he would help improve the code quality. And this was the last thing we needed to make the code public.

The name for the library was chosen this way: Alexander mentioned the library at work, and his colleague proposed the name Albumentaitons, which is the mix of Albu, the nickname of ALexander BUslaev, and the word “augmentations.” I do not remember the exact details, but it could happen that we picked this name as a temporary, but there is nothing more permanent than temporary, and we never even considered changing it.

June 2018, a few days before CVPR 2018, Alexander published the first commit.

At CVPR 2018, we picked the best sport for posters - near free coffee, but it did not help much. Not that many people paid attention to our posters or our library. But at least we tried :)

June 17, 2018, Alex added tests.

June 21, 2018, Alex added automatically generated documentation with Sphinx.

July 1, 2018, Alex added a benchmark that compared Albumentations to similar libraries. It allowed us to be sure that we were faster, and from that moment, if we had seen that other library made a faster transform, we looked in their code, learned their tricks, and absorbed them into our code.

In that phase, we added all augmentations that we could find. We did it because we could and because it was fun—no meetings, plannings, or writing tech specks. We moved fast, and it gave us drive. It felt good to see the progress. On the other hand, some decisions that we made, for example, names of the transforms or their default parameters, are far from optimal, but when we realized this, it was too late to change.

July 1, 2018. The first pull request from a person who was not on the core team

August 2018: Eugene Khvedchenia joined the team. By that time, he was actively participating in Kaggle competitions, and it was evident that you need to have good augmentations to get to the top. During the Data Science bowl 2018, he moved from Keras to Pytorch and developed his augmentation pipelines.

He moved to Albumentations after the first release. The library looked very similar to the code in the winning solution of the Data Science Bowl 2018, and he liked it. By August, he decided to add the support of bounding boxes to the library, which was valuable. We had images and segmentation masks at the time. His deep understanding of the code and high-quality pull requests were impressive, and thus he became a member of the core team.

During the summer of 2018 we:

  • Refactored the code
  • Improved test coverage
  • Added linters, formatters, improved Docstrings and Readme

We added several Jupyter notebooks with examples on how to use the library. We also added an example of how to move from Torchvision to Albumentations in a few lines of code.

November 28, 2018, Alexander added “multi-target” functionality. It allowed applying the same transform to a set of images, masks, and bounding boxes. We added a jupyter notebook, and later, in 2020, I wrote a separate blog post on the topic.

January 3, 2019, Eugene added keypoints. None of us needed them, but there was a request from users, and Eugene addressed it.

April 7, 2019, Eugene Added Lambda Transform. It allowed adding custom transforms without pain.

May 18, 2019, Alex added serialization. The choice of augmentations is a hyperparameter and should be defined in the config files. Serialization allowed to save/load to/from YAML, JSON, and python dictionaries.

In August 2019, Mikhail Druzhinin started contributing to Albumentations.

He wanted to join the Open Source project, but all projects he was interested in were too complex, or he could not figure out how to have an impactful contribution.

For one of his projects, he needed complex image augmentations. He found ImgAug, stumbled on problems, and created a pull request. He said that it was pretty painful. After this, he wanted to create custom transforms, but the limitations of the architecture of ImgAug made it hard to add. Hence he wrote his augmentation pipeline.

At the end of July 2019, he joined Severstal: Steel Defect Detection at Kaggle and realized that many people in Kernels use Albumentations. He looked at the library’s code and liked the elegance and the simplicity of the interface. Roughly at the same time, I tried to engage people from the ODS.AI community to contribute to the library. I promised we welcome everyone and no need to be shy.

Mikhail had experience optimizing code for weak hardware and instantly found a few places where Albumentations could be made faster. He made a set of high-quality pull requests with new transforms, bug fixes, and enhancements.

September 12, 2019, Mikhail joined the core team.

September 27, 2019, Alexander added Replay mode. Augmentation pipeline is composed of a set of transforms with different probabilities. Applying it twice to the same image will get different results, making debugging ultra hard. Replay/Compose allows full reproducibility when needed.

February 2020 Alex updated documentation and created a website for the project - albumentations.ai. The website looked so good that the author of the Insightface adapted the template for his website insightface.ai :)

July 2020, Mikhail, with the help of Alex, created Albumetations-experimental where experimental transforms are added.

September 2020 Alex created AutoAlbument - a tool that allows an automatic choice of augmentations for the task. By that time, I was not involved in ML projects and sadly did not have a chance to use it.

After that, we did not do anything fundamental.

For the most part:

  • Improved the documentation
  • Added new, often exotic transforms
  • Improved the code quality

April 2021, I met Alex and Mikhail in person. To my surprise, shy Mikhail is huge, a two-meter man, way taller than me :)

How to get the motivation to spend free time on the open-source?

In the beginning, it was just fun. You see how your project evolves and make the changes you want.

After that, we got dopamine boosts when we saw how the adoption grew. Some people even thanked us. Doing good things feels good, and it was a great motivation.

When the library became even more popular, everything changed. On the one hand, we implemented nearly everything we needed ourselves. On the other hand, we started getting bug reports, not about bugs but because users did not understand the documentation or read it.

Free spirit development started to resemble corporate support. We got to this state by spring 2020 and decided to relax and work on the library only when it is fun.

This approach has limitations. We look at pull requests not as often as we could, and bug reports could stay open for weeks. That is life. If you want the process to move faster - feel free to fork the code. The license permits it.

How did we promote the library?

There are many image augmentation libraries (blog post with an overview. We created another one. How to make it popular?

We wanted:

  1. The library should be fast. At least as fast as competitors.
  2. The core team should find it easy to use and extend.
  3. Others should find it easy to use and extend.
  4. People who are not the core team should know that the library exists.
  5. Users should recommend the library to each other without us asking for it.

We were addressing the first three points. But what with the last two?

We did not know and just experimented.

Academia

In August 2018, two months after the first release, I asked Alexander Kalinin to write a pre-print for arXiv about the library. The primary purpose was to make it easy for people in academia to cite us. For sure, we could add BibTex to the Readme, but citing GitHub repositories is not widely accepted yet, and many people refuse to do this.

In the beginning, it was self-citations. In 2018-2020 I actively published papers and always cited out pre-print. In the beginning, it was self-citations. In 2018-2020 I actively published papers and always cited out pre-print. But pretty quickly, unknown to us, researchers started to mention our pre-print.

In the summer of 2019, Alexander Kalinin lectured in summer school in Poland. Another lecturer there was Sebastian Raschka. Sebastian mentioned that he is an editor for a special issue to the MDPI and asked if we have anything relevant. Alexander showed the pr-print, Sebastian liked it, and invited him to write a full-scale publication.

We (mainly Alexander Kalinin), wrote the paper, and now it is the most cited paper of that special edition.

What else can we do here? People in academia introduce novel augmentation, and often someone adds them to the Albumentations.

We should write authors, thank them for their work, and mention that their transform is a part of the library.

Machine learning competitions

We worked on the library, competing in various challenges at Kaggle and other platforms. When we finished at the top and published solutions, we mentioned the library and how we used it.

Basically, as in other industries, recommendations of the influencers helped with the promotion, but in our case, we were these influencers.

We needed to decrease the pain of using Albumentations in Kaggle Kernels, and in April of 2019, the library was added to the Kaggle Docker.

If other winners used Albumentations in the solution but did not mention it, I pinged them and asked them to add it to the description of the solution.

But, being honest, I did not need to do it that often. People knew that we work on the library, they understood that when they mention it - it is the way to say: “Thank you for your work,” and they did it.

We created a page in the documentation with the list of competitions where winning teams used Albumentations + the description of the winning solution. Sadly, the page is outdated. I strongly feel that Albumentations is used in almost all winning solutions in Computer Vision challenges, but we do not want to parse competition websites manually.

I like this story. In fall 2019, I helped organize Lyft: 3D Object Detection for Autonomous vehicles challenge organized in conjunction with NeurIPS 2019. At the end of the conference, there was a dinner for all competition organizers.

A man was sitting next to me. I wanted to show off and mention that I am a Kaggle Grandmaster. He did not even understand that I was showing off. Typically only people from the Kaggle community see value in this title. Later in the conversation, he mentioned that the most profound insight from the challenge was that all top teams used Albumentations. His company did not wait till the end, retrained in-house models, and got a considerable improvement. When he learned that I am from the core team of Albumentations, he got excited and proposed to take a photo with me :)

Industry

Taking to computer vision people from various companies, I got an impression that Albumentations is used all over the place, including all FAANG.

But it is unclear how to make this information publicly available on the website.

When I worked at Lyft, I used Albumentations in several training pipelines. I asked the manager if I could add Lyft’s logo to the Readme and got a positive answer. I asked people from other companies, and more logos were added.

In general, the question is open. I do not know how to scale this up.

Open source

I can see 5700 repositories and 144 packets with Albumentations as a dependency.

In this case, it was clear what to do. I needed to take a popular repository with a computer vision training pipeline and rewrite the augmentation part with Albumentations. Repeat many times.

I was ready to do this, but refactoring someone else’s code did not thrill me. Luckily after the library became better known, popular repositories started adopting it.

Stars Repository 31430 https://github.com/commaai/openpilot 17495 https://github.com/open-mmlab/mmdetection 15302 https://github.com/spmallick/learnopencv 10793 https://github.com/deepinsight/insightface 3787 https://github.com/pytorch/ignite 3291 https://github.com/PaddlePaddle/PaddleSeg 2947 https://github.com/NVlabs/imaginaire 2793 https://github.com/catalyst-team/catalyst 2312 https://github.com/ml-tooling/ml-workspace 2155 https://github.com/saic-mdal/lama

These are top ten repositories, which have Albumentations as a dependency.

What can we do better?

  • We can look for questions at Stack Overflow about image augmentations and answer them with code using Albumentations.
Other experiments

Albumentations were not the only activity where I tried to learn how to do marketing. In 2017-2019 I gave talks in podcasts, conferences, meetups, and I was not allowed to talk about my projects at Lyft. Hence I spoke about Albumentations.

I live in the US and was promoting the library in English.

The rest of the team lives in Eastern Europe and promoted the library in Russian.

October 2019, the library became a part of the PyTorch Ecosystem. It was not hard to get there, but we thought it was very cool. There is even a conference about PyTorch Ecosystem. We could participate and promote the library there, but we were always busy, and it did not happen.

February 2020, we got into the Nvidia Inception Program. We did not plan to get there. Someone told us about this option, we applied and got approved. We got $100k AWS credits for a year and proudly added “Member of the Nvidia Inception Program” to the website. :)

Another small detail. If you look where people come to the repository, you will see link.zhihu.com and link.csdn.net in the top 10 resources for tech people in China.

I asked a student who speaks Chinese to translate my blog post about multi-target and post on the relevant platforms in China. I paid $50 for the work. The idea was - I already wrote the text, prepared pictures. China is huge. Let’s see how it will turn out.

Here is the source of countries who have come to the website for the past year. China is in the first place. There may be a relation between the translation and these statistics, but, most likely, without it, the statistics would be the same. Machine Learning is growing rapidly in China.

Every release of the Library, we did posts to:

  • Twitter
  • LinkedIn
  • Reddit

Not all, but most of the announcements spread well. I have 6k followers on Twitter; Alexander Kalinin has 4k. It helped.

The product is successful when people who are not affiliated with it write blog posts about it. If we use this metric, Albumentations is a success as people regularly create blog posts mentioning Albumentations.

My favorite is one by Ilia Larchenko. He used streamlit to create a web app that allows choosing augmentations and their parameters for a task at hand.

What will I do differently if I will do it again?

It is hard to tell. Right now, I am trying to learn to build a startup. Hence, if I work on open-source, it should have a way to monetize—something like an open-core model, where you have free features for developers and non-free features for the enterprise.

Can I convert Albumentations to Open-core? Most likely, but I cannot figure out how exactly. Maybe someone who reads this post can message me and give a hint?

It is hard to tell, but I would say - most likely. It was a lot of fun. Every time someone said: “Thanks” to us or just mentioned the library, we got dopamine boosts.

As a bonus, I improved my programming skill, and, line: “Co-creator of Albumentations” may have some value.

I use a lot of open-source projects, but I do not know the names of the people who develop and maintain them. Same here. Most people who use Albumentations do not know who the core team is.

But there was an exception :)

This summer, I visited Kyiv. One night one female Data Scientist and I went out and walked in the night city. She used Albumentations at work and knew that I was one of the core developers. It may happen that she would not go out with me if not the library :)

What about metrics? Conclusion

In the end, I would like to thank the core team and all the contributors. Open source is a bizarre beast. It takes a lot of time to befriend it :)

If you get to this part, I would like to ask for a favor:

  • If you use the library at work, could you please ask your manager to add the company’s logo to our website? If yes - you just need to follow this link and fill the form.
  • If you use Albumentations in your scientific projects, it would be great to cite our paper.
  • Pull requests are greatly appreciated if you found a bug or have ideas on improving the library.
  • And of course - if you did not give a star to the repository, it is time to do this :)
https://ternaus.blog/story/2021/12/15/Albumentations
List of good non-fiction books that I have read in 2018-2021.
booksbooks
List of non-fiction books that helped me why working at Lyft (2018-2021).
Show full content
Table of Contents

Practice without theory is blind. Hence we need to supplement our practical endeavors with theoretical works. And it often comes in the form of books.

Here is a list of books I have read while working at Lyft. I can’t entirely agree with all the ideas presented in them. However, they proved to be ultra-valuable in advancing my career and shaped my current understanding of the tech industry.

How to pick which one to read?

It is essential to say that the book’s usefulness is strongly related to the reader’s context.

Typically, I practice “pain-driven prioritization,” i.e., when I feel confused and do not clearly understand what is happening and what exactly I am missing, I pick books that cover the relevant topic.

Example 1: Reorganization happens, the director in your organization gets fired, the team gets merged with another team, and many projects, even considered “strategic,” are closed. It would make sense to read something about “Career, Politics” and a story about a similar company and the challenges it faced. Say, when you work at Lyft, a book about Lyft would be great, but the book about Uber will also work.

Example 2: End of the year, the company starts planning for the next quarter. Everyone talks about OKRs and how important they are, but when the planning is done, and the actual work starts, people at all levels ignore half of the OKRs and do something else. => You read about OKRs, prioritization, planning, and Product managers.

You can read the following books in any order you like, but you may consider my “pain-driven” approach as well.

If you are a product manager, I would recommend checking the list of the books by Adam Eldarov - chief product officer at YouDo. His list is excellent, although I have read only 10 books from there so far :)

It is not the complete list; it is the list that I have read so far and found insightful. There is probably another one for every book in it that talks about the same topic but much better. If you know such a better book - let me know :)

Product
  1. The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses
  2. INSPIRED: How to Create Tech Products Customers Love
  3. The Mom Test: How to talk to customers & learn if your business is a good idea when everyone is lying to you
  4. Radical Focus: Achieving Your Most Important Goals with Objectives and Key Results
Management
  1. High Output Management
  2. No Rules Rules: Netflix and the Culture of Reinvention
  3. The Goal: A Process of Ongoing Improvement
  4. Radical Candor: Fully Revised & Updated Edition: Be a Kick-Ass Boss Without Losing Your Humanity
  5. Managing Humans: Biting and Humorous Tales of a Software Engineering Manager
Communication, teamwork, and collaboration
  1. Nonviolent Communication: Create Your Life, Your Relationships, and Your World in Harmony with Your Values
  2. Crucial Conversations Tools for Talking When Stakes Are High, Second Edition
  3. The Culture Map: Breaking Through the Invisible Boundaries of Global Business
  4. The Asshole Survival Guide: How to Deal with People Who Treat You Like Dirt
  5. Everything is Negotiable
  6. Influence: Science and Practice
Marketing
  1. Пиши, сокращай
  2. Contagious: Why Things Catch On
  3. The Catalyst: How to Change Anyone’s Mind
  4. Purple Cow: Transform Your Business by Being Remarkable
  5. The Art of Seduction
Strategy
  1. Good Strategy Bad Strategy: The Difference and Why It Matters
  2. Blue Ocean Strategy
Career and politics
  1. Rise: 3 Practical Steps for Advancing Your Career, Standing Out as a Leader, and Liking Your Life
  2. Лестница в небо. Диалоги о власти, карьере и мировой элите
  3. The 48 Laws of Power
Startups
  1. Zero to One: Notes on Startups, or How to Build the Future
  2. The Founder’s Dilemmas: Anticipating and Avoiding the Pitfalls That Can Sink a Startup
  3. Blitzscaling: The Lightning-Fast Path to Building Massively Valuable Companies
  4. Secrets of Sand Hill Road: Venture Capital and How to Get It
  5. Venture Deals: Be Smarter Than Your Lawyer and Venture Capitalist
Stories about people and companies
  1. Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future
  2. Super Pumped: The Battle for Uber
  3. Steve Jobs
  4. The Airbnb Story: How Three Ordinary Guys Disrupted an Industry, Made Billions . . . and Created Plenty of Controversy
  5. The Idea Factory: Bell Labs and the Great Age of American Innovation
  6. Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration
  7. Autonomy: The Quest to Build the Driverless Car—And How It Will Reshape Our World
  8. The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers
  9. Principles: Life and Work
  10. AI Superpowers: China, Silicon Valley, and the New World Order
  11. Bad Blood: Secrets and Lies in a Silicon Valley Startup
Well-being and self-improvement
  1. The Upside of Stress: Why Stress Is Good for You, and How to Get Good at It
  2. The Mind Illuminated: A Complete Meditation Guide Integrating Buddhist Wisdom and Brain Science for Greater Mindfulness
  3. The Happiness Advantage: How a Positive Brain Fuels Success in Work and Life
  4. Willpower: Rediscovering the Greatest Human Strength
  5. Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones
  6. Tiny Habits: The Small Changes That Change Everything
  7. Why We Sleep: Unlocking the Power of Sleep and Dreams
  8. Джедайские техники: Как воспитать свою обезьяну, опустошить инбокс и сберечь мыслетопливо
  9. So Good They Can’t Ignore You: Why Skills Trump Passion in the Quest for Work You Love
  10. The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life
  11. Deep Work: Rules for Focused Success in a Distracted World
  12. Mastery
Other
  1. Thinking, Fast and Slow
  2. The Black Swan: The Impact of the Highly Improbable
  3. The Righteous Mind: Why Good People Are Divided by Politics and Religion
  4. Extreme Ownership: How U.S. Navy SEALs Lead and Win5.
  5. The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
https://ternaus.blog/books/2021/08/19/List-of-Books
YouDo Product Skills Track
booksbooks
A list of books to boost your Product Management skills.
Show full content
Table of Contents

I came across a great list of resources for product managers by Adam Eldarov, the Chief Product Officer at YouDo. YouDo is a Russian company, and the list is in Russian. I believe that it is great (I have read 10 books from it so far and loved them.) and that it is worth showing it to the English-speaking audience.

The following article is the translation of Adam’s text to English. I removed links to the resources that require the knowledge of Russian. For example, the sections of “Customer Journey Map” and “UX Writing” were removed.

If you speak Russian, I recommend using the original article.

In the modern tech world, engineers work closely with product people. Hence, I recommend these books to all engineers. It will boost skills and help to advance the career.


In this article, you will find resources to boost your product skills. After finishing the list, you will know the theory at the Lead Product Manager level. It is only a theory. It complements but does not substitute practical skills.

Each section corresponds to the specific skill that is required to build successful products.

There is no particular order in which you need to choose the section, but after choosing one, I recommend studying the material within this section in the given order.

Common PM User Research Common UX Research Jobs to be Done Numerical Analysis Decision-making and the analysis of metrics SQL Strategy How business and economy work Business models and promoting the product UI & UX Lean Agile System thinking Design thinking Communication Negotiations People management Politics Brain science
https://ternaus.blog/books/2021/03/17/PM-track
I want to organize ML competition. Where is the playbook?
kagglekaggletutorial
Organizing ML competions is not hard. Still, companies do not do it well.
Show full content
Table of Contents

There is no such a playbook, but there is a standard set of mistakes that organizers make.

You may think about ML competition as an exchange.

You provide:

  • Human Resources to organize the competition.
  • Prize money
  • Intellectual property in the form of the data.

and expect to benefit in technical, hiring, and marketing domains.

  • Technical: improve production ML models.
  • Hiring: you will get access to the pool of skilled candidates.
  • Brand awareness: more people will know who you are and what problems your business solves.

To be clear. You will invest money and human resources. Insights from the competition could be used by your competitors. Still, there is no guarantee that your company will benefit in any way.

Competitions provide opportunities, but it is up to you to use them.

And, of course, the better you organize the challenge, the more options you will get.

A better organization does not mean investing more money or human resources. It means avoiding mistakes.

It is trivial to organize “some” competition. But I hope that “some” is not your level and that you are shooting for a great one.

In the article, I will describe topics that organizers need to keep in mind. The text consists of small subtopics; you can read them independently.

It is a “longread” and unlikely that many people will read till the end. But I hope that future organizers will consult with the text when they work on their own challenge.

Compass

From the first day, you will encounter questions like:

  • Where to host? Do it yourself or use a third party?
  • What is the format of the data?
  • How to promote?
  • Should we allow participants under 18?
  • etc

I recommend the following compass to guide your decisions:

Every time you make a choice try to maximize the number of participants.

A participant is a person who made a submission at appeared on the competition Leaderboard.

Suppose a person downloaded the data, made an extensive exploratory analysis, read 100 relevant scientific papers, hosted 10 webinars on approaching the challenge but did not appear at the leaderboard. If you want, you can call him an “influencer,” but definitely not a participant.

The number of participants is a straightforward metric that is easy to measure. When you optimize it, all other metrics will go to the “green zone” automatically.

Quality of the competition vs. the number of participants:

  • Bad: 0-10
  • Ok: 10-100
  • Good: 100-1000
  • Great: 1000+

For sure, not all participants are equal. One person will spend a few hours and forget about the challenge. The second will work on the problem 24/7, sharing valuable insights with others at the forum.

It is not that important. The compass is not about the specific numbers but about the order of magnitude. When the engagement increases 10x, absolute numbers for all categories of participants go up.

Human Resources

At a minimum, you need two people:

  • Technical: engineer or scientist
  • Non-technical: product manager or marketing person

They will work on the problem part-time, but there should be clear owners for various parts of the project.

Besides, the project needs support at the director level. You will have requests from other teams, and you need a decision-maker that will help you.

Find an advisor

Competitive machine learning is different from machine learning in industry or academia. Some things that you assume are crucial are not important at all. Others, looking non-relevant, would be worth thinking about.

Look within a company. There may be a person in another department who organized or participated in the ML challenges before. Involving such a person as an advisor could save you a lot of time and energy.

Example: In 2019, Lyft Level5 planned to organize an ML competition on the perception dataset. I got involved in the project by accident. Stopped to grab a coffee in the kitchen and got into a morning chat with a person driving the effort. I shared what I learned during my career as a professional competitor. He asked me to join the team of organizers. Based on my feedback, we adjusted a few things. I hope this made the challenge better.

Understand the participants. Who are they, and why do they compete?

Your motivation as the organizer is different from the motivation that your participants have. I recommend understanding who the participants are to make your incentives aligned.

You want the challenge to have a positive impact on your business. Participants do not care about your company, its mission, and how the competition will affect your business.

Working on ML competitions is a second, unpaid full-time job. You need very good reasons to invest your free time in them.

I like to think about machine learning competitions as about weight lifting gym but for ML muscles.

People wake up early and go to the gym to develop the body and look better, but the main reason is that they enjoy the process. (Endorphins get released.)

The same story here: there are new skills and knowledge, but people mainly participate because it is a lot of fun.

People imagine the typical participant as:

  • Wakes up in the morning, full of energy.
  • Actively looks on the internet for an exciting challenge.
  • Finds one where he has some expertise.
  • Ignores her day job, university, personal life.
  • Without any guideline or tutorial, figures out how to get access to the data. Through pain, learns how to work with it.
  • Writes the code and trains the model.
  • Uses her intuition to decipher a poorly written description of the submission format.
  • Creates the submission and uploads it.

The reality is different. I would not even recommend thinking about participants as someone eager to work on your problem through the pain.

In reality, the two hours of the challenge look like this:

  • The person comes home in the evening, tired from the job or classes.
  • She already knows about the competition because she read about it on some blog or got an email about the competition announcement.
  • She chooses to work on this problem because the challenge looks interesting. She does not know how to solve it, but it is her chance to master the task.
  • She downloads the data and tries to write an end2end pipeline.
  • If she does not understand how to approach the task, she gives up and never comes back to the problem.

Think about participants as your clients. You want to maximize their number and engagement, and you do not want to lose them to the competitors.

Many ML challenges happen at every moment. If you do not treat yours seriously, others will be more interesting, have larger prizes, and better organization. Participants will go there.

Recommendation: Participants should feel the least amount of friction possible.

Do not host yourself. Use third-party.

Most of the ML challenges follow the path:

  • Find $1000 for prize money.
  • Find an intern or graduate student. Tell her to prepare the dataset, introduce the metric, and define competition rules.
  • Announce the problem in the post on the company’s blog.
  • Ask participants to send predictions by email. Intern will evaluate them manually.

When the competition is over, split $1000 among winners, call it a “Great success.”

Most of the ML challenges that happen in conjunction with academic conferences follow precisely this path. This was also the original plan for Lyft’s 2019 challenge.

Such competitions have 0-10 participants. This is bad.

I participated in a few such challenges at CVPR and MICCAI. My team got to the top in all problems that we tried. Weak participants and the whole process is not very enjoyable.

Ideally, you want:

  • Thousands of people working on your problem
  • Insights are generated through the whole challenge and not only at the end.
  • Participants have access to hardware even if they do not have powerful desktops at home.
  • Scores are computed automatically, and the leaderboard is calculated automatically.

Competitive machine learning is a maturing field. Many companies provide platforms for challenges: Kaggle, Topcoder, Datasoluls, Drivendata, Aicrowd, Signate, etc.

I am not affiliated with Kaggle. But I believe that Kaggle is the largest and the most established platform. Most of the challenges that I joined as a participant, and both that I organized, were at Kaggle.

If you have enough budget and they are interested in your problem - I recommend working with them. Kaggle is a popular platform that hosted hundreds of ML competitions. It also provides CPU, GPU, TPU hardware to participants.

The representatives of the platform will help avoid common mistakes about your dataset.

Example: In 2018, I participated in the Giana challenge (part of the MICCAI conference). At the end of the challenge, I sent my prediction in email and went rock climbing. I thought that my work on the challenge is over, but I got an email that there is something with my prediction file, and the organizer cannot calculate the score. As a result, I was editing my prediction file from the cell phone in the middle of the Sierra mountains. We fixed the issue, but I did not like the experience. If there is an issue with my submission file, I need to know it instantly. In 2019 I decided not to participate. If organizers worked with a platform, they could avoid this issue.

Recommendation: Do not try to host the competition yourself. Use one of the platforms.

Prize money

What is a good prize money?

There is a correlation between the number of participants and the prize amount. But it is non-linear. For sure, $1000 is questionable, but the difference between $100,000 and $1,000,000 is relatively small. I would not say that it worth an extra $900k.

If your dataset and your problem are bad, a large prize will not help. If it is interesting, you will get good engagement even without huge reward.

Recommendation: do not overthink this, assign $25k for prizes, and focus on other more important things.

Kaggle tiers and points

One of the features that distinguish Kaggle from other platforms is its gamification system. A small number of people spend their free time on challenges to each money. It is either novice who did not have a chance to measure their skillset or elite competitors who know what they are doing.

Kaggle has a system of tiers:

You get the title based on the medals that you obtain during the challenge: gold, silver, bronze. Getting to the next stage is exponentially hard. There are only 200 Kaggle Competition Grandmasters out of 3 million who joined and 150k that actively participate.

These titles matter a little outside of the competitive community but matter a lot within. People spend days and nights to get rare badges in World Of Warcraft and other computer games. The same story here.

Besides, kaggle assigns points to the competitions. Points from all challenges you were part of are summed up. This tells your place in the overall rating. These points expire with time. Being within the top 100 out of 3,000,000 feels like an achievement.

(At the peak I was 19th).

By default, Kaggle enables tiers and points for all competitions except those where the organizer cannot guarantee the data’s high quality. One of the criteria is that the test set cannot be accessed online.

Recommendation: If you work with kaggle, discuss with them the situation with tiers and points. Points will lead to better engagement. If kaggle disapproves your dataset for the competition with points, this is a big red flag about your dataset.

Competition format

There are three main formats:

Offline / offline

Participants train and perform inference on their own hardware and send the file with predictions to the platform.

Pros:

  • Participants will squeeze the most out of the dataset and the problem.

Cons:

  • Top solutions will be rather complex
  • Some teams will perform better because they have access to more powerful hardware.
Offline / online

Participants train offline on their own hardware and perform inference on the platform’s hardware—examples: Kaggle kernels or Docker container submissions at Topcoder.

Pros:

  • You can restrict hardware and inference time to bridge the gap between the competition and production setting.
  • Participants do not have access to the test set. Better check for the generalization of the model.

Cons:

  • Harder to organize. The platform needs to support this functionality. Example: Kaggle does not support docker submissions.
Online / Online

Participants train and perform inference on the platform’s hardware.

Pros:

  • You can restrict hardware, train, and inference time to bridge the competition and production setting gap.
  • All participants are in the same boat for the train and for the inference. Getting to the top is about skill and creativity and not about the number of GPUs at your disposal.
  • Participants do not have access to the test set. Better check for the generalization of the model.
  • Participants that do not have their own hardware can join.
  • Full reproducibility of the pipeline.

Cons:

  • Not all potential of the problem and the dataset will be explored.

It looks crucial to choose the best format for the competition, but it is not.

Insights from the winning solutions have at most 30% of the value. The largest (70%+) impact comes from the tricks and ideas that participants generate during the challenge and share at the forum. The format of the challenge is secondary.

My favorite is the offline/online format. It allows to squeeze the most out of the problem, but the inference is closer to the production.

Recommendation: Pick a format that you like, and if it does not work for some reason, relax the constraint. It is not a big deal. You’d better focus on absorbing the most from the tricks and insights that the community will generate from the first day of the challenge.

Competition Metrics

The leaderboard is computed based on the metric score. Every problem could be evaluated using various metric functions, for example, for classification tasks: log_loss, Accuracy, Roc_auc score, F1, etc.

I would recommend choosing one that is closest to the production setting.

It may happen that platforms do not support your metric. They can add any reasonable metric, but you need to discuss it way ahead.

Dataset

Dataset is the most critical part of the whole story. If you mess it up, nothing will save you.

To host the competition, you will release some of the internal data. The first question is what license to use? I would recommend picking one that allows non-commercial use.

The size of the dataset correlates with the number of participants. There are two edge cases:

The dataset is small. People can train models on laptops—a lot of participants. Dataset of 10-500Gb. Participants will need access to the cloud or desktop with GPUs. The engagement will be smaller, but it is still quite possible to have more than a thousand people.

Tabular data problems fall in the first category, computer vision, natural language processing to the second.

The difference between 10 and 100Gb size datasets is relatively small. You approach both with the same data processing and training pipeline.

Recommendation: Do not overthink this. The size of the dataset is secondary. If you can release more - release what you can.

Data format

Participants will take the data in the format you provide and convert it to the form that suits them best. But taking extra steps can feel frustrating and push people away.

Example: companies that use Tensorflow like to release the data in the tfrecords format. Do not do this. This is interpreted as laziness and a lack of respect for the community. If the person needs to look at the image in the dataset, she should be able to click on the file, and that is it. If she needs to write code to visualize the photo, it is frustrating.

Recommendation: Go with jpg for images, png or RLE for masks, CSV, and json for tabular and text data. Easier to access the data - better. Do not use uncommon formats to make your dataset 10% smaller.

Dataset SDK

Suppose your dataset is involved and unfamiliar, as was in both Lyft’s competitions. It is crucial to provide an SDK and tutorial that help to work with the data. An ideal situation is if you can open-source the library that you use internally.

We did this in 2020 with L5kit.

As a result, improvements that participants made from the first day positively impacted similar internal tasks.

While open sourcing the L5kit, we improved it’s structure and decoupled it from the internal libraries. Participants found a bug in the SDK, identified I/O bottlenecks, and proposed ways to address them. I/O improved 6 times.

For the SDK license, I recommend having something permissible like Apache-2, MIT.

You will have discussions with the legal team like: “The license for the dataset is A, for the SDK is B, what license do we pick for the winning solutions?”.

If you go with a permissible license for SDK, it will make such discussions faster and easier. At the end of the day, your SDK is just a set of helper and visualization functions. It is not an ultra crazy intellectual property that can give your competitors an edge.

People should be able to install your SDK with pip install in one line without any pain. A five-page installation manual is not ok.

Kaggle provides a way to work with the data and train models on their hardware (CPU, GPU, TPU) in Kaggle Kernels. If you work with Kaggle, you need to check if your SDK works in Kaggle Kernels without any problems.

Recommendation: Easy access and visualization of the dataset are ultra-critical. It is more important than the prize value or the size of the dataset. Do not underestimate this!

Train / test split

As an organizer, you want your model to generalize well. I.e., work well on the data that it has never seen before.

To check the generalization ability of the model, people split into two parts:

  • Train
  • Test

People train on a train set, perform inference, and evaluate model performance on the test set.

Participants do not have access to the test set labels, and in the case of the ([offline, online] / online) format, even to the test set.

Issues begin when the train set and test share common data or at least information. The problem is called Data Leak.

If you have a data leak:

  • How well will models generalize to production is unclear.
  • PR risks. Data Leak is considered such a basic mistake that competition with it gets bad PR. The reputation of the company’s Data Science team will be affected. It will solidly get attention on social networks.

The thing is that this mistake is not basic at all. Data leaks happen pretty often in industry, competitions, and in academia.

It can take years to find data leaks in the academic dataset (no one is looking at exploiting the data). In ML challenges, it will take hours when thousands of eyes are actively searching for ways to get an edge over other competitors.

If you have a data leak in your data, it is not the end of the day. You can provide a new test set and reevaluate the Leaderboard. Still, it is better to avoid the situation from the beginning.

If you organize the competition at Kaggle, their specialist will help to avoid obvious pitfalls, but do not overestimate their help. They have a playbook about data leaks and how to avoid them, but your task is not their domain, and they can miss non-obvious problems.

Random train/test split is rarely the way to go. Split by patient/geography or time will serve you better.

Recommendation: Pay close attention to the way you perform train/test splits. It is better to avoid Data Leak in the first place rather than deal with the consequences.

Sample Model

After the participant appears at the Leaderboard, she is hooked. The dopamine gets in her blood, the focus shifts from critical tasks at her day job to the ML challenge. From now on, she will work on the challenge, replacing old modules in her pipeline with new ones that she will learn in literature and her own creative mind.

As an organizer, you need to ensure that this actually happens. This is consistent with our compass: “maximize the number of people at the leaderboard.”

It is easy to do. You need to provide a code, called sample submission that:

  • Processes the data
  • Trains the model
  • Evaluates the model using cross-validation
  • Performs an inference
  • Generates the submission in the correct format

There is no need for this submission to be ultra-accurate. The most important part is to ensure that it runs and that people can run it even if they do not understand what is under the hood. They will learn it, but later. It is much easier to understand the flow when you have an example in front of you.

Your sample submission will also affect where participants will invest their resources in the first month of the competition. Hence, if you are curious about how an algorithm from the recent conference that claims to be state of the art perform on your dataset - just write a sample submission based on this algorithm.

Recommendation: It is better to have a sample submission on the first day of the challenge. Sample models for straightforward datasets will not provide a lot of value. Participants will create such models and share them with the community. If you explore the new type of task and involved dataset, a sample model can drastically decrease the friction and boost the engagement.

External Data and pre-trained models

More data is better. Some techniques do not work on small datasets. You want to know how the amount of train data will contribute to the model’s accuracy.

As an organizer, try to provide as much training data as possible. If participants do not have powerful hardware, they will use the subset of what you provide.

There is another way to enrich the problem. Datasets from similar tasks. This is popular in medical imaging contests where datasets are limited. Pre-trained models. People train model on dataset A, publicly share weights of this trained model, participants use these pre-trained models in their solutions.

In many competitions, organizers allow external data. But things can go wrong, and this is related to the legal.

Classic story - Researcher takes the dataset that DOES NOT allow commercial use, trains a model, and shares it publicly using a license that ALLOWS commercial use.

The question is - can people use models that he trained or not for commercial purposes? Model is not data; it is a derivative of the data. It contains the information from the original dataset in aggregated form. There are examples of when the original data was extracted from such trained models.

Deep Learning frameworks by Facebook (Pytorch), Google (TensorFlow), Intel (OpenVino) provide a set of pre-trained models. Many of these models are trained on datasets that DO NOT allow commercial use. The most prominent dataset is ImageNet.

I created issues at Github regarding this question and asked companies to clarify the models’ license in their libraries.

The best answer: “We do not know. It is up to your legal department to calculate the risks of using our pre-trained models.”

You can interpret this as companies that use pre-trained models could be subject to a lawsuit. And this is 99.9% of the companies that use computer vision.

Concerning the competition, this means that the organizer wants models or some parts to be used in production. Hence you would like to limit data and pre-trained models to the set that allows commercial use.

Typically, at Kaggle, there is a thread at the forum where participants post links to the datasets and pre-trained models they plan to use. They ask for approval from the organizers. But!

  • Kaggle representatives cannot answer this question because it is up to the organizer.
  • The researcher/engineer that organizes the competition cannot answer it because it is a legal question.
  • She pings the legal department. They are busy, and the situation about competitions / pre-trained models is unfamiliar to them.
  • They set up a meeting in a few weeks to discuss the situation.
  • Legal gets the idea during the meeting, but they need to have a follow-up meeting in a week.
  • In the follow-up meeting, the legal team does not say: “Yes, they can use,” “No, they cannot use.” They instead talk about potential risks in either of the ways.
  • Now, the Researcher needs to take responsibility and publicly verify: “Yes, it is allowed,” “No, it is not allowed.”

In practice, participants will post in the external data thread, the host never replies on what is allowed and what is not. Participants assume that it is ok if they posted in the thread, but it could not be the case.

Similar situation with datasets.

Example: In the winter and spring of 2020, Facebook hosted the Deepfake Detection Challenge, the prize was $1,000,000.

External data and pre-trained models were allowed in the competition. Participants posted links in the external data thread.

Hosts did not want to take responsibility for the decision. They gave inconclusive vague responses.

As a result, the team who finished first and was already splitting $500,000 in their mind was disqualified. External data that they used and posted at the forum was not allowed. Still, they learned this only after the competition was over. The community was not happy with how the Facebook team handled the situation.

How to deal with this?

You will not use models that competitors trained in production ever. At most, you will take their code and train on your in-house datasets. Hence it is not really crucial if external datasets or pre-trained models could be used for commercial purposes. Allow those that you like, and do not overthink this.

The only exception is if the team uses a codebase that does not have a permissive license.

Example: YOLO 5 is a powerful modern Object Detector. It’s Pytorch implementation has a GPL3 License, which does not allow commercial use. In one of the Kaggle competitions top team used it as a part of their solution. They were disqualified.

The deepfake story happened due to miscommunication and the lack of transparency.

This was well-received:

Pre-trained models and external data is just a nice addition. If you will not allow them but will make this point clear, no one will get upset.

Recommendation:

  • Do not overthink the value of “commercial use only” pre-trained models and external datasets. It is not really important. Allow as much as you can, as long as all participants have free access to it.
  • Discuss it with your Legal department that this is ok to give the host (engineer/researcher) a right to say “Yes” and “No” about external data and private datasets without consulting with Legal.
  • State clearly in the “External data thread”: “If the host did not say explicitly yes, this is allowed. It is not.” Here is an example of the thread that I created for Lyft’s 2020 competition.
Support during the competition

After the competition launch, you need to provide support. It takes at most 30 minutes a day. Typically much smaller.

You will need to address things like:

Technical:

  • What does X in the dataset description mean?
  • I think there is a bug in the metric; I think this happens because of YYY.
  • There is a bug in the dataset SDK. Here is proof.

Non-technical:

  • Can underage people participate?
  • Can we use the dataset outside of the competition?

Some competitions may have drama:

Participants will find a bug in the metric or Data Leak. You will need to address the issue and restart the competition. Someone copy-pastes the code from someone without giving them credit, and somehow it ended with a heated emotional discussion. Someone unskilled pays a strong team to join them and get Kaggle medals/points without working on the problem.

In 2019 I answered all types of questions.

In 2020 we had more resources, and Luca Bergamini and I split the responsibilities.

Luca works on the motion Prediction tasks at Lyft Level5. He understands the dataset, metric, SDK, and all technical nuances better than me -> he was covering technical questions.

I took care of the non-technical and was prepared to partner with the Kaggle team and resolve all the possible drama.

Recommendation: The person that works on the problem internally must cover technical questions.

  • His answers will be precise and informative, => a better user experience.
  • He will learn from the participants. Only 30% of the value comes from winning solutions. 70% is coming from the forum. It is a separate world with its own slang. To absorb all of the information, you need to know the problem well and spend enough time communicating with participants.
How useful winners models to the production?

There are two opposite opinions:

  • Winners dream of writing their solutions in the production quality code, with good documentation and using only those libraries that will allow the organizer an easy integration.
  • Winning solutions are absolutely useless. It is always an ensemble of the models that are very far from the production setting.

Let’s look at what is right and wrong about them.

Participants care only about one thing - the score on the Leaderboard. They will overfit the dataset and the metric, fighting for the improvement in the 5th digit after the decimal point. The final standing of the participant is proportional to the number of ideas she tries. The majority of the code she writes will be scratched. The most important thing - is how fast do you iterate. The quality of the code is secondary. Winning solutions will be far from production quality.

Participants will use tools that are the best for strong models and fast iterations. How easy it will be for you to integrate the solution will not cross their minds.

Example 1: Until 2020 Lyft. Level5 used Tensorflow in the car. Participants of Lyft’s 3D Detection challenge used Pytorch.

Example 2: There were challenges where organizers encouraged participants to use MatLab. Matlab has a niche where it is the best language and set of libraries, but it is not machine learning. All the encouragement was ignored.

Every idea that the participant tries is a separate submission. They will pick the best, or you can combine them in an ensemble. Ensembles have slightly better accuracy and have better generalization ability. Everyone builds them. This ensemble is slow and heavy, far from the production setting.

But! Ensembles add just a little on the top of single models. The winning solution is not a combination of similar weak models. It is a combination of strong, diverse models. Typically the best single model from the ensemble is also a top solution.

Example 1: Carvana Challenge. Our ultra-heavy ensemble finished 1st. But single models were in the top 10.

Example 2: Lyft Motion Prediction Challenge. First place used a complex ensemble, but the best single model would also finish first.

Code from winning solutions will be hard to integrate into the production. You will need to understand what is happening in these solutions, take useful ideas, and rewrite them, following your internal coding guidelines.

Recommendation: You will not deploy winning solutions to your production as is. I recommend having a person that works on the competition problem internally as a part of the organizing team.

Marketing

Should you invest in the promotion of your competition? Of course. If you do not do this, the number of participants will be smaller. That is against our compass.

People need to know:

  • that your challenge is happening.
  • That problem is interesting
  • That the entry point is low even if the problem is novel and ultra-complex.

Things that we did in 2020:

  • We hosted at Kaggle, and Kaggle sends an email to 3,000,000+ people at the launch of every challenge. These people are related to ML. Most of them tried at least the tutorial competition.
  • We released the train set months before the challenge, published a blog post, and encouraged people to use it in their research. The goal is to engage the academic audience.
  • We wrote a pre-print about the dataset and published it to ArXiv. Later we submitted it to the CoRL conference. Our head of AV Research presented the paper there.
  • We organized two webinars, one as a part of the ECCV, where we talked about the dataset and the challenge. I was surprised, but we got hundreds of participants, and they asked the right questions. Later we published Q/A based on it.
  • Social Networks: LinkedIn, Twitter, Reddit, HackerNews
  • We applied to host the competition as a part of the NeurIPS competition track (we were rejected). Could give more attention to the academic audience.

I would say that the most impactful point is [1]. Everything else had some value, but my guess that it was not very large.

Do not be shy about marketing. You invest time, money and resources. You help with the democratisation of AI. Many people will start their journey to the world of machine learning because they will feel excited about your ML challenge. It is ok to be proud of this, especally if problem is interesting and organization is at the high level.

Hiring

During the competition, you can see how well participants perform on your problem and consider hiring them.

In theory - it is an excellent opportunity to evaluate candidates, but it does not work well in practice.

Geography: Many candidates live in the former Soviet Union, India, and China. You could be interested; they could be interested. But the situation with visas makes it hard to happen.

Expectation mismatch: Elite competitors do not have problems with work. Most likely, positions that you have will not interest them.

Example: My friend Selim Seferbekov participates in competitions to earn money. Last year he got $500,000 in the Deepfake challenge. After taking 6 months break from it, he got prizes in 7 other challenges—standard ML positions in Google, Facebook, Lyft to him look ultra boring. At the same time, he would be happy to explore options at Deepmind or another company that “works at the edge” and has an excellent financial component.

What about non-elite competitors? Are they good? They are great. But recruiters rarely look below the first three places.

To get a job, candidates need to pass resume screening and a set of interviews. Candidates with a competitive ML experience are so rare that recruiters are not used to the keywords they add to the resumes. For example, “Kaggle Grandmaster” will not mean anything to them.

There is an exception. In the internal doc for HRs at Facebook, “Kaggle Master” is treated as keywords that deserve attention.

Even if they pass, most of them will perform weaker than other candidates during interviews. Your team will not be able to evaluate her strengths, and she will not perform well on things you ask from an average candidate.

Example: A number of my interviews followed the pattern: I was told that my ML knowledge is impressive, and no need to waste time digging deeper into this. Instead, I was asked questions about algorithms and data structures. I remember, in one company, it was LRUCache. I had no slightest clue what it is and how to implement it. I did not pass.

Some companies figured out how to hire people from competitions and make them bring value. For example, H2O.ai, Nvidia, Deepmind. But it is an exception. You may contact hiring teams in these companies and ask about their ways.

There are secondary effects that help with hiring. With the competition, you broadcast what problems do you solve. This makes people who like these tasks excited. During 2019, 2020 Lyft’s challenges, participants messaged me and asked about Lyft, Level5, what problems we solve, and how to become a part of the team. We do not have a particular hiring pipeline for this case; I passed their resume to HRs and did not follow after.

Recommendation: Look for people in the top 50 and not the top 3. Think about candidates as Research Engineers and not Research Scientists or Software Engineers.

Wrapping up

After the competition is over, I would recommend:

  • encourage people to share their approaches at the competition forum.
  • writing a blog post that summarizes insights from the challenge.
  • organize a webinar where winners can share their techniques and answer questions from the community.
https://ternaus.blog/kaggle/2021/01/18/Competition_playbook
Airbnb. Thanksgiving. Burglary.
incidentcrimeincident
Bad things happen. And sometimes it is not our fault.
Show full content

Table of Contents

I used Airbnb for years in Russia, Germany, and the United States. All the time, the experience was great. At some point, I read a book about Airbnb to understand the company better.

For Thanksgiving week (November 21-29), five of my friends and I rented a house in Las Vegas.

The total price for nine days: $2543.

Previously people went to Vegas to spend time in casinos or walk on The Strip. This year, the pandemic made it impossible.

It was not a problem for us. We did not plan to socialize; we were going rock climbing.

Red Rocks, located next to Vegas, is an excellent place for traditional, sport, bouldering, multi-pitch climbing.

The plan was for some people to take days off and climb every day, and for others to work three days remotely and join the gang during the rest.

Incident

We moved in on Saturday night. The next morning, excited, we woke up at 6am, had breakfast, verified that we closed back and front doors and at 7am drove to the Red Rocks Park.

We had a fantastic day of climbing, but after the sunset, we went home to realize that the window in one of the rooms was open and some of our belongings had disappeared.

I called the host, told her about the situation. She shared a set of images from the cameras and the next day sent the whole video.

We believe this is what happened:

At 8:55am, two hours after we left, someone

  1. Openly came to the front door.
  2. Knocked in the door.
  3. Waved Lexus car keys at the camera. (We do not have Lexus.)
  4. Ringed the bell.
  5. Waited to verify that no one was at home.
  6. Got to the back of the house.
  7. Put gloves and balaclava on.
  8. Checked if the back door was open. It was not.
  9. Checked if he can open the window. And it was possible. 10 Moved the pool chair below the window.
  10. Got into the house.

We called the police, and in two hours an officer came. He was polite and professional. He got the list of the stolen items, our version of the incident in the written form, and left.

I was curious, how did the burglar open the window? I checked, and the answer was simple: the lock on the window was not operational. No need to break anything, anyone can open the window from the outside!

If the weather was warmer, we would probably try to open windows the night we came. In that case we would discover the problem, but the night was cold and we did not touch the windows.

Till this moment, I was chill. Bad things happen. Burglars get into houses, and no one could be protected from it. We were safe in this incident, and from the money perspective, our loss was not huge.

But the fact that the lock on that window was not functioning is an issue (few more windows had the same problem). If the front door lock was broken it would be worse but even windows that could not be locked are sketchy.

In general, I prefer to blame myself for bad things that happen to me. This time I cannot figure out what was my fault. For sure, we could check all the locks in the house, but it is an overkill.

I am ok with making mistakes and paying for them. But this time, we paid for the host’s errors and Airbnb’s as a company.

  • Host: the house was not ready for hosting the guests.
  • Airbnb: limitations of the onboarding policy. I believe there is a list of things that the house owner needs to mark to become a host, and either functioning window locks are not on the list, or it is not enforced.

After we told the host about our discovery, she sent a person to lock all the windows completely. From now on, no one would be able to open them from inside or outside.

Contacting Airbnb

I wanted to reach out to Airbnb, tell them about the situation, and ask about the next steps.

Safety comes first. Hence I assumed that even if you are under stress and your brain is not functioning well, it is obvious how to contact the support team.

To my surprise, it is not the case.

I spent some time on the Airbnb website but could not figure out which phone number I should call.

Message to AirBnb: It would be great if you simplify the design of the support page. When we talk about safety, it should be about efficiency and not about the visual appeal or cuteness level.

I would like it to be:

  • I open the Airbnb website => I see an obvious button/link to the support page.
  • I open the support page => I see an obvious button/link to the hotline phone number.

I posted the tweet about the incident and tagged Airbnb and Las Vegas police in it.

My friends and I rented a place in #LasVegas at @Airbnb.

We went rock climbing this morning. When we came back the window in our house was open and 4 laptops, 2 headphones, @oculus Quest 2 headset, and @GoPro disappeared.

The year 2020 continues to surprise me. @LVMPD pic.twitter.com/f0YoHZsw43

— Vladimir Iglovikov (@viglovikov) November 23, 2020

A miracle happened. Airbnb contacted me and asked for my email to identify the account and to get more details.

My guess is that the Marketing team at Airbnb has alarms that trigger when someone mentions the company on social media.

Finally, after two hours, we got into the conversation about the incident, but the path to get there was not obvious at all.

Imagine how many people got into trouble in similar situations and did not leverage this communication channel?

I got an email:

My name is XXX, from the Airbnb claims team.

I am contacting you regarding the incident that occurred during your reservation with YYY.

As my colleague informed you in the previous email, we are only able to offer up to $500 for any stolen property. In order to proceed with this refund, I will need the following:

*Original purchase invoice for the stolen items.

This is requested as “proof of ownership”, if you don’t have the original purchase invoice, a picture of you where the item can be seen will also be accepted as proof of ownership.

All this story is not about money. But getting compensated for the host’s and Airbnb’s mistake with cash or AirBnB credits would be nice. It does not address the issue with window locks but sweetens the situation.

We interpreted the email as: “We will compensate up to $500 per stolen item”. Using the numbers from the invoices, it summed to $2600.

We collected invoices, sent them to Airbnb, and got the reply with words: “After additional review, I’m happy to report that we have just released a payout in the amount of $500 to your Airbnb account. You can confirm in your Airbnb Transaction History.”

The guess about up to $500 per item was overly optimistic.

After the end of the stay, I got a message from the host:

Hi Vladimir,

Thank you again for choosing our home for your vacation stay.

I hope the home met (and even surpassed) your expectations. It was a sincere pleasure hosting you, and I really hope to host you again in the near future. I would be truly grateful for a 5 star review of your stay when you have a spare moment and I would definitely do the same for you.

Also, please in the private comments if there is something, the home or myself can approve of please let me know.

I understand that this is a standard template, but under the circumstances, it sounds strange and does not fit the story and overall experience.

I did not plan to share the actual address of the property. The harm to the renting business could be material. But after this text, I changed my mind. It does not look like the host plans to revise the house and look for things that can and should be fixed. For example, the front door lock is barely working. The door could be open with a good push.

Conclusion

We had a great vacation. The weather was good, and the red rocks are remarkable. We had a lot of fun and plan to come back sometime soon.

Work laptops were replaced. We accepted the loss of personal items.

The Airbnb experience was not as smooth as we expected. Our things got stolen, and even on the other days, we did not feel comfortable leaving belongings in the house.

The host was responsive, but it is not enough. Communicating with guests and collecting money should not be the only responsibility. It is worth inspecting the house and fixing things that do not work as expected proactively, without waiting till the universe gives you feedback.

Overall, I believe that staying at AirBnB is safe, and such incidents are the exception rather than the rule, but, for sure, Airbnb has some work to do to improve communication and increase the guests’ safety. The compensation of $500 for all the stolen items does not look fair either.

I would like to end the story with some action items like: “Next time, when I rent a place, I will do XXX.” Nothing reasonable comes to my mind.

One option could be to do an extensive check of all locks in the building after moving in. Another is to take all the expensive belongings with me when I leave the house. I will do it in the future, but it may decrease fun level of my vacations.

What would be your action item after such an experience?

https://ternaus.blog/incident/2020/12/01/Airbnb-Thanksgiving-Burglary
I trained a model. What is next?
tutorialkagglepythonmachine_learningdeep_learning
I wish I knew it when I was active at Kaggle.
Show full content

Table of Contents

I participated in machine learning (ML) competitions at Kaggle and other platforms to build machine learning muscles. I was 19th in the global rating, got Kaggle Grandmaster title.

Every ML challenge ended with new knowledge, code, and model weights.

I loved new learnings but ignored the value that old ML pipelines could bring. Code stayed in private GitHub repositories. Weights were scattered all over the hard drive. In the end, all of them were deleted.

The situation is not unique to Kaggle. Same story in academia. The student trains a model, writes a paper. After it is accepted to the conference, pipelines are abandoned, training artifacts deleted and student moves on.

This article will talk about small steps that you can do after the end of every ML challenge.

These steps will:

  • Boost technical knowledge.
  • Build a personal brand.
  • Improve career opportunities.
  • Make the world a better place :)

As an example, I will use the repository https://github.com/ternaus/retinaface

It was not a part of a Kaggle challenge but was created to illustrate the story.

+5 min: Release code to the Public GitHub repository

Most likely, code is already at GitHub, but in a private repo.

What will you lose if you will make it public?

There are situations when private should stay private, but in your pet project, your Kaggle solution, or your paper, it may not be the case.

The most common obstacle that I have seen: people assume that all public code should be perfect and that they will be judged if it is not the case.

In reality, no one cares. Just do it. Release it as is, without any polishing.

Making code public is an important psychological step. Releasing non-perfect code is a confident, bold move.

Besides, all later steps are based on this one.

Example: https://github.com/ternaus/retinaface

+20 min: Improve readability.

You can improve the readability of your python code by adding syntax formatters and checkers.

It is not hard and not time-consuming. Checkers and formatters will not transform bad code into good, but the readability will go up. Think about fixing syntax as about basic hygiene. It is like brushing your teeth, but for the code.

I wrote a blog post on the topic called Nine Simple Steps for Better Looking python code. Feel free to check it out.

Step 1: configuration files

Add these files to the root of your repository.

Step 2: requirements

Install the required libraries with

pip install black flake8 mypy
Step 3: black

There are 100500 ways to format the code. Formatters like black or yapf modify the code to satisfy a pre-defined set of rules.

It is easier to read codebase that has some standards. When you work on the code for hours and need to switch a context between different coding styles, it drains “willpower energy” — no need to do it without a good reason.

Running

black .

will reformat all python files to follow the set of rules by black.

Step 4: flake8

Running

flake8

will not modify the code, but will check code for syntax issues and output them to the screen.

Fix them.

Step 5: mypy

Python does not have mandatory static typization, but it is recommended to add types to the function arguments and return types.

For example:

class MyModel(nn.Module):
    ....

def forward(x: torch.Tensor) -> torch.Tensor:
    ....
    return self.final(x)

You should add typing to the code.

  1. It makes it easier to read the code.
  2. You can use the mypy package to check arguments and function types for consistency.

After updating the code run mypy on the whole repo:

mypy .

If mypy found issues - fix them.

Step 6: pre-commit hook

Running flake8, black, mypy manually all the time is annoying.

There is a tool called pre-commit hook that addresses the issue.

To enable it - copy this file to your repo: .pre-commit-config.yaml.

You need to install the pre-commit package on your machine with:

pip install pre-commit

And initialize with:

pre-commit install

You are good to go.

From now on, on every commit, it will run a set of checks and not allow the commit to pass if something is wrong.

The main difference between the manual running of the black, flake8, mypy is that it does not beg you to fix issues, but forces you to do this. Hence, there is no waste of “willpower energy.”

Step 7: Github Actions

You added checks to the pre-commit hook, and you run them locally. But you need a second line of defense. You need Github to run these checks on every pull request.

Way to do it is to add file .github/workflows/ci.yaml to the repo.

There are lines:

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install black flake8 mypy
    - name: Run black
      run:
        black --check .
    - name: Run flake8
      run: flake8
    - name: Run Mypy
      run: mypy retinaface

that tell GitHub what to check.

I also recommend to give up the practice of pushing the code directly to the master branch.

Create a new branch, modify the code, commit, push to Github, create a pull request, and merge to master.

It is a standard in the industry, but it is exceptionally uncommon in the academy and among Kagglers.

If you are not familiar with these tools, it may take more than 20 minutes to add them and fix errors and warnings.

Remember this time. In the next project, add these checks in the first commit, when no code is written. From that moment, every small commit will be checked, and you will need to fix at most a couple lines of code every time: tiny overhead, excellent habit.

I would also recommend reading a book Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones. It talks about small changes in your behavior that improve productivity and the quality of your life.

III: +20 min: Create good readme.

Good readme serves two purposes:

  • For yourself: you assume that you will never use this code, but “never say never.” You will, and you will not remember what was happening here. Readme will help you.
  • For others: Readme is a selling point. If people are not able to tell the purpose of the repo and what problems it addresses, they will not use it. All the work that you did will not have a positive impact on others.

For machine learning repositories, the bare minimum is:

  • An image that tells what the task was and how did you solve it. No words should be required. Most likely, after working on a problem for weeks, you have 100500 pictures. They are just not a part of the Readme. Fix it.
  • Where to put the data.
  • How to start training
  • How to perform inference.

If you need to write 100500 words to describe how to run training or inference it, it is a red flag. You need to refactor your code and make it more user friendly. People often ask - how can I become a better programmer? This is the exercise that helps. You will need to rewrite your code. Try to look at your Readme from the eyes of someone else.

It is also a great exercise that will help develop a habit of looking at the product from the user’s point of view.

Famous: “Customers come first.”

Example: For retinaface, I wrote a wrapper over a model that hides details of the postprocessing.

Postprocessing is straightforward hard, but it does not add value to people that do not care about it => users should be able to hide it.

Readme created at this stage will be reused later when we will build a library.

IV: +20 min. Make it easy to use your trained model.

I guess you write

model = MyFancyModel()
state_dict = torch.load(<path to weights>)
model.load_state_dict(state_dict)

to load pre-trained weights to the model.

It works, and steps are clear, but it requires weights on the disk and knowing where they are. A more elegant solution is to leverage the torch.utils.model_zoo.load_url function in torchvision and similar in TensorFlow or Keras.

We can do:

from retinaface.pre_trained_models import get_model

model = get_model("resnet50_2020-07-20", max_size=2048)

If weights are not on the disk, they are downloaded from the internet and cached on the disk. The model is initialized, and weights are loaded.

This is user friendly, and that is what you see in torchvision and timm libraries.

Step 1: host weights of the pre-trained model

This was the biggest blocker for me. Where can I put weights for the model if you do not want to deal with AWS, GCP?

Apparently, there is an excellent solution, I would say a loophole. You can put weights to the releases at GitHub.

The limit is 2Gb per file, which is enough for most Deep Learning models.

Step 2: write a function that initializes the model and loads weights

In my case:

This functionality will be leveraged when we will build Colab Notebook and WebApp.

V: +20 min. Make a library.

In this step, you lower the entry point to use your model. The goal is to perform predictions without

git clone retinaface
Step 1: add necessary dependencies to the requirements.txt

You can use

pip freeze > requirements.txt

or update it manually.

Step 2: change the file structure of the repo

Create a “main folder,” in my case, called “retinaface,” the same as the repository.

  • Move all the important code there.
  • Do not move helper images, Readme, notebooks, or tests there.

Doing it manually and updating all imports would be painful. PyCharm or similar IDE will do it for you.

This is a common way to structure code in the repositories.

I hope, in the future, you will follow this pattern from the beginning. If you want something even more structured, check out Cookie Cutter package.

Step 2: add config file.
  • Add setup.py to the root of the folder with the content similar to setup.py
  • Add a version for the package. In my case, I added it to the init file of the “main” folder.
Step 3: create an account at PyPI

If you do not have an account at PyPI, it is time to create it.

Step 4: build a library and upload to PyPI
python setup.py sdist
python setup.py sdist upload

That is it. Your repository is a library, and everyone will be able to install it with:

pip install <your_library_name>

If you check the package’s page at PyPI, you will see that it uses Readme that you have in the repo to present the project.

We will use the functionality of this step for Google Colab and for a Web App.

VI: +20 min. Create a Google Colab notebook.

It is a good practice to add a Jupiter notebook to the repository to show how to initialize the model and perform inference. Example.

We can do better.

We enabled a “fancy” model initialization and the pip install magic in the previous two steps. Let’s leverage this.

We can create a Google Golab notebook.

Now, the only thing that someone needs to play with your model is a browser! More people will be able to check it out.

Example.

Do not forget to add a link for a notebook to your readme and update the version at PyPi.

VII: +20 min. Create WebApp

Many Data Scientists assume that building a web app is a complicated procedure that requires specialized knowledge.

This assumption is correct, the web app for a complex project requires skills that Data Scientists may not have.

Building a simple web app that demonstrates the model is easy.

I created a separate github repository for a web app. Still, you can do it in your repository with the model.

A blog post that describes details: How to Deploy Streamlit on Heroku

Step 1: Add code for the app.

Code

"""Streamlit web app"""

import numpy as np
import streamlit as st
from PIL import Image
from retinaface.pre_trained_models import get_model
from retinaface.utils import vis_annotations
import torch

st.set_option("deprecation.showfileUploaderEncoding", False)


@st.cache
def cached_model():
    m = get_model("resnet50_2020-07-20", max_size=1048, device="cpu")
    m.eval()
    return m


model = cached_model()

st.title("Detect faces and key points")

uploaded_file = st.file_uploader("Choose an image...", type="jpg")

if uploaded_file is not None:
    image = np.array(Image.open(uploaded_file))
    st.image(image, caption="Before", use_column_width=True)
    st.write("")
    st.write("Detecting faces...")
    with torch.no_grad():
        annotations = model.predict_jsons(image)

    if not annotations[0]["bbox"]:
        st.write("No faces detected")
    else:
        visualized_image = vis_annotations(image, annotations)

        st.image(visualized_image, caption="After", use_column_width=True)

Less than 40 lines.

Step 2: add config files

You will need to add files:

  • setup.sh - you can use this file without changes.
  • Procfile - you will need to modify the path to the file with the app.
Step 3: add requirements.txt Step 4: register at herokuapp Step 5: push the code
heroku login
heroku create
git push heroku master

You are live. Check out the example for https://retinaface.herokuapp.com/

VIII: +4 hours. Write a blog post.

Many people undervalue their work. They assume that if they know how to do something, everyone knows it. It is not the case.

One man’s trash is another man’s treasure.

Your article will help other people and improve your career opportunities.

I work at Lyft, Level5, and apply Deep Learning techniques to self-driving problems. Before Lyft, I worked at the debt collection agency TrueAccord. You can read about my job search in the blog post: “Shifting Careers to Autonomous Vehicles.”

One of the reasons I was able to do this career shift is because I shared my knowledge in blog posts and meetups. This attracted the attention of recruiters and hiring managers.

Worked for me - will work for you.

For machine learning, I would recommend writing the text that covers:

  • What was the problem?
  • How did you solve it?

If you read till this moment and found this article useful, you can say “Thank you!” by writing a blog post about one of the machine learning problems that you faced and how you solved it.

Example:

IX: Days. Write a paper that describes your solution to the machine learning competition.

Even if your paper is not a breakthrough, it will be published and have value to other people. Writing academic papers is a different skill, and you may not have it now. Not a problem. You can collaborate with people that know how to write in an academic format.

Examples:

Just look at my Google Scholar. Bump in the previous few years comes from the papers that were summarizing participation in different machine learning competitions.

Besides - your paper will not be alone. It comes in a package with:

  • Github repository that has clean code and a good readme.
  • A library that non-machine learning people can use.
  • Colab notebook that allows fast experiments with your model in the browser.
  • WebApp that engages the non-technical audience.
  • The blog post that uses human being language to tell the story.

It is not a paper anymore, it is a cohesive strategy that shows your “ownership” and “communication.” Both are crucial for your career growth, but I will talk about it next time :)

Translations to other languages Chinese
https://ternaus.blog/tutorial/2020/08/28/Trained-model-what-is-next