Models are illuminating and wrong

Previously: 2020 2020 2018 2016

I realised that I hadn’t written one last year publicly. I think I have written some privately but I can’t find the document. I’m going to try to keep up this “year in review” thing publicly where I can.

Firstly it goes without saying that this year has been a tough year for a lot of people in tech. The layoffs situation and the funding situation is rather bleak. However, one thing I realised when reflecting on the year was that there were a lot of

Key Learnings/Achievements

I finally sold Probabilistic Programming Primer, I listed it on Microacquire and the process was quite painless when it all went through. It was a nice end to a journey to build a small business, have it profitable and finally sell the business to another party. It was also quite a release because the overhead of running a side business with the day job was quite overwhelming for me.

One thing I realised was it’s important to personally and professionally take more photos – when I was reviewing photos of the year I realised there were months without photos in my google photos collection or on instagram. Make sure to invest the effort.

I’ve been swimming regularly about 3 times a week normally – I’ve found this very therapeutic and it’s become the exercise I enjoy the most. I’d like to keep this up and just keep the goal of consistency.

Aflorithmic

The macroeconomic environment has been tough, and navigating that has been a huge challenge as a leader. Something I’ve found rather helpful has been the excellent slack from Rands. The reality is that pretty much every technology company is looking or have executed cost savings, and the new interest rate environment we’re in has caused companies to emphasise profitability and free cashflow more – as opposed to just top line growth. Listening to how other companies navigate such issues has been helpful, if only the fact that these things are hard.

Personally I’ve been very proud of the team and the resilience they’ve showed, we shipped more machine learning based features than ever before, and while we had to iterate through some team setups we got to a setup I’m happy with for future growth and reliability. We also shipped a big refactor of our API – which will be released next year (email me peadar[at]aflorithmic[dot]ai if you want to be on the beta list)

Parrot Week: We had our first in person event in late 2022. This was a lot of fun, with a variety of remote, and the two offices in Barcelona/London it’s very important to get people together in one room to talk about strategy/ best practices or just understand each others personalities. It’s one of those things that advisors and friends mentioned a lot to me personally – and to my cofounders – I wish we did it earlier, and we’ll be sure to do it next year.

Careers are long: Some of my team have moved on to other adventures in recent years. One nice thing has been that some of those relationships are still quite strong. I recommend that you keep the weak ties and don’t take it so personally when someone decides to move on.

Customers: We continued to grow revenue and usage solidly this year. I’ve been very impressed at the variety of use cases our audio infrastructure can enable – from marketing, to synthetic avatars to gaming. And I expect we’ll see more use cases in the future. One lesson to learn again and again is that you can’t under invest in great customer support, and pulling the threads of some bugs from customers has led to some great new features.

Management I’ve been working a lot more with advisors and colleagues on people management. And the Rands slack is good on this too. Setting goals and being accountable to them is good, there’s a challenge of setting goals that aren’t too overly specific though as often I find the market demands change over the course of 3 months. Performance reviews and those sorts of processes – while they can be hard to get into motion in a fast growing startup do definitely help. A great blog post on this is here

Hiring One of my highlights hiring wise has been Hugo our BizOps and Strategy manager. The energy, good humour and ability to help us as a leadership team scale projects and processes has been hugely helpful. He’s also helped us focused on learning from our employees (which leaders are lost without). I was a bit skeptical of this role before hand but I recommend it to other startup founders. You realistically need to bootstrap that function.

Goals for the decade

(I’ve been heavily influenced by Will Larson)

• Write 4 blog posts a year – Quality not quantity
• Do more learning experiments each year – One of mine recently was Probabilistic Programming Primer and of course Aflorithmic. One thing I’m struggling with is how to do more programming.
• 10+ folks who I’ve managed or meaningfully supported move into VPE or CTO roles at 50+ person or $100M+ valuation companies. – I like this one slightly adjusted from Will.
• Build up a reputation for Aflorithmic as an innovative technology company where people can do their best work – This one is a big vague and it probably correlates with valuation to some extent. However, something I think we’ve invested a lot in – is getting a good culture and encouraging creativity. It’s very easy to “screw up the culture” so I’d like to continue to invest in this.
• Do something substantial and new every year that provides new perspective or deeper practice – I took this from Will directly, but for me it’s found a company and get us to our next milestones – it was probabilistic programming primer, and it was Open Source work a few years back. Perhaps in a few years it’ll be join a board or take part in angel investing. I’ll leave it vague for the moment because I like the general idea.

Other stuff

• We went to Morocco a fascinating country
• My brother got married which was a huge milestone for my family
• I didn’t do too much public speaking or talks – except this one which was kinda fun I’d not like to do too much of this each year but if I do one or two podcasts/ talks a year I think that scratches that itch for me. More than that would be a distraction

I’ve been very excited about AI for years, and have worked with Deep Learning, NLP, formal logic, and Bayesian Statistics methods for over a decade. However, one problem that we’ve had is “making AI useful”. We haven’t really had a new paradigm as such in computing for a while. I feel this is the top story in tech in 2022

Now it seems that it is Generative AI, foundation models and Large Language models.

A good talk is by Greg Brockman on Open AI which comments on some of the problems/ opportunities

We’ve seen a range of tools in a range of verticals, and a lot of VC commentary about Generative AI and of course the hype about ChatGPT so while for the last 10 years AI may have seemed like a ‘nice to have’ now it is very much affecting product evolution. 

One of the first things we saw in this space was in the Image generation space. 

Let’s give some definitions: 

Generative AI is an umbrella term for a number of machine learning methods — including Large Language Models (LLMs) and Generative Adversarial Networks (GANs). What has been most exiting in recent months have been the improvements in “foundational models” such as GPT3, PaLM, and various others – these are enabling media generation – at almost a human level. We’re also seeing some impressive innovation from companies such as Stability.AI and RunwayML

Source: https://www.craiyon.com/ 

At Aflorithmic we’re leveraging some of these technologies to make generating audio easier. Fundamentally what makes that exciting is that you’re leveraging technology that didn’t exist before to do “new things”. We had similar changes in mobile 10-15 years ago, and now it seems that there’s a coalescing of understanding, technological changes and consumer changes to unveil new products. We’ve also had improvements in moores law, an explosion of data on the internet (wikipedia for example) and improvements in training models. Not to mention a lot of innovation in the open source community. 

I’ve been working at this intersection for a while. And I simply want to bring some hard earned advice for working on AI-enabled products. I only want to encourage other entrepreneurs, to take advantage of this coming revolution. 

I think firstly like anything this is a new area to explore, and I simply want to remind you all there is no map. I’ve been working with these technologies on products for developers and consumers for nearly a decade. And I’m amazed at some of the innovation that’s happening. 

I’ve tried to distill some learning

So here are 5 principles for building good AI-native products and AI-native companies

• Distribution matters There can be all sorts of wedges like widgets, plugins or chrome extensions that can get you the distribution you need. You may need to reuse the same underlying technology in a variety of ways. Startups always need to find distribution
• User experience and a feedback loop matters. I feel we’re also still figuring out what’s acceptable in AI-native products – one of the good insights that Github copilot had was that it wasn’t ‘annoying’ as AI will get some things wrong (what is sometimes called “hallucination”), we need to be careful to not overwhelm users or “surprise” them. 
• Use R and D to your advantage. While it can be good to use existing APIs and existing models, you often need to tailor R and D to suit your workflows. So that means R and D/ML is crucial and you need to be building up those skills in your team, to have a long term competitive advantage
• Unlock new workflows. What does this technology allow you to do that previous paradigms didn’t? While it can be good to replicate existing workflows, it can be a good starting point. Great products allow you to do things that you couldn’t do before. One example in AI-products we’re already seeing is “generating 100 variants of the same media with slightly different parameters” that’s something that was hugely cost prohibitive before. 
• Latency matters – intelligent use of caching, the right models and the various tradeoffs of model size are very important for managing the cost of a product but also more importantly the customer experience. Some of these learnings only happen when you ship something to users, so ship iteratively!

So go forth and build – I can’t wait to hear what you’re working on!

Some further reading

• https://arxiv.org/pdf/2207.10342.pdf (A great paper on cascades – what’s the user interface for this) 
• A great blog by Nat Friedman is www.aigrant.org
• Intercom has a great blog on this too

If you have any feedback I’m peadarcoyle[at]gmail[dot]com or peadar[at]aflorithmic[dot]ai or you can find me screwing about on twitter – under springcoil 

At http://www.aflorithmic.ai we’re building infrastructure for creating audio experiences. Fundamentally, there’s a huge part of creativity involved there. And personally I believe the human creativity is almost limitless.

Previously I spent a lot of my career building tooling and infrastructure for data scientists and researchers. So while I’m not a super expert on Api design, I know a few things about building developer tools and thinking in terms of usability.

Einstein allegedly said something like “My pencil and I are smarter than I am”. So computers, and various technologies are only part of a long standing interaction between humans and tools. I’m fascinated by some of the literature on the effect of the light bulb on human cognition.

I’ve been recently looking at DALLE by Open AI and I think the same as Mark here. These tools will unlock a lot of human creativity. One challenge is the question of whether it ‘replaces’ human creativity. I’ve found in the past that a lot of creativity tools need prompts, there’s something about writers block, or as someone said “there is nothing scarier than a blank page”.

Sources:

A great book on industrial design which I found particularly eye opening is Designing for people

A friend said to me recently – ‘startups need to delight their users, and have an exceptional customer experience’. And that sounds like a trite, and repetitive sentence. But like a lot of cliches in life, it’s true.

I was thinking recently about how many experiences I have with companies as a consumer or a business user or a citizen that are frankly suboptimal. There’s various reasons for this, such as cost-cutting and misguided M and A. I’ve had a telco experience and banking experience that were awful. And a fintech experience with Monzo recently that was amazing.

For startups to win they need to delight their users. If they delight their users, your users evangelise for you and you have a word of mouth effect that attracts new customers. Often finding any ‘market’ is hard for a startup. It’s brutally difficult to find your first 10 customers. And there’s a lot we can do as founders and in startups to make our early customers have a great experience. There’s the anecdote that Wufoo sent each new user a handwritten note. And this worked for much longer than they thought it would.

So founders. Go out tomorrow and try to make your users happy and give them a delightful experience. If you focus on that, everything else will work out.

At http://www.aforithmic.ai we will be working on delighting our wonderful customers. We do this every week, but we can definitely do better!

peadar[at]aflorithmic.ai is my email so let me know your views if you have anything you think we’re doing wrongly on the customer side.

Great reading on this

http://paulgraham.com/ds.html

This is s short blog post. And is merely some links I came across this week.

• https://www.fullstackpython.com/wsgi-servers.html – a great introduction to WSGI servers in python. I hadn’t seen this written up before
• https://sudhir.io/understanding-connections-pools/ — Long read. This is a thorough and in-depth discussion of how connections and pools work. I must admit that while I had seen a bunch of the terminology before, I don’t think I really grokked this until reading this. It’s a great addition to your pocket.
• https://sudhir.io/the-big-little-guide-to-message-queues/ — a thorough introduction to message queues.
• https://developer.mozilla.org/en-US/docs/Web/HTTP — the Mozilla Web Docs are amazing and I was doing some training on APIs in general recently at my company.

I have a small document where I try to weekly write goals and what I’ve achieved. Sometimes I don’t update it, but I reviewed my 2020 version and will try to take out some takeaways.

1. Covid-19 was incredibly disruptive to our lives, but it accelerated ‘digital transformation’ in a number of industries, and we saw at Aflorithmic that some industries grew and others were forced to come up with ‘digital-first’ product offerings. Since we are focused on helping businesses enhance their audio experiences this was good for us.
2. Managing remotely is hard. Really important to spend a lot of time on zoom calls with your team. And it’s very easy with slack and emails to be misinterpreted.
3. I learned a ton of leadership lessons. Some of which exposed some of my failings (I’ll do another post on this at some point) and some which made me realise that when stuff goes wrong, it’s STILL your fault.
4. It’s really important as a manager to get into the trenches, it’s very easy for ‘metrics’ to lie to you.
5. We onboarded our first paying clients and pilots this year. And went through various iterations where we discovered what worked and what didn’t work. A key lesson we learned over and over again – is that you can be ‘too early’ for a market, and if you’re ‘too early’ it doesn’t work.
6. Learning a ton about building teams/ building platforms and the importance of great design and user experience. User experience is a first order problem to solve. Machine Learning is a second order problem.
7. Learning a ton about communication styles and about how important it is to spend time on whatever ‘story’ or ‘vision’ you want to portray to others. And it’s important as a leader to provide that. People need to understand how what they’re working on, connects to the big picture.
8. Learned a ton about fundraising, cashflow, growing a team, shipping product, negotiating with lawyers, fixing bugs, responding to customers and the importance of reliability and operational excellence.

Onwards and upwards. Lots more to learn. It’s great to reflect on how we’ve gone from an MVP to a working product in the space of a year and a bit.

I’m an Entrepreneur, and a few years before I officially became one. One of my best friends said to me ‘I’ve always thought you were very entrepreneurial’. I think I shirked away from that.

I’ve thought recently about why a I shirked away from that for a few years. I think one reason is that when I looked at successful entrepreneurs – the Reid Hoffmans, the Elon Musk. I compared myself to them and thought – ‘I can’t be at that level’. Yet they probably weren’t that polished at the start. We often make the fundamental mistake of misunderstanding growth. And startups are about growth!

I saw the following quote about DoorDash –

By the time a company is successful enough to IPO, the many doubts and uncertainties of the early stage startup are largely forgotten, replaced with a hindsight that makes their path to success seem so much more obvious and predictable. The founders are likewise transformed by the many years of struggle and personal growth necessary to build and successfully lead a large business. Gone is the uncertain new startup founder, replaced by someone almost impossibly formidable and experienced.

So if you’re thinking of starting a company don’t worry if you don’t have it all figured out. None of us do!

And if you’re talking to an early stage startup as a potential partner, employee or investor look past the challenges and look for a diamond in the rough! Maybe you’ll be lucky enough to stick money in an Airbnb or a Doordash

Everything great starts small! So go out and build!

I’ve been thinking a lot lately about Engineering Leadership. This is a very difficult job for all the reasons that leadership is difficult, management is difficult and that software development is a difficult discipline.

For various reasons – we as a species suck at writing software. Writing great software is HARD. Nevertheless we as a community are learning a lot more about this, and we at least have some patterns that work.

So I wanted to share some lessons I’ve personally learned over the last few years from operating in teams and building teams. I’ll also link them to a sport I love – Rugby. Even if you don’t know Rugby, the lessons should still be relevant.

1. Know your role: It’s important to know your role and operate in that role – a mistake a lot of leaders and operators make is that they either try to do someone else‘s job – and then try to do too much and fail. Or they don’t realise that they’re part of a larger system. In Rugby – this is like situations where the defensive line becomes misaligned. If there’s not a solid defensive line other teams can attack. Alignment is super important in executing any strategy, and not forgetting that as a leader a big part of your job is to set the direction for the team.

2. Building and forming trust is important: A lot of training for sports is about forming trust among the team. This can be fostered by drills etc – however it’s often best formed by performance. This also means though that like in sports, the relationships matter. If the team doesn’t trust the coach, or even within the team, such issues need to be addressed.

3. The competition are human too: It’s easy to respect your competition too much, or even think you can’t win a big customer because your product isn’t good enough. But self-belief matters, and you need to play your own game-plan in Rugby, in business you need to play your own strategy which suits your team.

4. Best form of defence is attack: In Rugby it’s often better to not focus on defensiveness, it’s best to just attack. In startups it’s the same thing – the most debilitating thing for a competitor is to simply be able to move faster than them. Speed wins in business as in Rugby. One of the questions to ask is what you can do as a leader to encourage agility and speed. It’s often worth thinking about what you can do to improve the speed of the team. In sport that may be better training, or better drills. In business – there’s a lot you can do to encourage speed – send that email today, get that prototype done today, don’t wait until next quarterly planning to schedule that sprint. Set your operating cadence to run.

5. You have to analyse and learn from what went wrong: In software we often have ‘postmortems’ about outages. A fundamental part of that, like with sports is that in a fast moving environment you need to adapt to change. It’s sort of a cliche. But it’s good to be honest with yourself about what’s working and what isn’t. Fundamentally though this is hard because people feel shame. And it’s easy to regress to ‘blaming others’. Remember you’re all part of the same team. This is where techniques like 5 why’s can come into it. Because often it’s more about the system than about a particular player. It’s easy in sport to say think – ‘it was the player who missed the last tackle who gave it away’, but often it’s poor alignment, or other mistakes that happened earlier in the game that caused things to go wrong.

Those are just some thoughts, I’ll keep fleshing this out. Thanks for reading!

Some good resources on Engineering Leadership and Postmortems:
https://pragprog.com/titles/jsengman/become-an-effective-software-engineering-manager/

At Aflorithmic we recently have been experimenting with Friday Wins. It’s important in a startup to enable the company to build things and talk to customers. This matters a lot to ‘building something people want’. However, I wanted to talk a bit about a meeting style that I’ve found that works pretty well, and why it does.

Culture is what you celebrate! And in software I firmly believe we should celebrate product getting out to customers and what we learn. It could be internal customers, it could be that a prototype we shipped – didn’t work (this happens often in ML stuff), it could be a hypothesis we invalidated.

Well it’s very easy I think as humans to be scared to show things to get feedback. Let’s use an analogy – I’ve got friends who love music, and create music. But they don’t share their art with other people. This fundamentally stops the feedback loop and means you don’t actually know if you’ve made something that people want.

Why does this happen? I think one of the reasons is to protect the ego. It’s easier to believe that what we’re working on is great, and it’s difficult to handle potential rejection.

These are evolving as rules but these are the rough rules we have for Friday wins.

Focus on shipping

I think it’s really important to focus on shipping. Nothing matters until it hits a customer and it’s very important that shipping is central in your rituals. And shipping is a primary measure of a teams effectiveness and shorter cycle times lead to better feedback, better morale and better learning.

We even have a picture in our offices devoted to this

It’s an inclusive and high energy meeting

It’s super important that the intern can share their work, it’s also important that it’s something people look forward to. When people share on slack during the week – ‘we implemented this feature and it enhanced our process by 10%’ I tell them ‘this is great for a Friday win’.

Not everyone needs to have a win every week – some teams have longer cadence – but everyone in the engineering org should be invited and able to participate.

5 minutes summary – and focus on the customer

This is hard because sometimes things over run – but I try to make people keep their presentation to 5 minutes. You don’t need slides – but try to focus on the most general take home points. And why it is cool.

Sometimes it’s very apparent that a technology needs a longer talk – and that’s ok that can go into a ‘tech talk’ meeting which is a longer one and happens every week.

Wins and modern software is cross functional

A win is very general as a word. It doesn’t mean ‘this moves the needle of the company in a game changing way’. Graphs count as wins, UIs count as wins, analysis of customer requests count as wins, a version of an AI model (that is in some version of production – it can’t be on someones computer locally) counts as a win.

Learning oriented

As the CTO of a tech startup, it’s very important that people in my teams look forward to reviewing their work. Reflection is an important part of the learning cycle – and a weekly cadence helps with that. For example from last week – someone said ‘we learned this, and a better way to do this is to store our parameters this way’ or ‘we built this model and we discovered that the technique doesn’t work with the amount of data we have’. These are all VALUABLE contributions to building a great product and help us serve our customers better in the future. Through better product innovation

How software is made is important

I firmly believe that since ‘software is eating the world’ that all companies are becoming some form of a ‘tech company’. And we see this in the world, in terms of investment – and in terms of how significant ‘tech’ is becoming (for example see the public markets). Therefore if every company is a tech company it’s very important to build organisational awareness of how software is made. One way we’ve done this is we invite someone from the commercial side each week, and it’s good when they give an update on how partnerships, marketing or their speciality is working. This helps a lot with momentum and team spirit. And as a leader – your job is to build a culture.

These are some of my comments. I call it ‘Friday wins’ because that sounds better than ‘sprint demos’ or ‘sprint reviews’. It’s very important that we celebrate the work we do and how we get stuff out to customers as soon as possible. And I’ve found in a few weeks of running it, that it’s great that the knowledge sharing happens.

Building software is hard – and modern software is a difficult team sport. To build great software we all need to learn – largely because it’s a complex act, and the pace of technological change is very high.

I firmly believe that the way to do that is to create a learning culture. This meeting is evolving but I wanted to share what we learned so far

What meetings do you have at your company that you want to share?

If you want to reach out to me to talk about this stuff – I’m firstnamelastname@gmail DOT com or you can easily find me on Twitter

I’m going to use a bit of a click bait title for this article. But the aim of this article is to share experiences I’ve gathered from about 10 years building ML systems, and building ML teams.

I saw the following tweets by Erik so I’ve added them.

The key word here is ‘the plumbing around it’. I think looking back at the whole ‘big data’ wave and the ‘Machine Learning’ wave, is that there was a lot of vendor-driven hype around ‘technology’. As I wrote in this post

biggest mistake teams make in ML is they don’t focus on getting something working end-to-end. They either go off and yak-shave a large infrastructure project, OR they build a complicated model without the infrastructure that’s needed for that model. When something is in production and can be shown to users it has a LOT of value – they can respond, and the model can improve. Get something into production. Not just a Jupyter notebook.

Peadar Coyle

I sometimes call this the ‘hackers news’ effect. Which I’ve often found in teams. Where someone comes to me with ‘we want to use deep learning because Airbnb/Uber/Google/Stripe/Etc is using it’. They may explicitly say that or they may say different words but have that effect.

The purpose of any team is to add value to the business. So don’t call yourself a programmer or a data scientist.

I’ve just spent near two years building out our pipelines for audio production at my startup and the resulting core value in terms of business value is rarely the ML side. This isn’t to say that ML isn’t important, but it’s a small part of the puzzle.

• Algorithms are just cogs in a system
• Algorithms are dumb and are often optimising for a specific use case
• Algorithms live in a mess

Most ML systems live as part of an end-to-end system. In the case of our audio production system. There’s the following rough parts

• A UI for writing text and annotation
• Data collection and automatic transcription of audio for training jobs
• ML models for synthetic voice
• cron jobs for copying wav files and mp3 files from system to system
• A business rules engine for quality insurance
• Audio post production process
• An asset store for storing assets and delivering them via an API

All of these are connected via batch processes and API calls and various cron jobs. And testing this stuff end to end is tricky and involves a lot of nuance/ understanding of a domain specific problem.

ML is just one part of a much larger system. It’s frankly a messy process, and it can often be very difficult to figure out what’s broken. However, in terms of leverage – our leverage is often on the automation side. Improving the reliability of our software allows us to onboard more clients, not to mention offer more product features.

There’s an element of human-in-the-loop here as well. Especially for focusing on edge-cases.

So what should you do?

• Focus on understanding business problems, if you can reason about business and talk in trade offs – you’ll always have a job.
• Learn to sell your work as a data scientist – you need to be able to communicate results
• Work on your software skills – learn stuff like React, Vue, Serverless etc – so you can show results end to end
• See yourself as a problem solver.

We’re probably seeing a bit of pushback against AI/ML etc. However, we shouldn’t neglect that we are seeing process improvements and enhancements in productivity. However, these are often due to automation and the interaction of humans with algorithms as opposed to anything else.

Further reading

Don’t call yourself a programmer
The future of Data Science is past