abusing sqlite to handle concurrency
I wrote a post for the SkyPilot blog. It's pretty interesting. You should read it. <https://blog.skypilot.co/abusing-sqlite-to-handle-concurrency/> May...
Abusing SQLite to Handle Concurrency
blog.skypilot.coSkyPilot uses the venerable SQLite for state management. SQLite can handle millions of QPS, and terabytes of data. However, our efforts to scale our Managed Jobs feature ran up against the one downfall of SQLite: many concurrent writers. Since SkyPilot typically runs as a CLI on your laptop, we wanted to stick with SQLite, so we decided to figure out how we can make it work. (Update: SkyPilot can now be deployed as a remote team server and supports PostgreSQL!