The LLM space moves fast - wildly fast. This is really exciting from the role of experimenting with new technologies but is a source of heartburn when trying to figure out how to actually use these systems and build products on top of them. Last post1 I mentioned how technical limitations of GPT required only sending through a subset of text because the word limit for the model was less than a podcast transcript. I also had this comment buried in a footnote:
For now at least, I’d expect the cost of these systems to drop over time and for the number of possible tokens to be fed in to increase.
Well 10 days later OpenAI announced gpt-3.5-turbo-16k
, which removed the technical barrier of just dumping an entire podcast transcript into the LLM and getting a result back.
Great - was all that work of trying to programmatically find ad text in a transcript useless? Is this a technical moat I lost overnight because of a larger LLM context?
Thankfully, no. This time at least.
After a few days of futzing and prompt “engineering” I was not able to find a reasonable way to accurately return ad content from a transcript. Thats not to say its impossible, but a feature that I’m keen on building isn’t necessarily at risk of just being an API call away from being copied.
There’s a ton of ChatGPT use cases flying around these days and anything thats just wrapping the OpenAI API isn’t really a defensible product. If your “product” is nothing more than a specific UI/UX wrapping a single API backend then you are at the mercy of both that API and anyone else who wants to challenge you on cost. This is the realm of side projects and beer money.
This is the fear of the froth of a fast moving technology - that one day you wake up to doomsday. Doomsday is when your product has been made irrelevant by an advancement in a vendor’s API and your competitive advantage has disappeared. Your product is just a hacka-a-thon away from a less capital intensive copy cat willing to sell at a cost you can’t compete with.
This need to de-risk a product from advances in a vendor’s API is broadly how the Siev data stack works - there are many vendor APIs put together, each producing a unique datapoint that can be combined into a data product that is not just an extension of a single API. It can’t be copied in a hack-a-ton and the nuance of why it works is the value - at least until GPT 5 comes along.
Bullet dodged this time, but this risk that products are just one LLM advancement away from being irrelevant is going to hang around for a while. I’m looking out for doomsday, trying to avoid the trap as much as possible, but the risk is always there.
I’d love to use the phrase last week here but the new Zelda came out and sucked more of my productivity time than I really would have liked. I suspect many work-from-home ICs were moving just a bit slower over the past two weeks.