Teddy Liu

During yesterday's LibreQoS APNIC webinar, I posed the question: won't more bandwidth solve these problems? As Herbert Wolverson said, yes, more bandwidth is good, but, still these problems remain if queueing is done incorrectly. Here's my take on why bandwidth alone is not as good as bandwidth + good queueing policies: Generally it was believed that 'data is important, so don't throw it away; hang on to it and send it later'. In practice, this has proven to be suboptimal as two issues may emerge: 1) latency increases for some flows due to heavy demand from other flows using the same bottleneck link, 2) even a single flow can have excessive latency due to aspects of typical TCP behavior (referred to bufferbloat) when the buffers grew large enough that the data being buffered was retransmitted anyway. It turns out that not all data is equally important. Active Queue Management is the art of deciding priorities, both in deciding what data to throw away, but also in allowing some later arriving data to be transmitted ahead of data in another connection that arrived before it. These problems have been studied, and good solutions have been found by using certain queueing policies in routers and switches, referred to as “fq_codel’ and “cake”. These track the different flows not by classifying the data, but by watching the behavior. Flows that send relatively little data (DNS lookups), or at a measured pace (video chat) have priority over flows that send a lot of data as quickly as possible (App and System updates, Video and ISO downloads).

9

5 Comments

Nic Poulos and I wrote an essay for Euclid Ventures, "An Era for Vertical Software." The TL;DR is that vertical SaaS is a distinct, enduring discipline that we believe is poised to dominate the software landscape over the next decade.   "2024 will represent a turning point, perhaps even a new era for venture capital. While the fog feels like it’s lifting, the path forward isn’t yet clear. One includes a soft landing for the US economy and the continuing recovery of software multiples, with ZIRP-era behaviors continuing around hot themes. Another sees a recessionary period in which VC capital inflows remain stagnant and startup funding the year dips below $100B. Either way, a more measured VC market in 2024 is a certainty. For many investors, that will come with a rude awakening as their strategy becomes more difficult to execute or their thesis of choice is negatively impacted by tightening markets.    We see this reversion of the venture ecosystem to its mean as a strong tailwind for Euclid and vertical software. As a category, Vertical SaaS is uniquely well suited to thrive in volatile market conditions and produce fund-returning outcomes. The sections below aim to answer the question: why is vertical software different? Our answer discusses the factors driving the natural market resilience we see in vertical software as a category–resilience that will help them thrive in 2024 and beyond."

117

16 Comments

Authorization is one of the more complex (and, let's face it, thankless) tasks about building an application. Oso has emerged as the leader in abstracting #authorization as an independent service; just like authentication.

43

Want a front-row seat to the frontier of tech? We’ve got the role (or two) for you! We’re expanding our frontier-tech team at Blackbird by hiring a Frontier Tech Investments Associate and Foundry Fellows! Got questions? We've got answers... 1️⃣ What are the roles? 🧪 Associate = a full-time VC investment gig in our Blackbird Investments team, working directly with me. We’re looking for someone with a science and/or engineering background and more curiosity than they can handle. You’ll grow your own investment brand and practice, support our portfolio founders, and will help build Foundry, our early-stage frontier-tech accelerator. ✨ Foundry Fellow = a casual/contract gig in our Blackbird Investments team, ~15h per week for 3 months. The Fellowship is ideal for PhD students and ECRs who want to learn more about startups and VC. You’ll go deep on emerging areas relevant to your expertise (or curiosity!), get a front-row seat to groundbreaking companies in those areas, build out your non-academic network, and develop a solid writing practice. 2️⃣ Why are you hiring? We love frontier tech, and we’re ready to grow our team. 3️⃣ Wow, it’s so great that you’re starting to look at deep tech! Look, we get it: we don’t make a lot of noise about our frontier tech investing. Buuuut we’ve been deep-tech investors since we backed Tim Kentley-Klay to found Zoox back in 2014—we’ve been on incredible journeys with PsiQuantum (building the world's first utility-scale quantum computer right here in Australia!), Inventia Life Science (transforming drug discovery with high-fidelity cell models), Remedy Robotics (surgical robots for remote endovascular procedures), Opto Biosystems (minimally-invasive neural implants to treat cancer), and more. We believe that frontier technologies, and great frontier-tech investing, will be part of the solutions to the greatest problems humanity faces today. 4️⃣ When do applications close? May 31st at 11:59pm AEST. 5️⃣ I have more questions! I’m sure you do! Clare Birch and I are hosting an AMA to answer any and all questions about these roles. Want to know what a week in the life of our team looks like? What’s keeping us up at night? What our ideal candidate looks like? Come along and find out - registration link in the comments 👇 Apply for these roles: Associate - https://lnkd.in/gCfj4EUJ Foundry Fellowships - https://lnkd.in/gj6ATZVZ If you know anyone that we should meet, send me their details! Cameron Elise Ben Andrew Robin Joseph Adelaide James Olivia Lucinda Raghav Jesse Christie Mohamed Tom Amee Pablo Haya Loong Hon Joshua Benjamin Megan Harry Denzil Matthew Diana Daniel Tom Deanna Justin Amar Lilly Stone Thomas

114

4 Comments

An excerpt from our weekly newsletter: (a link to subscribe is in the comments) Enterprise software malaise. The biggest news in tech this week seems to be a widening sense of capitulation around enterprise software valuations. Jamin Ball of Altimer Capital wrote that software sentiment had crumbled. There are concerns around AI, but the main issue seems to be a slowdown in growth in general. As Jamin writes, “growth hasn’t rebounded - there was no v shaped recovery. In fact, forward estimates are starting to go down, not up. Sure, the strong headwinds have abated, but the headwinds haven’t been replaced with tailwinds. Everyone is still waiting for growth to “re-accelerate,” and a number of companies are implying this will come in the back half of the year, but will it? Feels like investors are broadly more comfortable waiting for the data to show up. What’s more likely is the current state we find ourselves in is likely to be the equilibrium.” Anita Ramaswamy, writing in The Information, argues that the selloff and the collapse of sentiment is overdone: “The market’s panicked reaction may be overdone. It’s far too early to judge how much companies will shift their spending from existing enterprise software firms to new AI services, and when. Given questions about large language models’ accuracy and cost, which have caused many businesses to hold off spending on AI-powered services so far, the near-term impact on software spending may be minimal. And, of course, software companies such as Salesforce are adding AI tech to their existing products in an effort to cash in on the AI boom. That complicates the issue, as some software believers think AI can help drive growth for older companies, while others think it will kill them. Whatever happens, the shift is likely to take years.” Jared Sleeper pointed out that software is just naturally growing slower as it goes logarithmic. “As bad as all of that sounds, there is nothing to fret about here per se over the long run. It isn’t surprising that an adjustment away from behaviors learned over two decades. It is actually an altogether more benign explanation for the current malaise than some of the others circulating. In this scenario, SaaS still has a healthy decade of well-above-GDP growth ahead of it (without considering AI’s impact). There are still plenty of exciting startups which for one reason or another are less mature than the sector overall and merit investment levels that harken back to headier days. We're experiencing a normal adjustment period of a disruptive, exponentially growing technology- not an existential crisis.” Tomasz Tunguz also addressed these concerns, pointing out that “In the last 12 months, public software companies will grow on average 17%, which will add $100b in revenue across these businesses.”

13

3 Comments

💡 Why do we need so many evaluation tools for LLMs? As engineers, we build 'production-ready' LLM products using these metrics. But what happens next? How do we maintain control and ensure reliability? At Feedback Intelligence, we’ve crafted a cookbook to keep your LLMs reliable and aligned with user expectations. 🍲 📖 Give it a read and let’s chat!

15

Fascinated by the capabilities of agentic AI as discussed here! The example of an AI adapting during a live demo to overcome unexpected challenges showcases its potential for business applications, particularly in automating and enhancing decision-making processes. Excited to see how these advancements will transform operational efficiencies in various industries.

7

3 Comments

I think it's import to know that large language models generally suck when it comes to numbers and why. I recently read an article on how poorly LLMs reason, and the problem they used was of the "Bob has X sisters and Bob's sister Mary has Y brothers, how many brothers does Bob have" variety. This requires the LLM to work with numbers by calculating how many siblings there are in order to produce the right answer, and it's something that LLMs have fundamental problems with. It's not an example of poor reasoning, it's an example of LLMs being bad at working with numbers. And they're bad at working with numbers because of "tokenization". Computers can't work with text. They can only process numbers. If you want a computer to do anything with text, there needs to be a numerical representation of the text for the computer to work with. And the way LLMs like GPT-4 turn text into numbers coincidentally makes it very hard for them to work with numbers. For computers, text needs to be split up into smaller parts, and each part is assigned a unique number. If we give each letter in the English alphabet a number like a=1, b=2, etc. then the computer can say 20, 8, 5, 19, 9, 19: 20=t 8=h 5=e 19=s 9=i 19=s And we can see: "thesis" That's what tokenization is. The letters are tokens. 's' is one token with the number 19, and this token appeared twice in "thesis". But separate letters make it difficult to understand the higher meaning behind words. Think how hard it is to understand what someone is saying if they spell every letter individually. It's even worse for computers. It might seem reasonable to just use words as the different parts. "thesis" could just get it's own number. Let's say '1'. And we'll give "the" the number '2'. Then '2,1' would mean "the", "thesis". That approach didn't work though. There's too many words! Especially when you want to include other languages like Mandarin. What modern language models use instead is tokens that are 1-6 characters long. Not just letters, but not full words either: they use tokens as building blocks for sentences. For example: "the" = 1 ' ' (space) = 2 "sis" = 3 "ter" = 4 "the thesis" is represented as 1,2,1,3: 1: "the" 2: " " 1: "the" 3: "sis" "the sister" is represented as 1,2,3,4: 1: "the" 2: " " 3: "sis" 4: "ter" We re-used token 3 ("sis") to represent both "thesis" and "sister". That's how LLMs build sentences. There's a token visualizer for GPT-4 here: https://lnkd.in/gpuQrGP3 So why do LLMs perform badly when it comes to numbers? Because those numbers are also represented as tokens! "5 + 12 + 3 = x" could be represented with the tokens "5 + 12 + " "3 =" " x" It's a mess! LLMs can't really do math, because the tokenization makes it impossible for the LLM to operate on a number basis. It can't separate out "+" or "=". For math, we'd have to go back to the separate-letters method. A model can either produce poems, answer questions and hold conversations, or do math, but not both.

7

I really like this PMF Framework from Sequoia. https://lnkd.in/gT2nWttx

25

2 Comments

To everyone finding it challenging to grasp how to utilize Large Language Models (LLMs), think of LLMs as an operating system. Instead of trying to develop your own LLM modules, which most companies lack the resources or budget to do, focus on building applications on top of existing LLM platforms. It’s similar to how apps are created for iOS or Android. #genai #llm #llmapps

15

1 Comment

It's rare to get a shout-out in a CEO's earnings call -- I remember two times when @zuck mentioned our work on enabling GPU Inference and the impact it had on our business in general, and on Reels in particular (+30% viewing time in the first two weeks after release!). I think it's even more rare to get a shout-out in the earnings statement of the CEO of another company. But that just happened!  With the ARM CEO’s earnings release statement here => https://lnkd.in/e8GyGygE with a shoutout to Llama and PyTorch/ExecuTorch. Even more amazing that PyTorch/ExecuTorch is the only framework called out! A big shoutout to the ExecuTorch team and the PyTorch team at large for delivering this amazing software stack that’s driving AI across the industry and is now making the dream of on-device AI a reality — it’s a true privilege and a great pleasure to work with them! I'm looking forward to the software optimizations being promised in the statement, in the hope that they will deliver additional quality and performance boosts in Executorch/PyTorch performance and running Llama3 on-device even better than it already does => https://lnkd.in/eA6AfCpu. A big thank you for the shout-out from ARM, and to the long running collaboration. I am looking forward to making on-device AI even better #together with our corporate partners and with the open source community, one optimization and one kernel at a time. If you haven't checked out ExecuTorch, this may be a good time to check out our GitHub repo => https://lnkd.in/g9uVqKgR, see how it works in this overview on PyTorch.Org => https://lnkd.in/e8CpE-eP and set up Executorch => https://lnkd.in/e68DzSrc Then, give it a spin and tell us how you are using ExecuTorch in the comments! And if yoou find ExecuTorch as exciting as we do, please join us and contribute your ideas and code to make high-quality on-device AI even better! #PyTorch #PT2 #ExecuTorch #LLMs #AcceleratedAI #MetaAI #Meta

145

3 Comments

Ever wondered how fine-tuning impacts the safety of your LLMs and applications? Find out below... I have been having a lot of conversations with folks who are building with LLMs, and one of the most common topics that comes up is the impact of fine-tuning on the safety and security of an LLM. Sometimes it comes up as a question and sometimes as a myth that fine-tuning makes my model "more secure" because it will only answer based on the fine-tuning data set. We at SydeLabs will soon be publishing a more detailed report around this, but here is a very quick experiment we did with one of the most popular LLMs. We fine-tuned it on a fairly small dataset (~1000 rows of data). The data involved simple questions and answers with no toxic or unsafe content. Here are the results of the responses we got for a question before and after fine-tuning the model.

64

Vishal Sachdev highlights the strategic integration of open source and proprietary tech in architecting tech stacks, developer ecosystems and resulting business models. The world class companies effectively balance value commoditization in open source and value capture in proprietary tech. This is the strategic challenge for companies operating in #blockchain and #AI. I believe those who operate at the intersection of blockchain *and* AI stand to win this strategic battle...

3

1 Comment

In other news: Meta launched Llama 3 last night. 16 hours later, Simplismart has come up with this update. Finest real-time example of what you need in early-stage teams: bias towards action. My guess is this team managed to work all night and come up with this in less than 15 hours. Fastest inference engine in ≤15hrs? Crazy! Simplismart ❌ Simpli-FAST ✅ Amritanshu Jain Devansh Ghatak

21

Symflower Launches DevQualityEval: A New Benchmark for Enhancing Code Quality in Large Language Models Symflower has recently introduced DevQualityEval, an innovative evaluation benchmark and framework designed to elevate the code quality generated by large language models (LLMs). This release will allow developers to assess and improve LLMs’ capabilities in real-world software development scenarios. DevQualityEval offers a standardized benchmark and framework that allows developers to measure & compare the performance of various LLMs in generating high-quality code. This tool is useful for evaluating the effectiveness of LLMs in handling complex programming tasks and generating reliable test cases. By providing detailed metrics and comparisons, DevQualityEval aims to guide developers and users of LLMs in selecting suitable models for their needs. The framework addresses the challenge of assessing code quality comprehensively, considering factors such as code compilation success, test coverage, and the efficiency of generated code. This multi-faceted approach ensures that the benchmark is robust and provides meaningful insights into the performance of different LLMs. Read our full take on 'DevQualityEval': https://lnkd.in/guRuBjaB GitHub: https://lnkd.in/grssvCVR Symflower #artificialintelligence #ai #datascience #llms

44

3 Comments

Thrilled to announce the launch of our inaugural edition of Beyond Benchmarks at Emergence Capital. This comprehensive report dives deep into the metrics and trends shaping the early-stage enterprise cloud market. A huge thank you to our VC partners and contributors for making this possible! Here's a sneak peek of our findings: --> 60% of companies have already integrated GenAI into their service offerings, with another 20% planning to do so this year. --> While most companies use OpenAI as their primary LLM, many are experimenting with multiple models. We’re seeing a trend toward intelligently routing GenAI inference requests based on cost, performance, and security. --> Companies that have implemented GenAI are showing promising results, with a 7% higher NDR compared to those that haven’t. Beyond Benchmarks goes further with more GenAI trends, insights on the current fundraising environment, and key performance metrics. Our goal is to provide founders and their teams with valuable benchmarks to help them make better-informed decisions. At Emergence Capital, we're committed to helping founders build iconic companies. Dive into the full report here: https://lnkd.in/g6bnvAZM

70

2 Comments

In this week's episode of the Mobile Dev Memo podcast, I speak with notable X personality Trung Phan about the evolving social media landscape, and how to tame the mayhem and pandemonium of X. https://lnkd.in/dVHMzbui

1

There is an obvious need for standardized metrics across the decentralized computing marketplace landscape, which is why Sami Kassab wrote a killer blog this week proposing the use of Floating Point Operations per Second (FLOPS) as that standardized metric. In yesterday’s Big Brain Breakdown Outpost Strategies, we expand on Sami’s original post, and add some additional context of our own. Here are the key takeaways from yesterday's breakdown: > The decentralized computing marketplace sector lacks a standardized metric that measures a marketplaces total computing capacity, making it difficult for investors to accurately assess the value of these projects. > Sami proposes using FLOPS (Floating Point Operations per Second) as this standardized metric. FLOPS is widely recognized in high-performance computing and can provide a clear, hardware-agnostic measure of computational capacity. > FLOPS can accurately represent the computational capacity of diverse hardware, including GPUs, CPUs, and specialized accelerators like TPUs and FPGAs. This standard helps users assess the feasibility and cost of tasks like AI and ML workloads in real-time. > 6079’s Proof of Inference Protocol (PoIP) is one solution that can be used to verify and record FLOPS onchain, ensuring transparent and accurate reporting of computational resources through cryptographic proofs and game-theoretic mechanisms. > FLOPS should be complemented with additional metrics such as benchmark scores for AI/ML workloads, memory bandwidth, and energy efficiency to provide a complete picture of computational resources. This combination ensures a robust and sustainable growth of decentralized compute marketplaces. > The adoption of standardized metrics like FLOPS benefits users by providing clear insights into computational capabilities, aids investors in making informed decisions on the value of a project’s marketplace, and supports legitimate projects by highlighting their true value, thereby weeding out low-value, vaporware projects. Of course, we go into much more depth in the full breakdown. If you thought that was interesting, go ahead and give the full piece a look and do not forget to read Sami's original piece. He is an excellent writer! https://t.co/BtVuG82fcB https://t.co/ryQf2e8Zuu

9

On AI landscape, my current working hypothesis is: Model layer: commoditized other than specialized verticals like bio. Infra layer including compute capacity: hyperscalers like nvidia, Amazon will win. Groq likely gets acquired by Google, Amazon or Microsoft. Dev tools: Most startups will not monetize but some will emerge as large winners. App layer: This is where there could be 1K+ winners. Would love to hear where I’m wrong or missing the mark completely.

91

24 Comments