Mohan

## DJI_20260626154558_0027_D Hello, today we met Mohan. Hi, Mohan. Nice to meet you. Let's do some introduction about yourself. I heard you were a LIM researcher. Yeah, I was a researcher actually before the LIM era. So I did my computer science degree back. So I entered the University of Toronto back in 2015. 2015. Yes. And I studied computer science and back in 2017, actually in the University of Toronto, we have a thing called POST, which means that you get to choose what your major is and then you get to choose your specialty. So for example, in computer science, there are different specialties, including machine learning, including artificial intelligence, including computer graphics, including human computer interaction. So when you are like in your third year, you got to choose which one you want to specialize in. Okay. So in the beginning, you learn everything. And for the third year, started to choose. So what did you choose? I did choose like artificial intelligence. Okay. AI. Yeah. The reason is like, you know, Jeffrey Hinton. Oh, yeah. Hinton. Hinton was from University of Toronto. Yeah. Did you? I was not like, I don't know, like brilliant enough to join his research lab. But basically he and many of his like students were like still teaching at the University of Toronto. So I was working with them. And I was part of the was part of the early team of early research assistants of Vector Institute back when it was founded. What institute? Vector Institute that was co-founded by Jeffrey Hinton to basically establish the broader interest in artificial intelligence research and commercialization as an effort back in Toronto. Back in Toronto. Oh, okay. So yeah. Oh, okay. So yeah. Hinton had a had a separate institute. And you were the assistant. Oh, yeah. Research assistant back then. And directly reporting to one of his students. One of the not directly reporting. There's no like strict report line. Basically, whoever founded you. Okay. Yeah. So you're getting paid from the Vector Institute sort of like sponsorship. But anyhow, like that's basically when we were, you know, helping. We were we were studying and helping with with the establishment of Vector Institute. Okay, cool. So that's 20 2017 17 and the LMS was not a thing. LMS was not really a thing, but like neural networks was a thing. So like, you know, back in 2014 when Jeffrey Hinton and his students like the crack the image net challenge. That's 2014. That's 2014. I cannot really remember, but 2012 or 2014. That's when everyone changes their research directions from, you know, expert systems from, you know, support factor machines to neural networks. Oh, to neural networks. Yes. So in 2017, neural networks was already a trend. A large thing. Yeah. But like when we do, let's say, for example, back then there are many different branches of natural language processing. So let's say, for example, when you want to do like name entity resolution, you need to basically tag around the paragraph of things and basically mark, okay, the 27th position and the like, you know, 29th position are our name that is called Harry AI, right? In one paragraph. So you use different like models, including recurrent neural networks, which we call RNNs. And also LSTMs, you know, those things to, to, to put tags and also like you can use classifiers to classify whether a sentence is expressing like neutral sentiment or positive sentiment or negative sentiment or negative sentiment. Back then every set task of natural language processing was, was processed by a different architecture. And that's what I learned when I was an undergrad. Okay. Yeah. So your works were more mostly in that institute. When I choose my, my specialty, that was the state of the art. City. Yeah. State of the art. So that was the best solution. Okay. Yeah. That was Sota one. Yeah. And also like, um, after like, around the end of like 2017, there was a paper called the attention is all you need. Okay. You know that paper, right? Yeah. Yeah. Yeah. Yeah. Yeah. That was published by. Also like I, I happen to know Adam Gnoms, like who is right now the CEO of Cohere. Okay. Yeah. We, we attend the same course together. Like, but right now he is like the CEO of Cohere. So same age, same course. Yeah. Same age, same course. We actually, I think we entered the school around the same year. Okay. Yeah. Yeah. He was a brilliant guy. And, um, he was an intern at Google and they published like Google brain. And so that was, I think it was, it was like, yeah, around the end of 2017. And then they published that paper and everyone just like, wow, you can do machine translation in that way. And then they, they explored like sort of a broader applications of that new architecture. And then like, it seems like everything has been tackled by that architecture. And also your work, uh, pivot to. Yeah. We, we kind of changed our, our focus from, um, from how would we make LSTMs deeper by LSTMs, like two layers of LSTMs to stack transformers. And it works amazingly great that back then, I think like, uh, around the end of 2017 and early 2018, people realized, okay, this thing can really scale. You don't need to solve the kernel functions like you do with LSTMs. You don't need to figure out like how would the gradient actually not exploding or not. Like there are definitely many different sort of techniques that you, you have to closely monitor when you stack different like layers of, uh, of transformer models. Uh, for example, you need to watch the norms, you know, layer norms, batch norms, need to like, um, monitor the, seek, seek, seek connections. You need to like, um, use tensor board to, to basically closely check the, um, still check the gradients. But like, it's much easier than the, all these traditional architectures. So we just stack different, you know, transformer models for different, for different tasks. So for the first time, it seems that all the tasks, different tasks can be unified in, in, in, in one single model. Wow. So, uh, and then you continue working on that. Yeah. Yeah. University of Toronto too. Yeah. Yeah. So, and then like back in 20, the end of 2018, Google again, published a paper called BERT by directional transformers. So they use mask language modeling and also next sentence prediction to basically correct the glue benchmark. So glue is basically the generic language sort of like a benchmark, including eight different tasks, including co-reference resolution, including like name entity recognition, including, um, a sentiment analysis. And basically all the major NLP tasks are unified in one benchmark. And that benchmark was cracked by Google slapping 12 different layers of, of, of like transformer models. And that's when the paradigm of pre-training actually come into existence. Uh huh. So Google use that BERT model to pre-train on how would you fill in the, the empty masked sort of, um, uh, so for example, uh, I met with, um, um, um, water today. We were having a really happy chat, right? Uh huh. And then you, you, you basically must some part of it. Uh huh. I met with Walter today. We had a really great, whatever that thing. You ask the model to fill in that blank and the feeling that a feeling chat. Uh huh. And then like, we had a great interview. The interview goes well. So we had a great, whatever that interview goes really well. So you ask the, you basically mask around 20% of the, of the whole paragraph and you ask the model to fill in it. Uh huh. That task is considered unsupervised because you don't need a specifically dedicated thing. For example, for, for traditional, like, um, uh, you know, like sentiment analysis, you need to basically ask humans to label. Okay. I, I, I watched this movie. The movie is so, so, so. What is the sentiment? You ask human beings to rate. Okay. Okay. This is neutral or this is negative. Right. But for those kinds of like clones tasks for those like mass language modeling tasks, you don't need any human label labelers. Okay. That is where, what back then people were caught unsupervised learning the crown of the, the, the, the brightest like pearl of the crown. And also like the, the most important task because you don't need human laborers. You just ask the machine longer, machine learning model to learn whatever is there. Oh. And then you don't adapt to the downstream task. For example, you give it like, um, uh, I don't know, a few, few solving examples of like sentiment analysis. And the model itself, the bird model itself learns to give you, get back to you, whether it's positive, whether it's negative, whether it's neutral. You don't need to learn on this whole lot of different cases. You don't need a whole different new architecture. You just scale it. That's the 2018 one. That's 28 back the end of 2018, early 2019. And the, uh, the, the scientists seeing the domain start to realize, okay, there's a way to solve different tasks. With one model. Yeah. One model. And also with one unified task, with one unified reward function, not reward function, but with one unified loss function. Wow. And you were in that very early signal. Yeah. And realize, okay, this, this is something definitely we can do. And back in 2019, I was trying to adapt this thing to image. To image. Yeah. So basically you have an image, you just must some part of it, ask a model to fill in the blank. Uh-huh. Yeah. I was doing that sort of research, which kind of didn't really went through. Uh-huh. Because the compute I was having is not enough. Later on, uh, professor, um, Kai Minghe published a paper called like, Vision Transformers. Uh-huh. That was, he basically had the same idea. Uh-huh. And, uh, he patched basically the image and asked the, the, the transformer model to fill in the blank. They, they, they're doing it much clever. And they have, they, I would say, but the, but the, but the, but the sort of reasoning behind the scene and the, and the, the analysis is the same. That is, can we apply this task to also image, to also video, to also maybe voice. Yeah. People are, are starting back then starting to realize, okay, we need an unsupervised way to, to let the model adapt and learn about language, learn about the image, learn about video, learn about voice before you apply this model to downstream tasks. Okay. Yeah. That's pretty true. That's pretty training. Uh, I think that's pretty lucky. Yeah. I was pretty lucky to be around the people, around the brightest mind to, um, to do those things. Yes. Yes. You were in the, um, revolutionary century of the, this center of gravity of like machine learning research. Yeah. Back then. University of Toronto. Yeah. And, uh, under, uh, Hinton's students. Yeah. And, uh, uh, witnessed all this changing. Yeah. So, uh, 18, you pivot to the same direction. 19, you did, uh, visual. Yeah. Transformer. Yeah. And what else, what happened later? Actually like I, I, I graduated like within, I completed my coursework within three years. Mm-hmm. So, uh, I, I, I started school in 20, uh, 2015. Mm-hmm. And then in 2018, I, I, I went out to one, uh, worked, uh, uh, for, um, for, um, for Axiomobile, uh, for Axiomobile for one year as a machine learning engineer. Axiomobile. Axiomobile. So the, there's one of the largest oil and gas company. Not heard of them? Yeah. What, what, what, what, what it has to do, right? Like, yeah, that was, um, they are trying to apply different solutions to their internal documents. Mm-hmm. Because Axiomobile was a gigantic enterprise back in, back to the standard oil, right? Uh, uh, uh, uh, Carnegie sort of family, right? Yeah. So they have so many things they want to form a intelligent layer of people's knowledge. That was the early incentive of Glean of today. Glean. Yeah. Yeah. It was similar to Glean. Basically you, you want your employees to ask questions and then they don't need to dig pile of the documents and they just get an answer. Okay. That, that's all my expectations. Axiomobile. The. I know, right? You, you, you, you, you cannot imagine. People were aware, aware of the advancement of the technology. Uh-huh. They were trying different things, but it was too early. Because back then, BART model is not really good at answering questions. Uh-huh. It's really good at some of the classification tasks, the infilling tasks, but not, no, like, it's not good at like, you know, like back then when we were trying to do summarizations of a, of a paragraph, what we were doing is that we locate the play, plays of the most information dense part of things. Mm-hmm. So basically you locate the key sentences instead of generating them. Mm-hmm. Because the generations are gibberish. Okay. Okay. So our efforts of like trying to let BART models to answer questions, it's a partial success. It was usable, but it was not great. But still, how come they, so who hired you? Uh, people from Axiomobile. Yeah. But what, what's they in charge of? AI? Yeah, there's the, they have an AI department? No, they don't really have an AI department, but that was a really bright mind who is in charge of the company policy that hired me and another intern. Yeah. To do things like this. Okay. To apply BART. Very traditional industry. Yeah. And they have bright minds that are aware of the technology. Interesting, right? Yeah. And actually that is some of the scenario about happening today, right? Like people in some of the very traditional enterprises, there are people aware of the technology and they're pushing very hard to apply those things into their internal organizations. Cool. Yeah. Cool. So you did that, an earlier version of Glean for Axiomobile. Yeah. And then? I returned back to the University of Toronto to complete my master's. Okay. Yeah. So that was somewhere internal, that was a one year's full job. Um, there was eight months of intern. Eight months of intern. Yeah. Yeah. And then you choose to go back to the school for your master. Yeah. For my master. And when your master was, uh, also in computer science, also in like, uh, artificial intelligence. Okay. Yeah. So what's a project you were working on? Yeah. Yeah. I wasn't working with, uh, uh, some professor, professor and image guard. I, because again, like I'm, as I mentioned, right, I was trying to apply the bird technique to image generation and also like super resolution. And, uh, he is a domain expert in also like reinforcement learning and also image generation. So I, I, I, I, I worked on a research team with him and, uh, and then I was exposed to reinforcement learning. Mm-hmm. So I also did a little bit like multi-agent reinforcement learning. Muti. Multi-agent reinforcement learning. Back in 2019, the word Asian already exists in academia. That's something people are working on this year. I know. I know. That's. Like you have to understand the history before you like, you know, it's, it's like when you play football, if, if different football players are controlled by a different policy and you have a central policy overseeing them, you basically have a better football team. Uh, so we're researching on how to let different agents control the players of the football to, to achieve a goal. And also different like sort of characters of, of my reigns in, in Starcraft 2, to basically, uh, tackle down the, the micro, uh, the micro tasks, you know, solving, uh, Starcraft 2, solving Dota, solving, um, football simulators. That's a, that's a 20. 2019. 2019. Yeah. Multi-agent. Multi-agent. Multi-agent reinforcement learning. Uh, do you remember like actually, uh, OpenAI was back in, I think 2018 or 20, because I was a Dota player, right? OpenAI beat the worst champion of Dota, like in, back in 2018. I think that's one or two years after the alpha goal. Yeah. Yeah. Yeah. Yeah. Yeah. That was a Chinese sort of research domain. Uh, and they were not, they were by no means a revenue generating company, but like they, they, they saw the challenge. Yeah. I still remember the news when Elon and Sam, uh, founded the OpenAI. Yeah. Yeah. Yeah. Uh, and so it was a multi-agent. So the agent to this kind of definition was already there. And they were, they were there for a long time. So since what? Since I cannot really remember, but probably early 2010s, the agent was describing as a, it was a concept in reinforcement learning. Basically some, something that can execute a policy. So does that has anything to do the, with the agent today? Yeah. Yeah. I mean like if you seriously consider it. The, the, the Yao Shen, you know, one was working in OpenAI and go back to Tencent. Was, was that anything to do with him? I'm not super sure though. So the, the, the plan, uh, I think it, okay. So the plan model, it wasn't there. Yeah. The planning models and then all the things were there for a long time. Yes. For a long time. Yeah. For a long time. They're basically the same, but different. Right now there's no legit policy that needs to be learned from the environment. It's all like large language models, right? That are executing or, or, or dominating the reasoning of the models and dominating the execution. But however, the, the core concept is still the same. How can something react to the environment and execute a task? Right. Oh, that's, that's a long existing thing. Long existing thing. Yes. And, uh, in 2019, there's, uh, and, uh, use, uh, there's no large language models. There are language models though. We consider BERT large in that way already, but like BERT has, I don't know, 400 million parameters. 400 million. Okay. Yeah. So there was a language model, but, uh, not a large language model. Not a large language model. And, uh, in the language model, there was already multi-agent. No, there was, the language model is a concept in natural language processing and computational linguistics. Um, re, agent is the concept in reinforcement learning. Those two has not merged together yet. Okay. People were re-sinching into their, their own domain. Okay. So we're, you were in, uh, reinforcement learning. Yeah. I was in reinforcement learning and also had the exposure to language modeling. Okay. But our master was mostly on the reinforcement learning for the multi-agent architecture. Yeah. Yeah. Okay. And I was thinking, okay, why can't we combine those? Oh, you were, you were, okay. You were thinking. Yeah. Yeah. Why would you need an explicit or implicit, I don't know, policy that, that learn only with, with a few parameters? Why don't you use some language modeling to basically describe the environment? Uh huh. Uh huh. Uh huh. If I have enough compute, I probably can achieve something, but. Uh huh. Uh huh. I've heard of that from my other, uh, friends too. Yeah. Also in AI. And, uh, I think I talked with her many years ago that it was also already, she was already telling me that it, the industry has a lot of GPUs. Yeah. Yeah. Yeah. Yeah. In research, you have only access to, uh, maximum like four GPUs that you cannot really do anything, you know, real, realistically. Uh huh. Because Bird was trained on a massive GPU cloud. Yeah. So you need to go to industry to do that. Yeah. Yeah. Okay. So that's your master. Your master was doing, uh, multi-agent. Reinforcement learning. Reinforcement learning. Yeah. And then after that, you start to. Yeah. I worked on different, like sort of startup projects, um, collaborating with, uh, uh, a professor, a professor Andrew Skarg and also professor Jimmy Lin from, uh, the University of Waterloo. Oh, Waterloo. Yeah. Yeah. So one interesting fact, there were some early efforts on combining language modeling and also like, uh, agents. Uh huh. Back in 2020 or 2021, there was a paper called decision transformers. Decision transformers. So they use transformers to generate decisions, generate actions. Does that look similar? Does that sound similar? Right now you're using large language models to generate actions, right? Yeah. Generate what things to use, generate what are the next steps. Yeah. Back then it was only an action. Move forward, move backward. They need to define a really rigorous, you know, action space and let the transformer pick from it. Right now it's more freeform, right? Uh huh. The agent could generate the whole paragraph, the whole article to, before it move forward. Right? So it was back in 2020, 2021, our current state of how we are using agent was, was already there. And people were actually amazed by the performance because people were, they, the decision transformer is, is a really strong baseline for anyone who wants to do reinforcement learning. The same transformer? Yeah. Okay. So recently I heard of something called, um, compression is the prediction. Yeah. Compression, no compression is intelligence. Uh huh. That is basically, what is the best way to represent knowledge? Uh huh. You can definitely use earth as the center of the gravity of the solar system, but it's going to be really, really complex. You need multiple functions to, to accurately describe the, the, the movement of, of, of, let's say for example, uh, Mars. Uh huh. But however, if you put the sun at the center of the solar system, then everything is pretty smooth. So you can use, you can use 1,000 different sort of ways to describe the Newton's law. But Newton's law is actually the most straightforward and most intelligent way of expressing the, the, the rules. If for example, you can use lesser things to express border knowledge, then basically you have the intelligence behind the scene. That was proposed by Ilya, who is a Jeffrey Hinton student, obviously, and former chief scientist of, uh, OpenAI, right? I forgot the title, chief scientist or Earth CTO. So he proposed this, I think, very early, before, even before GB3. He, he basically claims that if, he basically, because many people were challenging, okay, so you're a, basically you're, you're, all your neural networks are Chinese. Right? Basically you give us something and it speed out something. And what is the meaning behind the scene? You cannot really understand it because you, you cannot really do any analysis, like meta learning or anything. You don't know what exactly is happening inside the neural networks. So what's the meaning of it? Mm hmm. That, though, that was challenged by, challenged by all the, the likes of sort of people. And, and, and Ilya basically proposed this idea, compression is intelligence to contradict them, to basically claim, okay, even though we don't know what is exactly, what exactly is happening inside the neural networks, we know that it's intelligence in a way. Mm hmm. Mm hmm. So what was a long lasting argument between symbolism versus connectionism back in even 20, back even 20, not, not 20, but like 1970s. Mm hmm. That is, how would it express the word's knowledge? Mm hmm. There are two ways. The one way is basically, Twitter is a bird, bird can fly, so Twitter can fly. You use entailment, so logic inference to represent something. You try to encode everything, everything world has to come to you into, into rules. Mm hmm. Right? You basically, when you try to speak a word, you need to use a dictionary. Mm hmm. And what are the next words that are possible? And then you get based on that. The other one is that you simulate the neural connections of human beings, of all the animals around the world. Mm hmm. And try to let it learn from the environment to have a knowledge. Mm hmm. Early pioneers of the connectionisms are Geoffrey Hinton. Mm hmm. Le Quyn, Yang Le Quyn, and also Gio Chai Banjiu. So, Yang Le Quyn was in this direction. Yeah, yeah, yeah, yeah. They were, they were numerous back then. The, the, the most renowned, um, journal in artificial, uh, conference in artificial intelligence was denying their paper because they're, they're proposing connectionism sort of approaches. But why he not criticizes a lot about this transformer mode? Because, again, he has different ideologies about how connect, connectionism should be made. Uh huh. But he is also, also still like a connectionism. So, so he is also following the idea of connectionism. Hmm. Okay. Yeah. Uh, that's where, where, where did we go? Your master? Yeah. That's which year? Yeah. Yeah. That's, that's 2019 and early COVID. Early COVID. Yeah. Okay. So what's your life journey then? Um, then, I mean, I was doing research, right? I was basically researching on how would we use reinforcement learning models into, um, into factories, controlling the factories. Controlling the factories. Yeah. Okay. So basically when you ferment, uh, penicillin, when you ferment atropine, how would you use it? How would you use reinforcement learning to do those things? I tried to apply neural networks. Drugs? Sorry. Uh, you were talking about producing drugs or? Yeah. Producing drugs, let's say for example, right? Or you can ferment beers, right? Uh, basically how to control the processes of, of, of, of, uh, maybe we can call it fermentation. That's a specific type of, uh, of industry, but we are basically trying to scale it to different, to different, to a broader domain. I was basically working on that research project with one of the early startups that's called, that's called QWERTY AI. So, um, and I was trying to apply different techniques. I was trying to apply language models, but, um, it has some, some really good outcomes. I published the first author paper in Europe, but, um, didn't, um, we applied to different, like, um, uh, factories. Did well, but it requires intensive customization. That requires you to collect the historical data of the, of the, of the, uh, of the plant, and adapt your, your, your, your, your model on into those plants. Wait, so it did well, but? It did well, but it requires customization. Customization. So local training. Okay. Yes. It requires more training before you really can rely on it reliably. Okay. Yeah. So, uh, it was in your master or after your master? In and after my masters. Okay. So I was like obsessed with this question and I basically joined them after my graduation. Okay. Yeah. So, uh, you think that is not that scalable. I thought it could be scalable, but then it turns out to be not super scalable. Okay. Yeah. And, uh, maybe that time the model was, the, the base model was not that good. Maybe, maybe right now it might be better, but like right now, even, even right now, I don't think there's a really good solution to those problems because like large tank models has a really severe latency problem. You cannot really rely on them to, to control the factory. If the, the temperature is going to rise in, in, I don't know, 10 seconds. Right. Oh, okay. Yeah. Yeah. Yeah. But anyhow, I did like have many great, great friends and like, um, learned a lot from those projects. Uh, one of my friend who is my colleague right now is a associate professor at Shanghai Jiao Hong university. Yeah. And they're brilliant people, but they're, they were tackling a problem that might not be able to solve back then. Yeah. I mean, that's the beauty of research, right? You didn't know whether it is solvable or not until you, you, you, you, you try different things. And you try to contribute the information to the world. Yeah. Yeah. You know, it didn't really, um, was the outcome that we expected. I think that journey is fruitful for me. Yeah. And even two years ago when cloud trying to use the cloud to run a vending machine. Yeah. Yeah. It was still hard. It was still hard. Yeah. We don't know this year, but. That's the sort of the limitation actually, when you talk about longer models, right? You, for example, the vending machine problem and also the, the factory problems, there, there are still guardrails you need to make and make around it. Even though for now, right? You don't want your cloud to leak your information, all things. So to put things from academia to production, there are many, many real world challenges that you need to take. ## DJI_20260626161706_0028_D Very seriously. Yeah, that's a lesson I learned. Okay, and then we come to 2022? 2022. Yeah, I joined Thomson Raiders as a research scientist, working on language modeling. Back to language modeling. Okay, I think this big company, they have connections and information. Yeah, yeah, yeah, yeah. So I did language modeling back in 2017 and now back in 2022. I never gave up language modeling, but for a time, it was not my mainstream. Okay, so for Thomson Raiders, what did you do? So I was doing training, evaluation, deployment of language models into the information retrieval systems. So they had the... What's that? What's that? So basically, they were trying to retrieve information with queries, and their systems were legacy. So basically, they use language models, the encoders, to basically vectorize documents in their system, compute the similarities between vectors, and retrieve the most relevant vectors. And basically, that represents the corresponding documents that you can see. So Thomson Raiders, they like Bloomberg. And Bloomberg. It's like Bloomberg, but they're actually... They have more departments. So basically, Thomson Raiders has Reuters news, which is like equivalent to Bloomberg, right? And they have like Thomson one, which is competing with the Bloomberg terminal. And also, they have the legal, they have the tax. They basically have more information that... They're one of the richest hub of information. And they need an efficient system to retrieve relevant information given queries. So they want to build a ChatGPT for them? Not a ChatGPT, but basically an internal search system for them. Similar thing, a chat box, I ask... Yeah, I ask it to put information back, yes. And the information is the list of different documents, or... It's different documents, it's not an answer. It's a document. Okay, cool. Yeah. So that's almost we're coming to the ChatGPT 3 moment. Yeah, ChatGPT 3 moment, ChatGPT 3.5 moment, and... ChatGPT 3 moment, ChatGPT moment happened, you know, like... The most important thing happened at the end of like 2022, right? Yeah, 2022. It was December... I think it's 22, so 24th. Yeah, yeah, yeah. It was... It was back then? Before the Christmas, I think. Yeah, yeah. Even before that, there was actually a paper called Chinchilla. Chinchilla, what's that? That's a cute animal, but anyhow. Like, the paper... The paper's name is Chinchilla. Basically, it proposed the... The scaling law. Scaling law. Again, Chinchilla was published by Google. Google... Google did a lot of work. Did a lot of work. But, uh... Google did a lot of work. Yeah. I think that's a good part. Even... I think that... If not Google, the world would not know this path is viable, right? Uh... Yeah. We have to give Google the credit. Uh... Generously publish all the things that inspire people. I think that reminds me of the Bio... Uh... Lab. Yeah. The Bio Lab, right? Who... That's a hundred years ago. They also have a lot of... Yeah, yeah, yeah, yeah. Yeah, yeah, yeah. People... People like... Sort of like not apply for a patent to... To... To certain vaccines that can cure diseases. You know, like sort of... That's the... That was the beauty of Google. Yes. Yeah. We have to give them the credit. Yeah, a lot of things. Yeah, yeah, yeah. Finally, the... Creating things... Discovering new laws... Different from the company... Yeah, yeah. Company architecture and things. Yeah, a lot of things. I think that's... It's actually a bit... Bit sad to see Google's more conservative on publications right now. Uh-huh. Yeah, yeah. I mean... But that business is business right? Yeah, it's a... It pushes the human... Yeah, it pushes the human knowledge frontier like forward. I don't think Google can actually do GPT alone. Uh-huh. Because Google were still obsessed with... With the BERT models. You know, the clones models. The few in the blank. Mass-Malinger modeling. Uh-huh. It would... It would not... Actively explore the... The other direction of... Of GPT. Which is the auto-regressive generation model. Um... If... If... If not... For the contribution of OpenAI. OpenAI also published a lot of things, right? Uh-huh. It was OpenAI. It was OpenAI. It was OpenAI back. Yeah. Yeah. But, um... Um... Basically realized scaling... Next token prediction is better than scaling Mass-Malinger modeling. Yeah. And, um... But anyhow. Basically the change-in-law paper... Talked about the... The scaling law. Actually, the more weights you have... The more parameters you have in the model... Accompanied with more... Useful like training data. Pre-training data. Your model is having a steadily improving... Performance of intelligence. Okay. Seeing that... Basically everyone tried to... You know... Scale things. So the people in the AI era... AI circle. Yeah. Yeah. All the others. In fact, you already... A hundred percent know this will work. Yeah. Yeah. And for us, we... Outside of the circle... We see the ChatGPT. In fact, I think in 2012... End of 2012, it was not even news. It was... Till the middle of 2023. Yeah. And... People were realizing... Okay, this is a thing, right? ChatGPT is there, but still a lot of dots... Like ChatGP... Because they gave a very big dream. I think they were... Yeah. Already telling about... Yeah. Yeah. People losing jobs. One-person company. And a lot of people tried... Okay, this dumb ChatGPT. Dumb ChatGPT, right? And a lot of... Overclaims, right? Yeah. That makes a lot of people... Including me... A little bit upset about that. And we don't know whether it will be... More accurate or not. Or is there any way to make it accurate? To... I think... To the... For me personally, it's... To the... To the... To the... Oh, okay. Okay, okay. The post-training sort of... The... The reasoning models, right? Yeah. Reasoning models start to... Let me think, okay... It... It is great, right? Yeah. Yeah. Yeah. In the... In the... In the... In the... In the end of the... 2022. The insider was also... Bipolar. Actually, I think more... More... Researchers were... Were... Were... Were... Not... Sort of... Very in favor of... Because... We were so close to the academia. ChatGPT was... Was... Having a similar performance... As... Ruby 3.5. Have a similar performance... Ruby 3. The benchmark was not... Really... Really... Astonishing. People were expecting this. People were saying... Okay. This is a trivial part. But... They didn't realize that... Auditor people like you... Have not... Really seen the performance... Ruby 3. Right? Have not seen the performance... Of... Of... Of the other... Like... For example... Google's... Barred or anything. Right? For all... The other models... Google T5 models. So... It was for the first time... Large longer models... Without... Other... Help... Can answer some questions. Some questions. Mm-hmm. It was never seen before... In the BERT era. Mm-hmm. Large longer models... Models are always producing gibberish. That's why we only... Put locations around the... Things that can answer questions. Mm-hmm. Right? And that's when actually... Models can really chat with you. Oh... Okay. So... Because we didn't see... We didn't see the... The... It was like... This for us. It was like... This for you. Yeah. And we've seen the scaling law. I see the... See the... Scaling law. So then... Some of us... Like for example... Also me included... Realized... How this can be way better... Just with more parameters... And with more data. Mm-hmm. So for the insiders... Uh... Ilya... Yeah. Yeah. They've seen that... I mean... Ilya produced the GP 3.5 and 3. It was such a leap phase for him. Uh-huh. Because it requires tons of money... Right? Just to train a longer model. Uh-huh. People won't throw that amount of money... Several... Several hundred millions... To train a model. Uh-huh. Uh-huh. And he took the leap phase. He proved... Actually the... The scaling law was correct. Okay, cool. Yeah. And then we have... Everything... Just happened like... In that velocity. Right? Because... Because... Because... OpenAI proved scaling law is correct. Mm-hmm. But it costs so much money... To prove scaling law is correct. Right? Yeah. It was such a... Remember... Ilya was the contributor... Was the core contributor... Of ChatGP. He was also the core contributor... Of AlexNet. Mm-hmm. Which cracked the ImageNet... ImageNet... Challenge. Which is... Which marks... Basically the... That... Um... Um... That learning-based models... Like neural networks... Uh... Are way up... Are performing the... Rule-based models... Like... Rule-based systems like... Sort of vector machines. Mm-hmm. So he is... The pivotal... The most pivotal... Sort of like... Scholar in this domain. I think. Okay. Yeah. So... Um... Thomson Reuters. Thomson Reuters. Yes. Internal... Retriving... AI. Yep. And then... And then... Basically... I was working at... Um... Thomson Reuters... Trying to scale our internal... Models... We... You know... We had... Many incentives... Training better embedding models... And... Training better... Internal... GPT sort of things... Cool. But... Yeah... We tried different things... And... The business outcome... Is great. Oh... Even then... Even then... The business outcome is great. It still helps... Helps the business... How much money do you have... To train those... Cannot really tell... But like... It's a fraction of the GPT thing... That was not open source... So... We have to... Um... Um... We have to use... Some open source... Models... Okay... And... Um... And... That was actually... Back then... People were trying to... Replicate... It was... Together AI... They tried to... Replicate... The GPT efforts... They're doing... Something called... Red Pajama... It was... Also... Like... A group of... Students from... HF... Um... Alpaca... And all the things... So... The committee was... Vivid... But... For commercialization... Organizations like us... We have to... We were... Basically working... In a constrained environment... And... But... But again... I think... That was... Actually... The booming... Another... Really... Really booming... Error... Of... Um... Of... Of... Of... The... Okay... So... After that... What happened? After that... There are many investments... Like... Happening... And... That was approached... By... 60 Degree Capital... And... They asked me to... Evaluate some... Some of their... Like... Companies... That are looking at... Into... Into... Into... Things... And... Capital that was an investment firm. In Toronto? Yes, based in Toronto. How did they fight you? I was a research scientist at Thomson Raiders, right? Yeah. So I kind of like know them from events and they're saying, okay, here is the deal. What's your view? And share with them the view. Okay, so do you mind to also, you know, do some due diligence with us? I did some due diligence with us and I learned about their working or work and I like that. So you joined him? Yeah, I did. I always wanted to do a startup. So I'm thinking, okay, this is a low risk way for me to observe the ecosystem of startups. And there are very few, they're one of the very few like Canadian funds that can invest in the United States, especially in the Sydney Valley. So I have a broader vision of like what is really happening behind the scenes. With them, I travel almost every month, sometimes twice a month to the Sydney Valley, to the center of gravity of innovations in commercialized machine learning, commercialized art and intelligence. So I like what they're doing and join the team. Okay, that's 20... Early 2024. 24? Yeah. Okay, 24. 24, what is that? It was May, it was May 2024. Yes. Okay, that's right after the GPT-03 was launched. Yes, yes. I remember that it was April 16th. Yeah, yeah, yeah. So good. How do you think of the reasoning model? It's another layer on top of like the current inference, right? Like, I mean, it's basically how would you see the growth of chips? Okay, you were hitting the limit of physics. You cannot really shrink it, but you add another layer of things on top and somehow it works, right? You're just tackling down the same problem. That is how could you answer this question more clearly and better in a different way, right? The models are not scaling that much in parameters. The models are scaling in terms of like the lens of the reasoning. That's another layer of scaling law on top of the current scaling law, which is, you know, you just scale it with more parameters and more data, right? Scale the post-training time, scale the trajectory of reasoning. Okay, so before the classic scaling law is more parameters? More parameters, more data, better model. But right now we're hitting a ceiling that is you've exhausted all the data available to human beings. And also how can you scale that? And then, yeah, and then like, OpenAI come up with a way, okay, I let the model's reasoning together in a better way, and they can produce a much better answer. That's another layer of scaling law on top of that, right? So that's an invention. That's an invention, yes. Do you think it's also very big compared to the scaling law itself? It's big, but the scaling law is the biggest because it told people that things can scale. Things can scale is non-trivial, very, very non-trivial in the history of machine learning. Because we tried to scale RNs and LCMs in so many different ways and didn't work. The reason is like, you know, the more than large, the more all the neural networks are learning with a thing invented by Jeffrey Hinton that is called backpropagation, right? You compute the gradient, you compute basically the differences between your target and your current state, and learn from the error, right? But when you model get deeper and deeper, the gradient just vanishes, or exploded along the way. There was no accurate way of passing the same information in the first layer to the top layer. It is a fundamentally very hard problem. And also when you project things, like all the information are represented in vectors. What are vectors? Vectors are things, are points, or lines, or whatever. It's a thing in the high dimensional space. And there's a curse of dimensionality. That is, mathematically, like, things are much, much easier to be perpendicular to each other in a high dimensional space. Because there are higher likelihood for you to get zeros, right? I mean, and vectors and points are more likely to go on the surface of many holes than the internal spot of it. The internal is almost always empty. So mathematically, it's very hard to scale scale the thing. But somehow, Transformer is able to do that. Okay. With, definitely, with skipped connections, with multi-hide attentions, with whatever, like, there are many, like, machine learning theory and meta learning theories behind the scene. But, um, but it's a magic that people invented, like, this kind of things. And reasoning, longer reasoning. Yeah, longer reasoning and scale things. Okay, so since 2024 to 2026 right now, it's two years. Yeah. A few months. Yeah. And you were in investing. Yeah, yeah. And that thing is also pretty interesting. Yeah, how do you like it? There's a fundamental philosophy of me is that the world hates scaling. The real world hates scaling. Like, how can you, can you scale this, like, building massively, infinitely across the sky? It's gonna break by its own weight. Can you scale the sun, the solar system? No, you cannot, because it's gonna turn into a black hole. Right? Can you scale, like, your projects, your coding, like, indefinitely? No, because it's gonna corrupt. It's gonna rot. Mm. It's, it's very, very rare for you to find a, a, a thing that can work, no matter the scale. Mm. Transformer is one, and I'm trying to, I was trying to find a one with, with capital. Mm. It's, it's fundamentally hard for you to scale the same strategy, because the more money you have, the more influence you have towards the market. Yeah. Right? You're going to provide something, whether you can call it king making, but also, like, like, Tiger Global, right? It kind of felt, felt miserably, because no matter how much money you spend in a company, if it doesn't, if it is not fundamentally good, then it's gonna die, right? Right? I mean, uh, different people, even in this industry, have different, like, sort of investment strategy. Yeah, because Tiger Global was my previous company's show, so I don't want to talk shit about them, but like, yeah. Yeah, but I think there's philosophy then was, uh, also similar, I think, because they have the deep logic theme, that is, yeah, it's really hard to go from C to A to Unicom, but something, if they pass a Unicom, the power law, maybe they will be much bigger, like, you can invest by dancing, or, uh, Facebook in a different stage. You just, like, invest in one thousand companies, and you hope that there is a Binance, or there's a... Yeah, that's their philosophy, yeah. But if you have not done it, or you're not doing it smartly, it doesn't matter, like, you just, like, waste your money on those one thousand companies, right? In a way. Yeah, maybe, but I don't speak for them. Right, because, again, like, for capital, it's really hard to scale. Otherwise, we would be filled with, I don't know, like, uh, filled with rich people back in 20, uh, 18, uh, 1810 or something, right? People, there, there's still revolutions, there's still, like, new technologies, there's still new Bill Gates, new Elon Musk, new, like, you know, some ultimate. So, the world hates scaling, but anyhow, in that way, it's just my philosophy. I like the theory, I like the theory. It's still very hard to fight that scaling one. Yeah. But because of the power law, the scaling one would bring a lot of, uh, return, and then it, um, uh, stimulates the overall world trying to fight that scaling. Yeah. Yeah. Yeah. It's good. Yeah. It's good. Yeah. Yeah. And, uh, and now, what's your plan? Now, my plan is, like, I just actually quitted my day job, like, as a full-time, uh, AI investor at 60 degree capital. I was leading the AI investment of that team. But, um, right now, I'm building my own company, trying to build the new scaling law for agents. Not really a new scaling law, but basically, we're providing an infra layer for agents to do web search. Um, okay, tell us a little bit more about that. Why? Why do you want to do that? Uh, because the, the architecture of the internet is going to change. Mm-hmm. Let's think about it. Like, when we were human beings, how we do, how we interact with the web, we use the, like, even, even before our search engine, let's say, for example, back in 1995, back in early 2000s, you have a, uh, a portal page. Yahoo, or how123, or whatever, Sohu, you know. Mm-hmm. The other page. If you want, yeah, because there was no abundance of information. Right. All the information are, are, are relatively in, in, in, in a university, or, or, or, or, or hosted by some, some rich, uh, playboy, like, you know, like, uh, building their own sort of, like, hobby websites. Mm-hmm. You don't need a search engine, you just, like, have a, a portal that lists all the possible websites that you can access, and you click into them. Right. And then, because of the later boom of the internet, later boom of the personal computers, later boom of the technology, people were seeing, oh, wow, there's so many websites that I cannot really host it on a portal page. Mm-hmm. I need a search engine to do that. Mm-hmm. I need something that is fuzzle. Mm-hmm. That is unpredictable, but it can get possibly something I want. Mm-hmm. That was built by Baidu earlier, and then Google later, right? Mm-hmm. Page rank. Mm-hmm. Page rank, yeah. Yeah. That is saying, okay, I'm going to get back to you the information according to the website, and I'm going to assess the similarity between your query and also the webpage. Right. That solves a lot of problems. It solved a lot of problems, and that was the paradigm until 2022. 2022. Okay. Yeah. What happened, Ray, it's that now we have agents. Yeah, we have agents. And it's very clear that agents are going to be the proxy to information. Right. That is, you're not going to directly see the websites. You're not going to directly keep the links. You're going to let the agents do the job. You don't need to go to Amazon to book for the right laptop. You can just give them your constraints, your preferences, your workflow. It should be purchasing you the right one. Similarly, you should not go to Booking or Wikipedia. Right. To book your travel if you would plan for 10 days, right? Like, you should just give them like your prior history, your visit, what's your preferences, and your memory, and all the things. It should be purchasing you the right one. Yes, I agree. Yeah. What that happens is that the fundamental architecture of the internet is going to revolutionize. In a way. Tell me more. It's going to be by agents for agents. There are two dimensionality of it. The first one is Google's page rank is fundamentally broke. Why? Why? Why? Because again, page rank creates the Matthew effect. What is Matthew's effect? It basically, the reachers get reach, and the poor will lose its whatever, what it has. Because it functions like, it functions like, if more people are clicking the URL of the website, Google has a higher likelihood of providing that information to everyone. Right. It concentrates the flow, concentrates the information. So maybe 10 websites dominates the majority of the internet traffic, right? Yes. But however, with agents, everyone can create something. You can create something. I can create something right now. I just like use one prompt to create my front page, right? And it's updating more and more frequently. Hmm. So, with a whole lot of information, Google was not able to provide the right information people want. Because people want something that is fundamentally related to them, but not necessarily the top 10. Right. I think Google is a statisticist. It's letting people try different things, and it gets 60% of people, what do they like? So I bring them, bring those two down. But as you said, right, agents will create more and more information. Yeah. Yeah. It's the updating frequency. Yeah. The volume. Yeah. The volume. And then the things like, the whole internet is going to be having more and more things generated by agents and generated by human beings. Hmm. It's beyond Google's. Beyond Google's scope. Beyond Google's algorithm right now. Yeah. Beyond Google's way of doing the same. Okay. Yeah. Sounds good. The other thing is that the whole internet is about ads. The whole internet is supported by ads. The whole architectures of information is supported by ads. If you think about it, Google is an ads company. Yeah. Google is an ads company. Out of like 400 billion, maybe 300 billion from ads, right? And also, Meta is an ads business. Amazon is an ads business. Mm-hmm. The shops pay to be right around top. Right? Every platform right now we've seen is an ads business. Social media is an ads business. Like even sports isn't. World Cup happening right now is an ads business. Right. Supported solely by ads. And ads are going to change because your proxy to information is going to change. Right? There are going to be agents who are doing, who are reviewing your information, who are comparing things. Why ads actually make sense back then is because with ads, there are ideas that are implanted into your mind before you realize that you need it. Right. You basically, okay, I want to purge today. What are, I probably have heard of, you know, Chanel or Prada before I even made a purchase. When I want to purchase a mouse, I probably have heard of Logic. And I've seen, you know, like so many commercials even before I made a purchase. Right? All the things that you purchase, you don't have the bandwidth to cross-compare them. You don't have the bandwidth to benchmark them as a human being. So you just choose whatever was there already in your mind. Choose whatever because you build trust on top of that. Right? And choose whatever that are available. That's why that's the core value of Amazon. Right? But in the future, if agents are going to be such adequate of intelligence and information, they should be able to do all the benchmarks with you, for you, according to your cases. That means whoever, like if you are building a, I don't know, a screen, right? And you spend like 50% of your revenue on ads, you're not going to spend as much money on improving the product. And you're going to be beat by the ones that are spending as much money on the product. Because your benchmarks are going to be lesser. Because the agents can do the finding. All the finding. All the finding. If, for example, we're providing with the agent the right way to search for information. They're able to locate, you know, the products that are having the information. The product that are showing that they're a bit better than whatever. Yeah, I think if you create that system as a seller, I would start to upload more and more of the information. There's a leap of faith though. The leap of faith is that agents know that the information is credible. If, for example, I have a website about my screen that is 50% brighter, whatever, like have better configs, but those things are not credible, then those are crappy information. What we're doing is we're building a credible source of information for agents, such that agents are able to find out those things that are not affected by the method effect. Not the concentrated information, but the things that are tailored and specifically provided for the agents. And that defeats the purpose of ads as a whole, of the current ads as a whole. Right? With that sort of abundance of information, your agents are able to locate the correct ones for you. You don't need Amazon. You don't need to pay at least those advertisements for Amazon. Yeah, I like this theory. I'm also a very anti-human internet. Yeah, right? I know, I know. I think we also, we both believe that Google, Facebook, LinkedIn, anything existing now, and big now, fundamentally, because they are not beautiful agents, they may fall. Yeah. And, okay, so tell us more. What are you are going to do and change that? And destroy that? What are you doing with those? I will tell you a bit more when we release the product. But basically, we're providing a more revolutionized way for agents to search by agents for agents. In that way, we're going to make sure, we're trying to, it's more of an ideology, right? I'm trying to make sure that information, especially the the information that was overlooked by the Matthews effect, are reachable by agents. Such that more and more people can achieve things that should be achievable, than be oppressed by the larger platforms or enterprises. That's a broader vision, but that's something that we're trying to do. I think that Google was founded in 1990 something. Yeah, 1990 something. And then the recommendation system was 2010 something. Yeah, the machine, the recommender system that was driving every core business of every large internet company was invented by 20, I think 2005, 2006. But they're all trying to solve the problem of information retrieval, though. By Facebook, right? Facebook almost. By all, by. No one. I think there are early versions of the recommender system, but you can think of it this way. Yes, sure. And they push information to you. They push information to you. Yes. And in 2010, was there anything happening? 2014, we- ## DJI_20260626164821_0029_D We can arguably say that that's the Thrive period of BuyDense. Sure, we do. Yeah, and also we have all the other new, and then we have Twitter, we have all the things basically pushing information to you. Right, and it's from words to the video, everything. Yeah, from text to video, yes. In 2020 something, when the agents become the major information consumer. Consumer, yes. Consumer and producer. There should be something. There should be something. There definitely is going to be something. And let's build a something. I know you're building this also like from a different angle. I'm building it from an infrared level. You're building it from an application level. But I think there are more and more people realize that this thing needs to be done, and maybe not by me, maybe not by you, but by someone who's going to revolutionize all those sort of like, I don't know, the Matthews effect. Yeah, I think so. I think for us, it's like before, because the consumer is human. Yeah, yeah. So human would chase, like what some Ottoman said. Yeah. What Elias said. Yeah. But today, if the consumer is agents. Yeah. So what Elias said can be heard? Whether Elias said can be heard by the people who really need our voice or not, is very, very important. Because right now, your attention is so constrained. It cannot really reliably hear the voice that you want to hear. The alphas, all the sort of like the mind that you want to hear, right? Right now, the recommendation system is still concentrated by providing with you the information that are, even though there's some effect of like trying to recommend to you something that you need, but also dominated by the influencers and dominated by that. Right, and the things people want are most popular. But I have been doing my ex growth for a while. And I mean, there's one challenge fundamentally that you cannot really solve with the current system is that, why would I even listen or see or watch someone who has only 10 followers? Are they credible? No, they're not, right? Right. And you see their profile, okay, 10 followers, you just left. Yeah. Um, but still, I think... But however, with future verifiable information, it might not be the case. Hmm. And then people don't need to struggle a lot to build audience. The idea can flow... The idea can flow freely, faster. Who only is true, has their idea flow, rather than someone who's trying to be fluffy, trying to be clickbait to you, trying to, you know, build all those malicious things around your attention. Yeah. I hate to act clickbait. Pitch your startup in three words. In one word. And tell me all those things and put your startup... Those are nonsense information. And there are going to be more of those information built by agents. Uh-huh. Right. My timeline is polluted by all those things. I know. Those pollution are something that we need to fight. If that thing cannot be fight, then we're going to be living in a really miserable world. Yes. And people are going to use agents to create those pollutions. Yes. Yeah. Agents create pollutions. Agents purified. Yeah. And there need to be a new system where people can see the information based on what they need. Yeah. What's valuable to them. Yeah. But not what other people want. Yeah. Yeah. Yeah. Yeah. Those are the most fundamental problems that need to be solved. That's huge. Yeah. Yeah. That's huge. And that's... I cannot... I don't want to live in a world that those things are not solved. Yes. There will be more noises. Yeah. Yeah. I like this moment. I think after we're recording this, maybe we'll look back 10 years later. Yeah. Let's see whether we achieve this or not. Or let's see whether someone achieved this or not. Yeah. This will be happening. And... Okay. We'll post this to the internet and it will be historic. I like it. It's cool. Thank you for your time. Thank you for having me. I like listening to all your stories. I think it's also valuable to the audience. It's just that you didn't build your page. You don't have 20,000 followers, but all those informations are valuable. Yeah. Yeah. I believe so. Okay. That's all. Yeah.