ThursdAI - The top AI news from the past week (From Weights & Biases, Join AI Evangelist Alex Volkov and a panel of experts to cover everything important that happened in the world of AI from the past week)

Welcome Friends, to the first episode of ThursdAI recap.

If you can’t come to the spaces, subscribing is the next best thing. Distilled, most important updates, every week, including testimony and tips and tricks from a panel of experts. Join our community 👇

Every week since the day GPT-4 released, we’ve been meeting in twitter spaces to talk about AI developments, and it slowly by surely created a community that’s thirsty to learn, connect and discuss information.

Getting overwhelmed with daily newsletters about tools, folks wanted someone else to do the legwork, prioritize and condense the most important information about what is shaping the future of AI, today!

Hosted by AI consultant Alex Volkov (available for hire), CEO of Targum.video, this information-packed edition covered groundbreaking new releases like GPT 4.5, Claude 2, and Stable Diffusion 1.0. We learned how Code Interpreter is pushing boundaries in computer vision, creative writing, and software development. Expert guests dove into the implications of Elon Musk's new XAI startup, the debate around Twitter's data, and pioneering techniques in prompt engineering. If you want to stay on top of the innovations shaping our AI-powered tomorrow, join Alex and the ThursdAI community.

Since the audio was recorded from a twitter space, it has quite a lot of overlaps, I think it’s due to the export, so sometimes it sounds like folks talk on top of each other, most of all me (Alex) this was not the case, will have to figure out a fix.

Topics we covered in July 13, ThursdAI

GPT 4.5/Code Interpreter:

00:02:37 - 05:55 - General availability of Chad GPT with code interpreter announced. 8k context window, faster than GPT-4.

05:56 - 08:36 - Code interpreter use cases, uploading files, executing code, skills and techniques.

08:36 - 10:11 - Uploading large files, executing code, downloading files.

Claude V2:

20:11 - 21:25 - Anthropic releases Claude V2, considered #2 after OpenAI.

21:25 - 23:31 - Claude V2 UI allows uploading files, refreshed UI.

23:31 - 24:30 - Claude V2 product experience beats GPT-3.5.

24:31 - 27:25 - Claude V2 fine-tuned on code, 100k context window, trained on longer outputs.

27:26 - 30:16 - Claude V2 good at comparing essays, creative writing.

30:17 - 32:57 - Claude V2 allows multiple file uploads to context window.

32:57 - 39:10 - Claude V2 better at languages than GPT-4.

39:10 - 40:30 - Claude V2 allows multiple file uploads to context window.

X.AI:

46:22 - 49:29 - Elon Musk announces X.AI to compete with OpenAI. Has access to Twitter data.

49:30 - 51:26 - Discussion on whether Twitter data is useful for training.

51:27 - 52:45 - Twitter data can be transformed into other forms.

52:45 - 58:32 - Twitter spaces could provide useful training data.

58:33 - 59:26 - Speculation on whether XAI will open source their models.

59:26 - 61:54 - Twitter data has some advantages over other social media data.

Stable Diffusion:

89:41 - 91:17 - Stable Diffusion releases SDXL 1.0 in discord, plans to open source it.

91:17 - 92:08 - Stable Diffusion releases Stable Doodle.

GPT Prompt Engineering:

61:54 - 64:18 - Intro to Other Side AI and prompt engineering.

64:18 - 71:50 - GPT Prompt Engineer project explained.

71:50 - 72:54 - GPT Prompt Engineer results, potential to improve prompts.

72:54 - 73:41 - Prompts may work better on same model they were generated for.

73:41 - 77:07 - GPT Prompt Engineer is open source, looking for contributions.

Related tweets shared:

https://twitter.com/altryne/status/1677951313156636672

https://twitter.com/altryne/status/1677951330462371840

@Surya - Running GPT2 inside code interpreter

tomviner - scraped all the internal knowledge about the env

Peter got all pypi packages and their description

swyx added Claude to to smol menubar (which we also discussed)

SkalskiP awesome code interpreter experiments repo

See the rest of the tweets shared and listen to the original space here:

https://spacesdashboard.com/space/1YpKkggrRgPKj/thursdai-space-code-interpreter-claude-v2-xai-sdxl-more

Full Transcript:

00:02 (Speaker A) You. First of all, welcome to Thursday. We stay up to date so you

don't have to. There's a panel of experts on top here that discuss

everything.

00:11 (Speaker A) If we've tried something, we'll talk about this. If we haven't, and

somebody in the audience tried that specific new AI stuff, feel free

to raise your hand, give us your comment. This is not the space for

long debates.

00:25 (Speaker A) We actually had a great place for that yesterday. NISten and Roy from

Pine, some other folks, we'll probably do a different one. This

should be information dense for folks and this will be recorded and

likely we posted at some point.

00:38 (Speaker A) So no debate, just let's drop an opinion and discuss the new stuff

and kind of continue. And the goal is to stay up to date so you don't

have to in the audience. And I think with that, I will say hi to Alan

Janae and we will get started.

00:58 (Speaker B) Hi everyone, I'm NISten Tahira. I worked on, well, released one of

the first Docker chat bots on the market for Dr. Gupta and scaled it,

and now we're working on getting the therapist bought out once. We

can also pass more testing and get Voice to work at a profitable

manner because we don't really have VC. So at the scale of few

hundred thousand users, the API bills matter quite a bit.

01:31 (Speaker B) So, yeah, these spaces have been pretty helpful because I have some

trouble with running a Voice transformer, trying to run it on the

browser on web GPU, and then the person that wrote Transformers JS

comes in here and just says, oh yeah, that back end is messed up.

Just try blas and synth and stuff. So these have been very

interesting and technical spaces.

01:54 (Speaker A) Yeah, we need to get Zenova in here. Zenova is the guy who NISten was

referring to. Al Janae, do you want to give a few words of intro and

say hi and then we'll start? Just briefly, please, because I think we

need to get going.

02:09 (Speaker C) Sure. Hi, I'm Janae.

02:11 (Speaker D) I'm the resident noob, I started messing around with AI at the

beginning of.

02:16 (Speaker E) The year, and I also host the.

02:18 (Speaker D) Denver AI Tinkerers coming up next week.

02:20 (Speaker A) And if you're in Colorado area, greater Denver, please join us. It's

going to be a blast.

02:27 (Speaker F) Hi, I'm Al Chang. I'm kind of an old school technologist. Just

getting started with the AI again and just here to help.

02:36 (Speaker A) Yeah. All right, folks, so I think we've had a whole space on this.

Simon Wilson and me and many, many other folks chimed in. The second

this was released.

02:50 (Speaker A) Was that six? Was that Sunday? It's hard to keep track of actual

days. Saturday, Saturday, last week, exactly during those spaces, by

the way, as we were talking, Chad GPT, Logan and everybody else from

OpenAI announced general availability of Chad GPT with code

interpreter. So GPT four with code interpreter.

03:12 (Speaker A) And I think we just heard from Matt that even some folks who got

access to the slept on it a little bit because it's maybe potentially

because of its very horrible name that's really hard to type

interpreter and get lost in the R's. But it's an extremely powerful

new superpower that we've got. And we've had the whole space talking

about use cases that people already had.

03:37 (Speaker A) It was like three days into it and since then I bet that many more

people tried it. I think Swyx 20,000 listens to that space, plus the

pod. At least people definitely want to hear new use cases, right?

03:53 (Speaker G) Yeah, not much else to add about it. I think it's the feature for

Switch.

03:59 (Speaker A) Posted a whole deep dive essay and coined it GPT 4.5 between us

friends. And one of the interesting things about it is that we think

at least that's where we are currently after playing around with

this, is that it's a fine tuned model. So they kept training this on

actually running code and executing code.

04:21 (Speaker A) That's what we believe. We don't know, nobody confirmed this and then

that it's fine tuned from an earlier checkpoint of GBT Four. And so

we actually had some folks on spaces talking about that it's less

restricted and better like previous times.

04:36 (Speaker A) So it's an interest, I think NISten right. We have some folks who

tell us they're using code interpreter without the code part. They

just stopped the GPT Four just because it's that model.

04:48 (Speaker A) And I think also they took down the 25 messages per hour restriction

on code interpreter. I've had like four hour sessions and it stopped

like I didn't saw complaints.

05:03 (Speaker G) So it's just better.

05:06 (Speaker A) It's also fast. I think it's fast because not many people maybe use

this by default and this could be the reason for the speed, but it's

definitely faster for sure. I think also context window, was it Yam?

Somebody summarized the context window and they told us the context

window for code interpreter is eight k versus the regular GPD for

actually that could be also a kick.

05:29 (Speaker G) You mean Yam copied and pasted.

05:34 (Speaker A) I would encourage you and Yam need to kiss in the cup because Yama is

doing a lot of legwork to take down the stuff that he posted and Yama

is working on that and it's very visible and you guys need to do

there you go, yam, you need to clear the air. However, Pharrell and

Gabriel bring you up as well. And we're going to keep talking about

code interpreter because that's what we're here to do. NISten and a

few other folks and we started cooking with code interpreter.

05:59 (Speaker A) And by cooking I mean we started stretching the complete boundaries

of what's possible there. And I think Simon Willison kick started

this with the latent space Pod. So for folks who are not following

latent space pod, feel free to follow SWIX, his main account, not

this hidden one.

05:59 (Speaker A) And SWIX reposted the spaces we had simon Wilson was able to run node

JS and Dino within code interpreter, even though OpenAg didn't allow

for that by uploading like a binary and asking code interpreter to

generate. Simon then promptly said they fine tuned the model away

from that and we found ways anyway to ask it to do some stuff. I have

a thread on how I was able to run a vector DB chroma inside code

interpreter.

06:10 (Speaker A) I ran whisper CPP. We saw some folks running GPT-2 inside code

interpreter, right? So imagine an Ll GPD Four running another and

talking to it. It's like a little brother inside.

06:10 (Speaker A) I personally love that inception. I don't know if the person who ran

GPD Two is in the audience as Dan I think was the nickname NISten. I

don't know.

07:22 (Speaker A) Surya.

07:23 (Speaker B) Surya. He also wrote the search to PDF plugin for GP Four plugins and

he wrote that in like two days and it's more used than any other

enterprise thing, which is pretty hilarious.

07:36 (Speaker A) We need to get surya.

07:38 (Speaker B) Yeah, he just did that as I'm just going to do a search plugins for

PDF and it's like the most used.

07:45 (Speaker A) So dope pretty amazing. Again, in that space we've talked about

having like a living manual, so to speak, for code interpreter use

cases because it's coding. So it covers pretty much everything that

we can think of as coders, maybe just in Python, maybe restricted to

an environment. And I've been trying to do that with the code

interpreter can hashtag and I encourage all of you, let me pin this

to the top of the space, to the jumbotron if you have an interesting

code interpreter thing and I'll bring up Skalsky P to the stage as

well.

08:03 (Speaker A) And Lantos, so many good friends. If you have a very interesting code

interpreter technique or skill or new thing that people can do

without coding skills, please tag with this hashtag so folks can find

this. Otherwise I will cover the main three things the code

interpreter gave us besides the new model.

08:42 (Speaker A) One of them is uploading files. And since we've talked, we've noticed

that you can upload up to 250 megabyte files and those can be zips of

other files. So we've uploaded like full models weights.

08:55 (Speaker A) We've uploaded bin files. It's incredible that you can now drag and

drop whole directory and have JPT just know about this and read about

this. We've uploaded weights in embeddings.

09:08 (Speaker A) You can then obviously execute code in a secure environment, which is

again incredible, and you can download files, you can ask it to

actually generate a download for you, which is also super, super

cool. Maybe one last thing I'll say before I'll give it to the

audience for a few more cool use cases. And folks in the stage,

please feel free to raise your hand.

09:21 (Speaker A) I'll get to you in the order that you raise your hand if you have a

use case. Some folks built like a built in memory built in brain

within code interpreter just to save to a file. That's what I try to

do with my vector DB and then they download that memory at the end of

every session and then upload this to the next one and have some like

a prompt that reminds the jgpd like to start from that point.

09:50 (Speaker A) So in addition to the context window, they're also having a separate

offloaded file persisted memory. So code interpreter incredible.

Again.

10:00 (Speaker A) Potentially GPT 4.5. And if you haven't played with this, feel free

to if you don't know what to play with, follow the code interpreter

can hashtag and let's get to Skowski.

10:11 (Speaker A) What's up, man?

10:14 (Speaker H) Hi, hello. Do you hear me?

10:15 (Speaker A) Yeah, we can hear you fine.

10:19 (Speaker H) Yeah, I've been playing a lot with code interpreter over the past

five days, mostly with computer vision use cases because that's what

I do. I haven't introduced myself. I'm pretty much doing computer

vision full time for the past five years and was focusing on like

when I saw that you can input image and video, that was immediately

what I was thinking, we need to make it to computer vision. So I went

through some low effort tasks.

10:46 (Speaker H) So I managed to run old school computer vision algorithms, face

detection, tracking of objects, stuff like that. But I also managed

to exploit it a little bit. So you can add yolo object detection

models to the list of models that were run in code interpreter.

11:15 (Speaker H) There are some problems with memory management, so I'm not yet fully

happy with the result. But yeah, I managed to run it on images and on

videos and the things that are super cool and are kind of like

underrated right now, false positive. So when the model detects

something that shouldn't be detected, you can really use text to ask

code interpreter to filter out false detections.

11:48 (Speaker H) You can just give it your feeling like why that stuff is happening or

when or where. And it's very good at cleaning the detections, which

was kind of like mind blowing for me. And one thing that I noticed

that it sucks at is I managed to create an application that counts

objects moving on the video when they cross the line.

11:55 (Speaker H) And I didn't use any off the shelf libraries, I just had detector and

say, okay, now draw a line and count objects when they cross the

line. It's terrible at that, writing math logic to figure out that

something crossed something, we had like ten prompts or twelve

prompts exchange and I basically bailed out on that, forget it. So

there are some things that blow my mind, but there are something that

probably not.

12:49 (Speaker A) So folks, feel free to follow Skowski. And also I just pin to the top

of the Tweet his brand new awesome code interpreter use cases, git

repo, and there's a list, there's a bunch of use cases there. This

could also serve as a de facto manual. So feel free to go there at

PRS and follow that for updates.

12:52 (Speaker A) And I want to get to Lentos because he seems to be unmuting. What's

up, Lentos?

13:12 (Speaker H) I was just going to say I can't follow him because he's blocked me.

13:15 (Speaker C) Sad face.

13:16 (Speaker H) Oh, no, I noticed that, but I'm not sure why. I will undo that.

13:20 (Speaker A) All right, I'm the peacemaker in the status. Please kiss and make up.

You two as well. Everybody should get along.

13:26 (Speaker A) Yay. I want to get to some other folks who came up on stage recently.

And Gabriel, welcome to talk about code interpreter and your use

cases.

13:35 (Speaker A) Jeanette, if you play with this, I would like to hear two more

opinions before we move on to the next incredible thing. Yeah. Oh,

you guys are talking about let's get together and then June sorry, I

should have been explicit about the order.

13:54 (Speaker E) No worries. So I just posted a comment on this space about the

message cap on a conversation. So even though in the UI, it still

says 25 messages per 3 hours, if you look at the network request, you

can see that. And I posted this, it's actually 100 messages per 3

hours now.

14:12 (Speaker E) And I don't know if they're scaling that up and down as demand

increases and decreases, or they're just trying to trick people into

conserving their messages, but it's definitely been on 100 for a

little while now. Can you confirm same thing you can see in the

network?

14:32 (Speaker A) Can you confirm the same for the regular mode, or do you think the

regular mode is still restricted? Well.

14:41 (Speaker E) Based on just the fact that there's only one message cap, they don't

have message cap per model. So I think it's just consistent across

all the GP four models. And that's also my experience in the last

it's been a little while now. It's probably at least a couple of

weeks that it's been higher.

14:51 (Speaker E) And same thing we discussed, I think, on Saturday about the context

window. And you can also see it in the API that the context window is

eight K for plugins and code interpreter, and it's 4K for the base

GPT four model.

15:16 (Speaker A) That's awesome. Like suicide. Better in every single way.

15:22 (Speaker D) Yeah.

15:23 (Speaker A) Awesome. Thanks.

15:24 (Speaker E) Yeah. In terms of use cases I can share, I've been digging around a

lot in the code interpreter, and I was really trying to hone in on

why are the packages that are installed there, the Python packages in

the environment? Why are they there? Some of them seem really random,

and some of them make a lot of sense. And they released it, saying

it's for, basically data analysis. And a lot of them make sense for

that, but some of them are just really wild, like the ML packages.

15:54 (Speaker A) And the Gabriel folks in the audience. If you look up at the jumbo

tone where we pin Tweets two Tweets before there's a Tweet by Peter

Zero Zero G, who actually printed all the packages and asked GPT Four

to kind of summarize what they do. So if you have no idea about the

potential capabilities of what it can do, feel free to pin that tweet

for yourself. And then it has a bunch of descriptions of what's

possible.

16:11 (Speaker A) So go ahead. Gabriel. Yeah, cool.

16:28 (Speaker E) Yeah, I've done the same kind of thing with just a short yeah, I got

it to do a four word description for each one. So if you're looking

for a really short description of each package, I'll post that tweet.

And if you're looking for a long one, I think Peters is great. And

what you can see there is that there are packages for web

development, right? There's Fast API, there's Flask, there's a bunch

of other packages for Web development.

16:40 (Speaker E) And besides the fact that there's no network access, which obviously

other people using it might be turning it on, but it was just

interesting to me. My perspective is that OpenAI has been using this

internally throughout all their teams for development and testing it

internally, but probably also using it pretty consistently. They

probably have access to the Internet.

17:14 (Speaker A) Yeah, I'm sure they have access to.

17:15 (Speaker E) The Internet and they can install new packages. But I think they also

have the ability, instead of uploading files and downloading files,

they have the ability to just mount persist memory, I don't think, to

persist. I think they just mount their local working directory on

their computer right wherever they're working. So they have their

active directory where they have their project, and they just mount

that and give the code interpreter access to the whole directory with

their whole repo of their project.

17:48 (Speaker C) Yeah.

17:48 (Speaker E) And then Chat Gvt is just writing code to the working directory and

reading from there and it can explore their whole project. We can do

that now by uploading, you can zip your whole project and upload the

whole thing zipped and have it unzipped. And then it can kind of

explore your whole project. But then once it makes some changes, you

want to commit them, you have to ask it to zip the whole thing back,

download it and upload it.

17:48 (Speaker E) And then I think what they're able to do is more of like a kind of

peer programming thing where the developer makes some changes and

then Chat GPT makes some changes and they're kind of working

together. This is taking it one step further. I don't know if they

have this or not, but it would be super.

18:29 (Speaker A) Cool in the realm of updates unless there is no speculation. But I

would love to explore this more with you in the next stage because

this applies to open source and how people already saw somebody tag

us after the last space and said, hey, I'll build this open source. I

would love to pin this to the top of the space. However, I want to

move on to new space and then move on to other updates.

18:51 (Speaker A) Sorry to interrupt, but thanks. I think that the collaborative,

persistent code superpower that probably maybe at some point will

come to us as well. Plus the internet access is like another ten x I

want to get to Skowskin and lent us and I think we'll move on to

Claude.

19:08 (Speaker A) Thanks Gabriel.

19:11 (Speaker H) Yeah, I have a question. I'm not really sure guys, if you notice that

I was obviously experimenting with PyTorch because I needed it for

computer vision. I noticed that the PyTorch version that is installed

in the environment actually pre compiled to work with CUDA. So it's a

GPU version of PyTorch.

19:31 (Speaker H) Even though that in the environment you don't have access to GPU, you

only have CPU. So I'm curious guys, what you think about that. Why is

that? Any ideas?

19:42 (Speaker A) Ideas that just come from what Gabriel just said? Likely we're

getting the same Kubernetes container. However, the open AI folks

have like unlimited stuff. They probably also have CUDA that would

make sense right there is probably connected to a GPU as well, but

that's just an idea. Lantos, I want to get to you and then we'll move

on to Claude.

20:02 (Speaker A) Folks and folks in the audience, feel free to hit the little right

button on the bottom left looks like a little message and leave

comments through commenting as well. Moving on to Claude V Two. Folks

in the audience and folks on stage, feel free to hit up the emojis

plus one.

20:19 (Speaker A) Minus one if you have tried Claude V two if you like it and you

haven't liked it. I'm going to cover this anyway because I think

somebody called me, I think Roy from Python called me a Cloud V Two

fanboy yesterday and I first got offended and I told him that I'm

just a fanboy for 24 hours. Before that I was a code interpreter

fanboy and then I figured with myself whether or not I am a fanboy of

Claude V Two.

20:43 (Speaker A) And yeah, I am and Sweet told me to relax and in fact I invited him

here to be the red blanket on the other side of the list. Anthropic

the company that we can definitely consider number two after opener.

I think that's fair in terms of quality.

21:02 (Speaker A) Have long released Claude version and they made some ways when they

released Claude AKS clong with 100K complex window, they have

released Cloud V Two and let me paste some Claude sorry, pin some

Claude thingies in the jumbotron, sorry. However, Cloud V Two

released with multiple stuff and I want to focus on two stuff and I

think we'll cover the UI first and then we're going to talk about the

model itself, UI wise and product wise. My hot take and I'll pin this

to the top.

21:38 (Speaker A) Unfortunately not debate this, but I love you, all of you. Is that as

products, Cloud V Two right now beats JPD as a product. My mom can go

into two websites and she'll prefer one versus the other one.

21:51 (Speaker A) Or my friends that don't know Xai as plugged in as we are, theirs is

free. And I think Cloud V Two beats GPD 3.5, which is also free, and

100K context window with the model being traded, 200 unleashes, a

bunch of use cases that were not possible before.

22:12 (Speaker A) It just frees you up. If you heard Skowski just say the limitations

of code interpreter. A bunch of these limitations stem from the eight

K context window.

22:13 (Speaker A) If you print a bunch within the code that you're doing, code

interpreter sometimes forgets what you guys talked about 20 minutes

ago. And the 100K context window also means a long, long conversation

history with the model. And I think it's really great.

22:37 (Speaker A) Not to mention that you can drag and drop full books in there. Those

books need to be in like one or two files and they still don't accept

zip files. And I'm planning to release an extension soon that does

this for us and unifies and single files.

22:51 (Speaker A) So hopefully by next week we'll have some updates. However, once you

upload that much or you can upload like a transcript or a podcast,

you can do a bunch of stuff because Cloud V Two is also better

trained on code and we saw a significant jump in wait, I'm switching

to the model, so let me get back to the UI. The UI allows you to

upload files.

23:09 (Speaker A) The UI has a command k interface, which I personally love. I hit

Command K in every website and see if they support it. You can just

start a new chat real quick.

23:21 (Speaker A) It doesn't have Share, but it's definitely refreshed and free UI.

It's called Cloud AI and that's the URL, and if you haven't tried it,

definitely try it. Comments about just the product side and the UI

side before we move to the model? Anybody play with this? Anybody

like it? Anybody loves the upload files feature? I would love to hear

hands and comments.

23:42 (Speaker A) Go ahead, Matt.

23:44 (Speaker D) A bit of a weird thing, but what I've noticed is it's actually quite

frustrating if you want to paste text in it actually, if it's over a

certain length, will paste in as a file. Little small thing.

Hopefully they'll change it, but it is really annoying because then

you can't edit it. Chat GP does do that much better, but I generally

agree with you that overall the product experience on Claude is.

24:03 (Speaker A) Significantly the new one. The fresh coat of paint they released for

us. I will say that Cloud so far was kind of a hidden gem, that only

folks who got access to the API actually got access to their UI, and

that UI was very restricted and folks who have access to Cloud API

know what I'm talking about. I think that UI is still around.

24:22 (Speaker A) It still shows your history. It's like very restrictive. It's not as

cool as this it's not as leak as this.

24:27 (Speaker A) So we like cloud AI, definitely a plus. Check it out. Now, let's talk

about the model behind this UI, because that model also changed and

several incredible things that changed with it.

24:38 (Speaker A) First of all, they released a new model, same price as the previous

one. We love to see this. Please everybody, including opinion,

continue giving the same price and cheaper and cheaper down the line.

24:41 (Speaker A) We love to see this. Second of all, they claim it's been fine tuned

on several things. One of them is code.

24:54 (Speaker A) And we actually saw a bump in the evaluation called Human Eval, which

is a set of questions that OpenAI released and I think the bump was

from like 55% to 78%, which I think beats 3.5 and is not there

compared to GPT four. Correct?

25:14 (Speaker C) Yeah, and four and four on past first on the first, not on GPT four

that is allowed to refine and fix it there, but on the first trial.

Yeah, by a little bit.

25:33 (Speaker A) So, news to me and thank you for joining in the past numbers is how

many times it's able to reflect upon the sensors and improve them.

25:43 (Speaker C) The past time is kind of what I meant by reflection is even stronger

GPT four. If GPT four sees the exception, it can come up with a

solution. So this is not in the Human Eval test, but if you use GPT

four this way, you get to 90 something percent, which is which I

think it's more realistic if you think about it. No programmer writes

the whole code in a one go.

26:10 (Speaker C) You write it intuitively, six bugs and so on. And also in code

interpreter, you see it. But it is remarkable to see state.

26:19 (Speaker A) Of the art on first and it's significantly better in code. And I

suggest folks who previously tried quad and haven't impressed to try

as well. An additional crazy thing that they've trained on is 100K

contacts window and they've actually trained, they claim on 200K

contact window, so twice as much as the previous round. And we follow

this one guy of your press, the guy behind Self Ask with Search and

the guy behind Alibi, the ability to extend complex windows.

26:55 (Speaker A) He just defended his PhD and he talked about complex windows and he

was impressed with the way they presented and the way they showed

their loss curve. And so this could be we saw the paper maybe this

week the folks saw the paper where the window dips in the middle.

There's like less attention in the middle of the beginning at the

end.

27:03 (Speaker A) And it looks like that's not the case for Claude as well. So I

suggest you try the huge context window and al you have your raised

hand and then we'll talk about some other model changes.

27:26 (Speaker F) Yeah, I would talk a little bit about I used Claude about a month and

a half ago to win Best Solo Hacker at the Craft Ventures hackathon

david Sachs won. Yeah, it had like 200 entries, but it's

exceptionally good at creative writing and also like comparing and

contrasting. I don't think people have really taken advantage of what

the context window is capable of doing. It's more than just loading

single files in.

27:53 (Speaker F) So what I did for the project was I loaded these large legislative

bills, these like 50 page unreadable bills, and you turned them into

relatable narratives. So one of the things that Claude can do is you

can adopt a persona. So a lot of times with summaries, summaries just

compress the text that you see, but you can tell it to say, write

1000 words from a social conservative point of view, or a bus

driver's point of view, or a social liberal point of view.

28:21 (Speaker F) And what that does is it takes all of its knowledge about the outside

world and gives you not a summary, but it gives you essentially an

essay about the practical effects of something like a bill. I've

actually been working with the idea of reading a book and having it

tell you what I would have learned from this, because that's actually

probably what you're more interested in. What it can do in terms of

comparing and contrasting large essays is exceptional.

28:51 (Speaker F) So you could have it say, write 2000 words from a social conservative

point of view, 2000 words from a social liberal point of view, and

then have it contrast the essays, which is something that would be

very difficult for a human to do. So you get to give it multiple

files and have it just give you a more balanced approach so you get

rid of some of the bias that comes in.

29:18 (Speaker A) My dream, go to my dream project that I never get to is to create

this for Twitter as like a Chrome extension that I can select a bunch

of tweets and then say, remove the bias from this and just give me

the debiased version of all of this. Yeah, completely. Like the cross

reference ability of Cloud between because of this context window is

incredible for many, many use cases.

29:41 (Speaker F) Yeah, I would say that as far it's not as good as GPT Four for

certain things. But that context window is fantastic. And I would say

a lot of people that are using embeddings and retrieval, you can

actually just put the whole thing in the context window and ask

questions to that and then you have a baseline to compare your

results from it. Most people, if they're chatting to a website or

something like that, you actually can just put the whole thing in

there as opposed to trying to chunk it up and do questions and you'll

see that your results are much better that way.

29:51 (Speaker F) And for most people, that would be good enough.

30:17 (Speaker A) So additional thing that the additional thing that Cloud was trained

on, they've talked about the output tokens, just the number. Of

output tokens of how much cloud is able to generate. And they've said

that previous models, I don't know if the same about GPT, I haven't

seen numbers on GPT Four, but they've said that previous Claude

models were focused on shorter outputs just as they were trained. And

this latest model was trained to output up to 4000 tokens in output.

30:47 (Speaker A) This is added to the fact that they also fine tuned it and trained to

output JSON files, complete JSON files as responses, which we as

engineers, we waited for this and Open Xai gave us functions via kind

of here you go, there's the function interface. And we love the

function interface. The function interface kind of locks us down to

the OpenAI ecosystem.

31:04 (Speaker A) And it's great to see another model that's like very close to state

of the art in human evil that also is now fine tuned to respond in

full intact JSONs. And those JSONs can be 4000 tokens at length. Any

thoughts on these?

31:28 (Speaker F) Yeah, I can confirm on it being able to write large amounts of

output. I mean, I was having it write like 2000, 3000 word like sort

of essays and outputs and it was fine with that.

31:40 (Speaker A) Yes. And I think it's I'm going to.

31:45 (Speaker B) Stick with GPT Four myself. But this might be pretty useful for just

dumping in an entire code base, given the 100k context window and

then getting some reviews and stuff, and then maybe moving some of

the stuff.

32:02 (Speaker A) Once I stop posting status and build that chrome extension that you

upload the zip and it flatlines it to one file and then upload it,

then we'd be able to do, like, a proper comparison, because code

interpreter can take zip files and then extract them. Oh, one

difference that I want to for folks in the audience, GPD Four with

code interpreter allows you to upload zip files, et cetera. We talked

about this. It does not load them into context window, right? So

there's like eight k context window.

32:30 (Speaker A) The files that you upload are not automatically in the context

window. The model doesn't it has to write Python code that actually

prints the files. And it usually does like the first few lines, hint,

hint.

32:30 (Speaker A) The folks in the audience who get my drift. But it doesn't usually

read all the unless you specifically ask it to and Claude does. So

everything you upload to, Claude goes directly to the immediate

working memory of the complex window.

32:38 (Speaker A) And that's a major difference to watch out for and also take care of.

Go ahead.

33:00 (Speaker C) I would like to ask everyone before I say my opinion, what do you

think about it in comparison to GPT Four about the performance? What

do you think?

33:10 (Speaker A) I would like comments from folks who actually use both and did the

comparison. And before I get to folks, please raise your hand to

answer. I want to call out SWIX's small menu bar which allows you to

actually Swyx. Can you give us like a brief two minutes on the menu

bar thing?

33:28 (Speaker G) Yeah, well, you don't have to choose. Just run it all the time on

every single chat. So it's a little electron app that runs in the

menu bar. And I've been maintaining it and I just added Cloud Two

this week.

33:42 (Speaker G) Cloud Two is not super stable yet. Sometimes it will fail to submit

the button. So you just have to retry manually to submit the button.

33:50 (Speaker G) But yeah, it's a great way to a B test models, but then also just

amplify every question with between four to five different chat

models with the answers. So I've been trying it. It's up to you if

you want.

34:07 (Speaker A) To.

34:10 (Speaker C) Find it.

34:14 (Speaker A) With the announcements, if you can. Yeah, awesome. Yeah, just

basically and maybe for instance, you don't have to stop using, you

don't have to choose. So I think the last thing that we need to

acknowledge it's, Claude, is the multilinguality.

34:28 (Speaker A) So they actually focused on showing us how much better, like, the new

ones from previous ones, and they posted blue scores, Bleu scores,

clock Two is significantly better at languages than the previous

versions. I think, to answer your question, I think it's close to GPD

Four, if not better at some things. Hebrew goes fluently, and usually

Hebrew is not that great.

34:57 (Speaker A) Russian and Ukrainian that I use also go fluently. And that part is

really good with a lot of context because you sometimes need to do a

lot of translation, or at least I need to do a lot of translation.

35:11 (Speaker C) Yeah, multilinguality works great. I was surprised. Absolutely. What

I think if you just compare the two on the same prompt, the same

question, I have a feeling that GPT Four is slightly better, but I

just don't have an example to tell you.

35:31 (Speaker C) Okay, here I don't know, it's a strange situation, but I really

wanted to ask you, like, what did you try and work better here and

there?

35:38 (Speaker A) So here's my use case that GPT Four currently cannot do. Yesterday,

Lex Friedman interviewed Israel's Prime Minister Benjamin Netanyahu

in one of the weirdest turns of history this podcast was, and given

that I know kind of who Benjamin Netanyahu is from, before I decided

to not listen to this, I decided to use the tools that we have at our

disposal. So I ran this through Whisper with Diarization. So I have,

like, a very nice transcript of who's talking.

36:10 (Speaker A) When I took that, I just dumped this as a text file. And I agree with

Matt, it's a little bit annoying that Claude turns whatever you paste

into like, a little text file uploads. That because you can't edit

it.

36:21 (Speaker A) However, I uploaded that transcript directly to Cloud, and then I

asked it to do sentiment analysis, entity extraction, and sentiment

analysis and entity extraction. Something that if I'd asked GPT code

interpreter, it would probably write some Python code to do this, and

Quad just kind of did it. And I haven't seen GPT Four being able to

do this for bigger files.

36:38 (Speaker A) And once I could just let me just this point. I continued by saying,

hey, because of the new coding abilities of Quad, I asked it like,

hey, print me a Python file that dumps whatever table of topics he

mentioned and sentiment, negative, positive, dump it into a word

cloud. That's something the code interpreters can actually do and

show you.

37:03 (Speaker A) But I asked it from Quad because previously Claude was s**t at coding

and it gave me Python files that ran from the first time. I didn't

have to change anything, there was no bugs. And then showed me a word

cloud of everything that was mentioned by BB in that podcast and it

all took like maybe seven minutes.

37:11 (Speaker A) And I don't know if for bigger complex windows, GPT Four can

currently do this. Go ahead, Al.

37:28 (Speaker F) Yeah, I've actually been putting a lot of transcripts for podcasts in

there and you can actually have the because it seems so much about

the speakers and it knows about the speakers, you can actually have

them continue a discussion about things that they didn't actually

discuss. Yeah, so it's like you can have it say, okay, well, what are

some topics they disagreed on and then some things that they didn't

cover? Tangentially, you can just have it give you another two

minutes of interview and it does a pretty reasonable job, especially

with public figures that it actually has a lot of their background

on. So it's pretty interesting.

38:01 (Speaker A) And not to mention free, ngbt Four needs a $20 a month payment and

quality is free.

38:08 (Speaker F) That's a good point, too. For those of you that have eval keys,

you'll notice that they're actually not charging you for them, so you

can actually go on as long as you want. The limitation is that you

can only do one request per organization. So if it's just a single

person, they only charge you basically when you start deploying for

commercial purposes.

38:21 (Speaker F) So that's something that people may not have realized.

38:32 (Speaker A) So I think we've covered everything right, trained on 200K context,

which they can enable tomorrow for us, and we'll get like two X. It's

going to be insane. There is some stuff that they have in Cloud in a

tropic called Constitution AI, so they have a mix of Rlhf access and

Constitution AI. So they're working on their model to actually be

more helpful, but also more safe and less jail breakable.

38:57 (Speaker A) They talked at length about this. We talked about human evil better

and same price and free playground. I think we've covered most of it.

39:03 (Speaker A) So anything else about Quad that we haven't covered, feel free to

raise your hand and tell us, and if not, I think we can move on. What

do you guys think?

39:17 (Speaker G) I'll mention briefly, did you talk about the multiple file uploads?

39:21 (Speaker A) No, go ahead.

39:24 (Speaker G) So I think it's just an interesting way difference between co

interpreter and Claude code interpreter. You can only upload one

file, right? But it can be a zip file with multiple files in Zion. So

it's de facto multiple files, but then you can only run code on that.

Whereas what Cloud here is doing is something slightly different,

which is to me is interesting, which is you can upload multiple

files, it just reads the file straight into the context and it's

using that 100K context to synthesize answers.

39:24 (Speaker G) So you can do, for example, PDF A and PDF B and give me a comparison

between the two of them or synthesize knowledge across them. And I

think that is something that code interpreter cannot do because code

interpreter will only run code across files. So I think that's

noteworthy.

40:15 (Speaker G) It's called genuinely coming up with one new thing that is not

copying chat GBT and good for them.

40:23 (Speaker A) Yeah. And unfortunately no zip allowed. But we're going to fix this

with an extension and hopefully talk about this next week. I want to

say hi to Weather Report.

40:33 (Speaker A) Feel free to chime in. Sorry you raised your hand open to come up

before. So if you have a comment about code interpreter, we've moved

past it, but if you have a comment about Claude, feel free to tell us

what's up with the report.

40:46 (Speaker A) Actually, I had only one thing about code interpreter that in the

previous space I talked about that there was a hypothesis I had about

code interpreter, which.

40:56 (Speaker B) Is to use it as a huddle because it's recorded.

40:59 (Speaker A) We'll move on and let's talk about code interpreter next time. I

think that some folks are saying that their audio is glitching and so

they're not able to and I want to see if I think Joseph has comment

about code interpreter. Joseph Polak. We'll give him a second to log

in and then I think we'll move on to other updates because we have

many other things to talk about.

41:29 (Speaker A) What's up, Joseph? Welcome to stage.

41:31 (Speaker G) Hi there, folks.

41:33 (Speaker A) Thanks for taking my question. I didn't even know all about that code

interpreter stuff with the file.

41:40 (Speaker G) So I'm really happy to have heard it. About Cloud, though.

41:46 (Speaker A) For Cloud. Well, I'm still on waitlist. First of all, it's free now.

You can access it right now.

41:53 (Speaker A) Cloud AI. There's no waitlist anymore unless you live in the States

and you'll have to get a VPN. Okay, I'll definitely check that out.

42:03 (Speaker A) My question was about using Cloud and actually code interpreter

through API. Do you think that's ever going to exist or if it's

coming so clogged API? But I think that's waitlisted. I have talked

with Claude folks and they said the waitlist is now going faster.

42:24 (Speaker A) So they are ready to get more people in. I think because of the new

safety updates, they're less afraid. So definitely apply for the

waitlist on quads account.

42:35 (Speaker A) Code interpreter is not available via API, and we've seen some folks

who hack it together with like, I think a browser plugin that proxy

something. Sweets I don't know if you remember the unofficial quote

unquote code interpreter API and it's how to access this, but it's

not available in the official OpenAI APIs as of yet. We haven't seen

them.

42:56 (Speaker G) No. For the record, there's no unofficial code interpreter API.

There's the browser side thing that we are trying to but nobody's

made any.

43:07 (Speaker D) Adapter for it yet.

43:08 (Speaker G) I think you can, if you want, using puppeteer.

43:12 (Speaker A) I would not recommend definitely, if anything, there was some folks

that tagged us and I need to go and find this that they're working on

like an open source version of code interpreter that uses laws and

stuff. And that one this will likely be the way forward. If you do

want something programmatic that has code interpret capabilities, go

ahead. NISten.

43:35 (Speaker B) There's also Chatbot UI on GitHub. So yeah, for the other people that

are hacking something together, I'll wait until there is something

public before, because then.

43:45 (Speaker D) We don't know everything.

43:47 (Speaker G) Open source is going to be worse. Because you are missing the model.

43:51 (Speaker A) Yeah, because we think that it's fine tuned on actually knowing how

to run code. Right. That's kind of the highlight that we get with

from the less space. We think it's smarter because of that.

44:01 (Speaker A) And one of the main things again, sorry, going back to code number

just real quick, it is able to then fix itself and ask itself, oh,

oops, I made a mistake. Let me try again. Matt, I saw you unmute

yourself.

44:13 (Speaker A) Feel free to go ahead.

44:16 (Speaker D) Well, yeah, just a quick thing. So from what I know, openi will be

offering fine tuning relatively soon. So at that point, you

theoretically could go and fine tune your own code interpreter like

Model, even if they don't offer it, which is going to you.

44:31 (Speaker A) Can also theoretically not that we would recommend, but theoretically

right now you could start distilling some stuff from code interpreter

by asking it questions. Generate code and store it to a file. Ask it

to download and then quote, unquote, generate the data set. But not

that you should, but you can theoretically as well, so that when it's

time to fine tune, you have some data set.

44:52 (Speaker D) Yeah, theoretically. I don't know if a shared GBT currently supports

those types of conversations, but if it does, I'm sure that's going

to happen really soon.

45:00 (Speaker G) I don't think it's maintained because chat GPT itself well, I want to

speak for share GBT. I know, Steven, but I can help you move the

conversation back to cloud.

45:11 (Speaker A) Yes, please. Let's move back to cloud. Thank you.

45:14 (Speaker G) So just between the how many people are listening to this chat

anyway? I think it's like 60 people. Email support@anthropic.com for

the Cloud API.

45:26 (Speaker A) Yes, email them, state your use case and they'll likely get you in

and you can use SWIX's menu bar to actually kind of run them in

parallel with the megaprom feature. Megapron super prompt, what is it

called? I think SWIX dropped. There is like one prompt that you type

and then it all goes to both to all the models. I want to recognize

some folks in the audience.

45:50 (Speaker A) Hey, feel free to regime if you.

45:52 (Speaker D) Want to come up.

45:52 (Speaker A) Obviously, I saw some other Euro I saw in the audience. Max AI.

Welcome, Dexter. There's a bunch of folks who are usually here and

it's great to see, and I think we're moving on to a very spicy one.

46:06 (Speaker A) What do you guys think about Xai? So I'm pasting the summary of the

people. Elon Musk and a bunch of other folks have announced X. AI

they're essentially answer to OpenAI.

46:22 (Speaker A) We've all seen Elon kind of talk about safety and talk about helping

open Xai and then could not be open since then. He talked about truth

GPT at some point. And finally they announced Xai as we were talking.

46:37 (Speaker A) By the way, I have an application from Xai which they're going to

have spaces tomorrow to go deep into deeper into Xai. But so far

there's not a lot of detail. There are some details about the folks

who work there.

46:50 (Speaker A) So they have folks who wrote the Adam Optimizer. There are other

folks thoughts about Xai before we get to hear what they do.

Obviously, there's no product yet.

46:59 (Speaker A) I don't think they've started training. The one thing that I will say

is that they will have premium access to Twitter, obviously, because

Twitter is now rebranded.com Xai. After closing down the APIs and

closing down the scraping for Twitter, xai will now have a data set

that's insane to train on Twitter.

47:21 (Speaker A) And we wish them, quote, unquote, good luck. I would love to hear

from folks on stage. What do you think about the announcement, the

direction, the people? And we're going to wait for tomorrow to

actually hear them talk.

47:24 (Speaker A) I know. NISten, you have some ideas if you want to share to get

started.

47:40 (Speaker B) Well, this is more of an old lady babushko opinion that's just

talking about stuff. I found it interesting that they went from, what

was it? Base GPT through street taking on GPT four and this entire

competition to doing something more noble like dedicating it to be

better at math and discovering new things in physics. So the way I

see that, that's pretty noble. But at the same time, I feel like

that's a result of having problems hiring in order to be competitive

with the other ones.

48:26 (Speaker B) So, yeah, this will be interesting. But the way I see the whole set

up right now is, as the kids say, it's pretty mid, in my opinion.

48:39 (Speaker A) As the kids you don't use with that. I will say that we will see

tomorrow from their space. They're probably going to use Elon's Cloud

to maybe try to hire and it's probably harder now to hire because

everybody knows how quick they're getting fired and how much. It's

not like super fun to work for X, but we're in for a nice ride

because they do have access to the cross pollination from Tesla as

well, right? So if they have big questions, tesla does have a few

good folks still, even after Andre Capati left, and so they'd be able

to ask them for assistance.

49:20 (Speaker A) There's obviously the whole Dodgy thing in play, which we can I don't

think we have time to talk about Dodgy, and it's not new, but there

could be something there. Gabriel, you wanted to come up? Maybe you

have. Yeah, go ahead.

49:33 (Speaker A) Gabriel.

49:34 (Speaker E) Yeah, I was just going to say about Xai, I mean, you mentioned

Twitter's data, and I'd be interested in hearing other people on the

stage opinion on this because recently there's been a lot of work

done on quality of data over quantity of data. And of course, Elon

also has a ton of GPUs. Reportedly, he's bought tens of thousands of

GPUs. So that's definitely important in building these big models.

49:58 (Speaker E) But I'd be interested in hearing from people on the stage if they

think Twitter's data and the kind of data that Twitter has is

actually going to be really powerful for training good models.

50:11 (Speaker A) Anybody wants to take this?

50:13 (Speaker F) Yeah, I'll take a little of it. One of the things that Twitter has

that other people don't is that people are actually debating issues.

So I think that's one of the reasons why he's really focused on the

idea of Twitter being a source of truth and being sort of

unrestricted so that you're not just following like, one thread, you

watch the narratives being debated and he has access to all that.

50:35 (Speaker A) Data and community notes. And it's really hard to scrape. Like, I

don't think it's API ball at all. It's not super simple to scrape at

all.

50:42 (Speaker A) I want to get yum before I think Matt wanted to unmute and go and

then yum. If Matt, you still want to chime in and then yum.

50:53 (Speaker D) Yeah, I mean, nothing too much to add here. I think the community

notes are very interesting as a way to sort of like, reduce

hallucinations. I think one of the things that they're going to want

to do heavily is invest in sort of filtering that data set because

there's a lot of great stuff on Twitter. There's a lot of crap on

Twitter.

51:07 (Speaker A) A lot of yeah.

51:09 (Speaker D) And the more of that that seeps in, the worse the model is going to

perform. Obviously, scale is important, but data quality is

incredibly, incredibly important and the scale kind of doesn't negate

bad data quality. So I think if they do one thing right, it's going

to have to be getting the sort of filtering of the data set down. But

they do have a ton of incredibly high quality data.

51:27 (Speaker A) Yes, I think Yam was next and then we have a few folks wanted to come

in. I think Pharrell wanted to come up. So yam. And then pharrell.

51:34 (Speaker A) And then Gabriel.

51:37 (Speaker C) I just want to say, of course, if you just take Twitter data and

start training your model, you can expect it to be average Twitter,

which is not what you want. What you can do, which is a gold mine, is

to transform this data or just rephrase it as other forms. And this

just makes the data a gold mine because Twitter does have very high

quality content here and there. Absolutely.

52:05 (Speaker C) If you can, and transform it and rephrase it to a different form if

you want an example. So the paper textbooks are all you need.

Basically, they just take data and make it into a tutorial, make it

into a textbook, like perfect, clean and everything.

52:22 (Speaker C) It is very easy to do, and you don't need a powerful model to do

that. You don't need chachi PT. You can use it to do it with a small

model.

52:30 (Speaker C) I'm currently doing off the record, I'm currently doing it myself in

a large model I'm training. It doesn't it doesn't matter matter

anyway. It's a gold mine.

52:43 (Speaker C) What I'm saying, it's a gold mine.

52:45 (Speaker D) About Twitter.

52:46 (Speaker A) An additional thing before I get to Farrell and then gabriel

additional thing. NISten I talked about yesterday at length in our

late night line cook space. That's not going to be scheduled. If you

guys are on, feel free to join that one.

53:00 (Speaker A) Twitter Spaces is also a gold mine. Transcribing Twitter spaces and

seeing all the reaction emojis that they have in real time. Like the

space that Elon ran with RFK Jr. For example, if you know in the

audience who are actual people instead of bots, and you're able to

get like emoji reactions in real time, that's a definite, definite,

very high signal kind of training set that they have and almost

nobody else has.

53:25 (Speaker A) And through how to get Pharrell, you are next, I think. And then

gabriel yeah, I wonder what.

53:30 (Speaker D) The relation is and how useful the Twitter data will be for their

goal of building a sort of math reasoning machine. Right. Also, do we

know if they're open source, as in truly open source or not?

53:49 (Speaker A) No, we don't know yet. Hopefully tomorrow we'll be able to answer

questions. However, we've seen Elon take Twitter's algorithm to open

source, and now he's like, boasting this comparatively competitive

advantage versus something like Threads. He's saying, like, hey, open

source.

54:07 (Speaker A) If you go to Threads, you're under the Zucks influence algorithm. So

there is definitely an attempt to open source from their side, but we

don't know anything about that beyond that. Gabriel.

54:17 (Speaker A) And then Johnny.

54:20 (Speaker C) Yeah.

54:22 (Speaker E) First of all, I think it's funny that Elon's s**t posting is

polluting his data set. I would say that.

54:34 (Speaker A) By the way, if there's anybody with the option to detect S**t

posting, it's them, right? They're going to be able to build a model.

Understand, this is s**t post. This is like somebody who made an

effort to give us clean information. But sorry, go ahead.

54:49 (Speaker E) Yeah, that's exactly my point that I was going to make, that Elon was

on this crusade before he bought Twitter. And this is kind of why he

got forced into buying Twitter, because he was going after the bots

and he made a big deal about the bots. And I think they spent a lot

of resources on figuring out what's good content and what's bought

content. And another thing is that we each are kind of experiencing a

different Twitter, right? Because we're within whether it's an ML

Twitter or Israel based Twitter, and there's many different

communities and their Twitter is very good at segmenting those

communities and figuring out which content belongs to what community.

54:55 (Speaker E) And they'll have the ability, I think, to segment this data and train

many different models that are good at different things because

they're in a literature community or in an ML community or MMA

community or whatever.

55:37 (Speaker A) I actually saw a map of like 5 million, 7 million tweets all embedded

in Nomic Xai Atlas. I don't know if you guys follow Nomic, they just

recently announced like a 17 million round A, by the way. So kudos to

Nomic good friends. Andre, the GPT for all team, and they have like

an embedded map before the API was shut down that they were able to

siphon, et cetera.

56:00 (Speaker A) And Gabriel, what you're saying is actually visible in the embedding

map. You can actually see those tweets and then different areas of

the political Twitter. There was a journalist Twitter until all of

the journalists started living there's like a bunch of different

pockets of Twitter that we don't get exposed to, not to mention the

different languages.

56:20 (Speaker A) There's a whole Japanese Twitter that's like insane. And people go

super, super hard. And translating is easy.

56:26 (Speaker A) We talked about Cloud being able to translate. So they have a bunch

of very interesting data. And I think Zuck is also going after that

data with Threads.

56:31 (Speaker A) And I think this is the reason why we'll see Threads getting

continued work and we'll see a lot of investment from their side. But

to compare to Threads, and we talked about this yesterday, is that

Twitter has back history and a lot of historical data that they can

train others. Threads is fairly new as well.

56:54 (Speaker A) So definitely a bunch of interesting data sets. Johnny and then

Lentil. Hey.

57:00 (Speaker H) So one I think about when I think about the data from Twitter that is

potentially lacking and some of the other data sets is colloquial

language. Because what Twitter has that Facebook doesn't have and a

lot of other things don't have, especially from what you're talking

about, like historic, is the way that people actually interact with

each other. You know what I mean?

57:26 (Speaker A) Not only that, how it evolved as well, right throughout exactly.

57:35 (Speaker H) To be honest, I think the data sets from earlier is probably better

and stronger because it's just gotten out of hand. But I agree with

what I'm not sure it was Yam or who said the filtering because all

right, this is black box, it's not open source. Elon has not been shy

about his kind of response to what he perceives as wokism and all of

that stuff. I'll be super curious.

57:36 (Speaker H) I mean, there's a big team on this, but I will be super curious to

see what that bears out in the actual model. Because, God, there's

equal parts or more parts disinformation on Twitter than there is

information. So if we're talking about source of truth, that rings

some alarm bells for me, for me personally.

58:21 (Speaker H) So those are just my thoughts.

58:29 (Speaker A) Yeah. Thanks, johnny Lentil. Go ahead. And then Gabriel.

58:33 (Speaker A) Let's finish on the Gabriel and then we'll move on to the next topic.

58:36 (Speaker H) Cool.

58:37 (Speaker A) Yes.

58:37 (Speaker H) So I think it's going to be hugely bullish for this data. And from

the perspective of relating idea space and people and the relations

between those, I think that's probably going to be more of a goat

information than conversation because you can build so much from

that. Like dating this is just one like a dating thing. Or finding

people, finding brain power compute, that's going to be huge.

58:40 (Speaker H) And to touch on the open sourceness of the data, I think not open

sourcing it at some point is going to be hugely politically bad for

Elon to do.

59:23 (Speaker A) That'S.

59:23 (Speaker H) My thoughts on that.

59:24 (Speaker A) Awesome. Thanks, Lance. Gabriel, let's end up and then, Matt, we're

going to talk about some interesting stuff.

59:31 (Speaker E) Yeah, just on the kind of data. I think for those of us who ran,

like, the early versions of Llama before they got fine tuned in all

kinds of ways, and you run it, and especially the smaller models, you

put in a prompt and it spits out some generic Facebook type of

content. It sounds like a Facebook post of like a 15 year old or

something like that. That shows what you get when you use all this

kind of unfiltered data.

59:59 (Speaker E) But I think the interesting thing is that Llama was then fine tuned

in many different ways and some really powerful models are built on

top of it. So I think in some sense, almost any data is valuable in

the sort of pretraining stages and maybe you need really high quality

for the fine tuning, but I think that big volume might be really

useful, maybe not the most economical.

60:21 (Speaker A) So I want to wrap up things why they potentially have like a leg up

versus not a leg up. We definitely know that Twitter was used to

train other models that we currently use. We know this for a fact.

This was the reason why Elon and Sam Hoffman, who used to be friends,

are no longer friends, sheet posting about them.

60:40 (Speaker A) And the current models we use. Do use this data set, but it's old for

them. It's no longer like recent and relevant.

60:40 (Speaker A) And we know for a fact that Twitter is significantly biased and

probably the best place in the world for uncovering news as they

happen before the bias sets in, before the narrative sets in, before

folks know how to before folks get their marching orders from MSNBC,

from the Other Side, how to think about things when not. The Twitter

is really good at talking about issues as they arise, the second they

arise. And I think that on its own is going to teach the models a

very great deal.

61:16 (Speaker A) Naval Ravican, if you guys follow Namal, he always said Twitter makes

him a better writer. So we definitely know also that tweet in short

form condense information better. And if their model trains on that,

obviously taking all the precautions we talked about before, bots and

s**t, posting, et cetera, if they're able to actually get this into

the model, likely their model will be more up to date and more fine

tuned like reaction.

61:20 (Speaker A) So with that, I want to close. We'll see about Xai. It's definitely

exciting, right? We're potentially getting another big one,

potentially open source one.

61:20 (Speaker A) So we'll see. I'm going to wrap up this update and I think the next

one I want to move on. Matt, let me know if you're still around if

you want to cover.

61:20 (Speaker A) So we have Matt, who introduced himself in the beginning. So I'll let

you do this quickly again because maybe and then we're going to talk

about the stuff that GitHub Stars is rising on, which I think is

super cool. And I invite you to give us a little bit of an interview

about this.

62:16 (Speaker A) Go ahead, Matt.

62:17 (Speaker D) Yeah, sure. So I'll try to summarize it a bit better than the last

time. A lot of practice, but very long story short, co founder, CEO

of Other Side AI, creator of Hyperwrite, and a number of other

things. Basically, we've been around for a number of years now.

62:30 (Speaker D) We're one of the first companies in the space working with LLMs. The

goal always has been to build a personal assistant that scales to

everybody, just like a real human personal assistant, but at scale,

way cheaper, digital. The tech wasn't there at the beginning. So we

built other products to sort of learn and gather resources, whether

that's users, revenue, bunch of other things that we can do.

62:50 (Speaker D) What we do today. Today we are actually building that personal

assistant. So an AI that can operate a computer, any software to do

what a human can do on pretty much anything.

62:53 (Speaker D) So it'll help you with your tasks. It's very simple. Today it's a

Chrome extension that lets you sort of like control Chrome just by

sort of talking to it.

62:53 (Speaker D) So you could say, go order me a pizza, or go send this person an

email or go filter my email, or anything else it works okay today.

The idea is that over time, it's going to get a lot better, a lot

cheaper, a lot faster, to the point where six months from now, a year

from now, it might actually be as good as, if not better than a human

on many tasks. But that being said, while I work on this, I also like

to learn about getting the most out of these technologies because

they're so fast moving and you really have to stay on top of it to be

effective, or you.

63:34 (Speaker A) Can every week and then stay up to date with us together. But yeah,

go ahead.

63:40 (Speaker D) Exactly. I mean, a lot of what I do to learn really, is just build

things that I find interesting, and I find that often, even if I'm

not expecting it, a lot of those learnings do translate to stuff

we're doing at other sides. So this sort of just came out of that.

Happy to sort of dive into the project, or if you want to sort.

63:56 (Speaker A) Of stop me and let's pause here for a second and I'll just tell folks

that I pinned Matt's Tweet from a couple of days ago with the

introduction. Since then you got a few thousand stars, I think, on

GitHub, and we're going to talk about the GPT Prompt Engineer project

and the different reasons why Matt and folks kind of written this and

what it's here to serve. So maybe give us an introduction to the GPD

Prompt Engineer and what kind of made you come up with this and how

it works. Yeah, go deep, man.

64:29 (Speaker A) Sure. Yeah.

64:30 (Speaker D) So forget about rambling in advance. Essentially, I find prompt

engineering so fun. I've been doing it pretty much every day for

everything, honestly, to the point of excess, from what I would do

for work to having it decide what I'm making for dinner for years

now. And as I've gone through this process, sort of like learning how

to use these models, it's become very clear that especially as these

models evolve, there's no best practice for anything.

64:54 (Speaker D) Prompts change ways to prompt change. Something that works for one

task might not work for a very similar task. And the only way sort of

get out of that is to sort of get an intuition of the model and try a

lot of things, but that doesn't always work perfectly.

65:01 (Speaker D) And also you don't really know kind of what works and what doesn't.

Even when you're trying things right, you have to do it sort of like

in a very scientific way, but there's no real right answer to

anything. It's kind of like alchemy.

65:18 (Speaker D) So starting to think I think this was right. When GPD Four came out,

I was using GPD Four pretty often to just ideate prompts. I would

say, here's what I'm trying to do.

65:20 (Speaker D) I would say, write a prompt me, and I would use the ideas from that

to help me improve my own prompts and that actually got a lot of

interest. We ended up building a sort of thing similar to that into

the hyperwrite platform. At the time it was really cool, but really

wasn't something that would replace what I do every day, which is

really hardcore prompting.

65:43 (Speaker D) Eventually I was just sort of thinking about it, and I think this was

on the 4 July, I was just sitting there kind of thinking, what if we

tried it? And I started thinking about how could you design a system

that actually comes up with good prompts? Not just a prompt that does

the job, but something that's actually optimal, because as humans,

right, we can only try so many things at once. But the magic of these

LLMs is they're creative and they think faster than we do. In the

time that I could write half a prompt, LLMs could write 5100.

65:48 (Speaker D) And what if you could leverage that? Because even if the average

prompt isn't very good, you're going to luck into one or two that

happen to be exceptional for your task. So I started by doing it

actually with a classifier. I only released this notebook yesterday

just because it's like a step on the road.

65:48 (Speaker D) And what we ended up using it for was actually something at other

side where we needed to build a classifier for something with

personal assistant. And I just wasn't getting good performance out of

the prompts that I was writing. So I said f**k it, what if we have

the AI try to do this? And I built this so that essentially I

describe the task, I give it some test cases, so I'll give it some

true false test cases.

66:11 (Speaker D) Because the classifier was classifying things as true or false. It

was like classified the statement as true or false. And it was like

New York is in America, it would be true.

66:54 (Speaker D) If it was new York is in Paris it would be false. And I basically

created like ten or 20 of these test cases. I described the task and

I had GPT generate something like, I think 20 or so prompts.

66:57 (Speaker D) And surprisingly, the quality of them just at first glance was pretty

good, right? It was kind of shocking considering I spent so much time

trying to do this manually. Then what I did was I just basically had

each of these prompts test against each of these test cases. And I

plotted sort of the success of each and turns out some of them

actually outperformed what I did.

66:57 (Speaker D) I was kind of shocked, right? Like you wouldn't expect that,

especially doing this for years.

67:30 (Speaker A) Just to recap real quick on this, the GPT four, I assume that's what

you're using generated prompts actually performed better than Match

rumors. Prompts and Matchroomr is the founder of a prompt company

with a lot of prompt use cases for a long time, from GPT-3 to four,

et cetera. And some of the ones that it came up with performed better

than yours.

67:52 (Speaker D) Yeah, it was kind of scary. Some of them performed way worse. But the

idea is that you're going to sort of luck into something that is

better. Maybe two out of 20 will be better, but they're great.

68:02 (Speaker D) So I was sort of just so fascinated by this, I was like, how do you

take this further? Because classification is one thing, but real

prompts where you're actually having it generate text, those are

harder. How do you judge that? You could use GPD four to judge them,

right? If you have two prompts and you say each of them generate me

something and they give you your responses and you want to know which

is better, you can ask GPD four. And so I figured we could apply

that.

68:29 (Speaker D) Turns out there's some issues with that and there are some papers

written about this where essentially it'll be sort of like more

favoring the one that's on the bottom. So just do it twice, flip the

order and see if one wins. And I took that approach and I sort of

combined it with sort of like an ELO style tournament where

essentially you have each of them go head to head, like one on one,

and each of them gets their ELO score either bumped up or down based

on whether they win, lose or draw.

68:53 (Speaker A) Can you give two sentences on ELO scores as a concept? Yeah.

68:57 (Speaker D) I'm actually not super familiar with it. Funny enough, I had GPC

write the code for that part, but basically think of it like a

ranking system in a video game. Yeah, think of it like a ranking

system in chess or a video game where you have two people competing

and the one that wins gets their score increased by x. The one that

loses gets their score decreased by x.

69:18 (Speaker D) And it also sort of like weighted based on the previous scores. So if

somebody that has a high score beats somebody with a very low score,

their score won't increase that much because they're very likely

going to win. So it's sort of just like a weighting system to help

figure out what's the best so instead of just sort of getting a clear

cut, yes, this is right, or no, this isn't what you can do with

classifiers, because there is a right and a wrong ground truth

answer.

69:39 (Speaker D) I just had each prompt sort of generate for a test case and the sort

of opposite prompt the competition prompt would generate for that

test case. So I was a little bit complex and they would have the

model judge which one was better. And it's expensive, right? It might

cost like $20 in GPT calls to get to an answer, but turns out at the

end, the prompts again were just kind of blowing me away.

70:04 (Speaker D) Awesome creativity in them. Like the words it used, the trigger

words, it didn't do what I would do. And in a really good way.

70:10 (Speaker D) And it also opened up my eyes to sort of like new ways of prompting

that I never would have thought of and just sort of like aren't

standard. And that's kind of the magic of all this. I think that this

sort of abstracts away the sort of atomic level of prompts, right?

You talk about prompts as sort of a prompt in and of itself and then

a system built around the prompts with many prompts kind of working

together.

70:31 (Speaker D) This makes it so that you don't have to guess about, do I have the

best prompts for this single atomic part of our system? Where the

magic really comes in then, is how do you string these amazing

individually crafted by AI prompts together to make something that

actually works really well.

70:46 (Speaker A) And how you robustly build the evaluation system, right? Because the

classifier is a simple example of evaluating, because maybe you know

this, et cetera, but how do you actually scale up the evaluation

system such that this could potentially run in loops and then

generate the best of the best prompts for a task?

71:03 (Speaker D) Exactly.

71:03 (Speaker A) That's also like a very interesting piece. How do you think about

evaluation going forward?

71:08 (Speaker D) Yeah, so I think it's sort of like that, where you could have this

thing run in the loop three times and take the three winners and then

have GPT read those winners right, and be like, here are prompts that

worked really, really well. Here are the test cases where they

failed. Now I want you to write new prompts that take what's good

about these but also mitigate the failure cases and generate a whole

new set of prompts. Sort of like evolution really doesn't just have

to stop at one point in time after the first run.

71:37 (Speaker D) It's like, let's learn from what these amazing ones still did wrong

and continue to make this better and better and better. Obviously,

this relies on a relatively large test set. I'm also experimenting

with ways where you can have the test set autogenerate, but that's a

little bit finicky.

71:50 (Speaker D) But I do think that sort of like evolution of this could lead to some

really exceptional prompts. But what I found was even on the first

run I was seeing it outperform myself. For example, there was a

classifier we were using GPT four with logic bias to do because it

was such a hard challenge and we were getting some like 90% accuracy.

71:50 (Speaker D) I had it do these prompts with GPT four, but then I had it run them

using GPT 3.5 and it got 96%.

72:19 (Speaker A) We've talked about this pattern before where you can outsource kind

of the hard work to GPD four, but then once you get really good at

prompting, GPD 3.5 is actually very decent in many things and it's

way faster, cheaper, and has a 16K context now that you can use. And

so we've seen this pattern with many folks that if you don't need the

full power of the GPT four, human evil for coding, et cetera. You can

go far into GPT 3.

5 and get very far along, especially as you're getting better

prompts. And now, Matt, you have like a recursive crafter helper guy

that's here. And my next question for you is, have you used anything

else? So you mentioned GPD 3.

5 where you run the prompts. Have you tried them on different models,

like Cloud maybe, or the open source llama ones?

73:07 (Speaker D) I actually haven't just because I wanted to see if this worked. It

was sort of just an interesting thing for me and my time is really

focused on other side and personal assistant, but it wouldn't be hard

to get Claude in. I suspect Claude prompts would perform better on

Claude. Open ad prompts would perform better on Open xai just because

the models give the prompt them very differently.

73:18 (Speaker D) Claude is sort of like a more emotional thinker. Open xai is more of

like a logical thinker. It's a very sort of simple, not perfect

analogy, but I suspect you'd want to sort of like stick within the.

73:36 (Speaker A) Ecosystems, maybe, not to mention inflections pie, which is like a

whole different beast.

73:41 (Speaker D) Yeah, that's an interesting one.

73:44 (Speaker A) We discussed by a couple of times and I've seen some reactions, but I

don't think maybe at the end of this, if we have time, matt, one

question I will have for you on this and I think we'll move on. Is

that where folks can find more work of this? Is it open source? What

are you looking for contributions? If you are. And yeah, just give us

a wrap up of this project.

74:07 (Speaker D) Yeah, so you can find it on GitHub. It's called GPT prompt engineer

Currently there are two notebooks. It's all done in Jupiter notebook

format, so it's pretty easy to edit. One is for the classification

system, the other is for the generation system.

74:20 (Speaker D) We're honestly sort of like at a point where it works well, so it's

like, what do you build around it? One thing that's missing is the

classification version only supports true and false labels, but it's

not hard to use TikTok into or TikTok and whatever it is to allow it

to support arbitrary labels like happy, sad, angry, whatever. That's

probably like a 20 minutes ad that if somebody goes in and does that

opens up a whole new set of use cases. The evolution idea that I

mentioned before, right? Taking the best prompts and then saying,

here's where it went wrong on these test cases, and then throwing it

back to GPT and having it generate more and rerunning it, that's

interesting.

74:45 (Speaker D) The ability to use Claude would be awesome if anybody wants to add

that. I could even see it evaluating each prompt on each model,

right? Because right now we only generate with GPD four. We only

evaluate with GPT 3.

75:19 (Speaker D) 5. But imagine if you generate with GPD four half of them, you

generate half of them with Claude and then you evaluate each prompt

on GPT four, GPT 3.5 and Claude.

75:27 (Speaker D) And you can see sort of the latency success rates for each along with

scores. I think all that would be super interesting. Also sort of

like just open to ideas.

75:40 (Speaker D) I'm not really sort of supporting this at all. So if anybody wants to

kind of take it and run with it, I am all for that. Also sort of just

like a shameless plug right now or thing that we're looking for just

because I have an audience here.

We are at other side in hyperwrite, really looking for somebody to

help on back end hopefully with a security set of expertise. And then

also if anybody is experienced in training machine learning models, I

would love some help there because we're doing a lot of LLM training.

75:55 (Speaker A) So just quick thing and also to add that now with the Prompt Engineer

that's automated, the results of this would likely generate like a

great data set that you can add and continue fine tuning, especially

as GPT four fine tuning is coming soon. So Matt, definitely store

everything you generate with the yellow score and everything and from

a GPT prompt engineer that runs and doesn't know about the rest run,

maybe there's going to be a path forward to actually fine tuning a

prompting model, which could be exactly. Well, yeah, exactly.

76:28 (Speaker D) Imagine taking a prompt and taking one that has a slightly higher

score and fine tuning a model to take the initial prompt and then

sort of output the one that has a higher score and you can do that

evolutionarily continue to get better prompts in theory.

76:40 (Speaker A) Awesome. So folks, if you want to work in a cool place, I can write,

hit met up and also check out GPD Prompt Engineer on GitHub. Thanks

for coming. Feel free to stay and kind of continue commenting and

talking with us as we go through a bunch of other updates that we

have.

76:57 (Speaker A) Just a quick check with NISten who promised me to follow Twitter and

see if anything new comes up. Breaking news as we talk. I haven't

seen anything besides the space of Xai.

77:04 (Speaker A) I will ask people's attention to the last pin tweet from Dr. Jim Fan

that talks about the context length dip. Matt, you also touched on

this context length dip. It's basically a paper, I think.

77:22 (Speaker A) Stanford I'm not sure that figured out. That even longer. Context

windows, they have a dip in the middle, which means that at the

beginning of the prompt at the end of the prompt, the model has more

attention to what you actually asked it to or the details that you

provide in the middle there's like a dip.

77:39 (Speaker A) And this was also released this week. However, the one thing I said

previously I will repeat here claude and some folks who know about

contact windows way more than me. They say the Claude is actually

really good at this without the dip.

77:54 (Speaker D) Yeah, I feel like that's saying. It's an interesting paper. I feel

like it's sort of saying like, hey, if you train on marketing copy,

then it's going to be worse at coding, obviously. Right.

78:03 (Speaker D) We do a lot of long context stuff at other side. That's actually what

I'm focused on right now, training really long context massive

models. And if you train it on data where there's context in the

middle that matters, it is going to be good at that.

78:16 (Speaker A) Interesting. So what you're saying, I think I've seen this kind of

opinion before as well. It's just the outcome of the data that was

fed in and for blog posts and other places, people want to hook your

attention in the beginning and then kind of finish strong. Basically

you're saying that this is potentially an outcome of that and not

necessarily the tech behind it.

78:38 (Speaker D) Yeah, I believe so. I mean, who knows, maybe wrong, but from my

experience, right, why I was given that analogy before is like if you

train it up to do one thing and then you're asking it to do another,

it's not going to do that other thing as well. And I'm guessing the

data set that they sort of did this evaluation on was something that

didn't have a ton of information at all. Part of the reason that so

few of the language model companies have super long context length

models and why it was such a big deal that Anthropic did is because a

lot of the challenge in training them isn't actually in training

them, it's in the data.

79:08 (Speaker D) Obviously, inference becomes a challenge. It's the cost and the

overhead there. But the data to sort of do this is really sparse.

79:10 (Speaker D) It's not very available. Right. So that's I think part of it right

there's not just like a sort of standard data set that has super long

context link, that has information in the middle.

79:25 (Speaker D) We do actually we've been building one another side and that's sort

of given me some of the ideas that I'm sort of spouting here. But my

guess is that Anthropic part of the reason theirs works is because

they focused on the data. The data is really important.

79:38 (Speaker A) Right.

79:39 (Speaker D) I will say model, it's just fine tuning.

79:41 (Speaker A) Yeah. I will say when I got access to Clouds Window, I did like a

bunch of tests with my Twitter data. I just pasted like a bunch of

JSON with Twitter numbers, twitter IDs numbers. And the smaller

model, the not 100K, gave me back results that actually didn't invent

those numbers.

79:57 (Speaker A) The 100K model lost in the middle and started inventing those

numbers. I literally saw this difference between the longer complex

one and the previous one and I thought it's because of like it loses

some complex in the middle. And I need to retry this on the new ones

because the new ones, they claim this doesn't happen with that.

80:01 (Speaker A) I want to go to Al and yeah, one of you I think raise your hand first

to talk about the context length dip and that paper if you have read

this, if you have thoughts and if you have noticed this as well.

80:29 (Speaker F) I just had a quick question for Matt about the differences that he

found in prompting between say, Claude and GPT Four. I noticed like,

the prompts aren't really reusable and maybe you could speak to that

in the general case.

80:42 (Speaker A) Yeah, let's end with maybe this question and move on to other updates

as we have. Go ahead, Matt.

80:48 (Speaker D) Yeah, sure. So it's like talking to two people with two different

personalities, right? They're both people, but they respond

differently to different ways. You're sort of prompting them, if you

will. Claude is sort of like more emotional, I guess, where open xai

is sort of more logical.

81:03 (Speaker D) And it's hard to sort of pin that down to any one thing, and it's

hard to give you sort of like techniques based on that because,

again, every use case is very different, but it's very clearly it's a

prompt them differently. I think also talking about the idea of fine

tuning a prompting model will be very interesting is fine tuning a

model that takes an Open Xai prompt and converts it to the idealized

version of a Claude prompt and vice versa. I mean, I think that could

be very powerful because there are ways to sort of intuit your way

there.

81:29 (Speaker D) It's just hard to sort of distill into a set of rules. One thing I

found actually quite interestingly with Quad two is that it is

insanely resistant to sort of like jailbreak attacks. So I was able

to get it to do it.

81:44 (Speaker D) Turns out the stupidest method worked. It was sort of like modifying

that dan prop that's been going around like reddit but the more

nuanced sort of like complex methods that typically work with OpenAI

they didn't. So I think the model is just qualitatively different.

81:56 (Speaker D) I think it's going to take some time to fully explore it and

understand why and how still super early days.

82:06 (Speaker A) I love the fact that all of us are getting an intuition about

different models and how to approach them right. And that's like

Sweet was here before. This is like a specialization of what I think

he talked about as an AI engineer. We're getting to start to

understand the differences between those to the little fine little

things that you can say.

82:11 (Speaker A) And I think it will be very interesting if you have a model that's

trained to actually convert them or translate them between the models

to work the same. I have an idea where not to get locked into the GPD

Four ecosystem with the functions. I have an idea of wrapping the GPD

Four API package with something.

82:47 (Speaker A) They will actually kind of print the functions into the context

because cloud now has a huge context window. And then try to see

whether or not cloud is able to kind of without additional tech,

without additional changes to the API to replicate the outputs of how

a GPT with functions would do. And that's going to be an idea I'll be

testing, hopefully, and talk about next week.

83:08 (Speaker A) Thanks, Matt.

83:10 (Speaker C) Today, there has been a thing today, maybe yesterday, but anyway,

today there have been a model that generates prompts. By the way, by

giving the data, you generate the prompt. I've written about it today

on Twitter. It is so powerful, it is such a cool method that you can

take whatever you have, like, I don't know, scientific papers and

generate instructions for them.

83:32 (Speaker C) Now you can fine tune a model that generate scientific papers. You

got jokes. Now you can train a model that become funny.

83:35 (Speaker C) You can generate the instruction, convert whatever you want into

instructions. Amazing it is today. One more thing about the deep in

the middle thing.

83:51 (Speaker C) I don't know why it happens. I have no idea how Open Xai trained

their models. But I think if you think about it, many missions, many

instructions, paragraph, and before the paragraph, you tell the

model, please summarize the following, or on the contrary, like a

paragraph and at the end, what was that? Something.

84:10 (Speaker C) So it makes a lot of sense that a model pays a lot of attention to

the beginning at the end, because of this. And on the same note, it's

very easy to fix. So I wouldn't just point fingers.

84:21 (Speaker C) It's good that they pointed it, but I think it's like, I don't know,

a couple of minutes of training, open AI, like, fine tune for a

minute and fix it.

84:28 (Speaker A) I just want to ask yum, yum. The the pin that I just tweet sorry, the

Tweet that I just pinned on top, this was the one that you talked

about, the instructions generation and the problem generation.

84:38 (Speaker C) Yeah.

84:39 (Speaker A) Awesome. So folks, definitely feel free to check this out. I haven't

seen this. You want to give a couple more words about that one.

84:44 (Speaker A) It looks like you wrote, like, a very deep dive. What's the model

like eleven B, three B?

84:54 (Speaker C) Sure. Two models put into the models, whatever you want. Okay, let's

go back. You got a data set of something, emails from your company,

for example, and you want a model that will help you write emails.

85:01 (Speaker C) Okay, you can start thinking about how to train this model, or you

can use this and now generate a text that basically says, help me

write the following email to this following person of something

something and the actual email. And all of a sudden, you have a model

that is extremely you have a data set to train a model or to fuselage

or whatever that is extremely tuned to this. So I think it's a very

cool technique.

85:40 (Speaker C) It's very powerful, has a lot of potential. And the trick, in simple

words, is training the model. What not to say? That's the missing

piece here, that they added the trick.

85:51 (Speaker C) They took instructions and outputs that do not fit just a different

random output from the data and train with a different laws. That the

model should not say this because this input does not with that

instruction, does not result in this output. That's it.

86:11 (Speaker C) That's the trick. And it works perfectly and really cool.

86:17 (Speaker A) Awesome. I have some folks who want to come up and ask questions. I

think we're almost there in terms of the updates. I will just briefly

run to some updates.

86:18 (Speaker A) I don't even have time to go and look for the threads, but if you're

not following Rama CPP, follow gerga is one of the groups that we

have in the States. I think he single handedly is in charge of so

many folks trying to get a MacBook, because it's incredible how much

performance they've been able to squeeze out of Llama. And it's

comparatives.

86:49 (Speaker A) And many people just, like, quantize their models, basically make

them smaller to run on this GGML platform that they have. The recent

news that I have from over there, there's like two pieces of news.

Last week, for those of us who were here last week, we talked about

CFG.

86:58 (Speaker A) I forgot something. I forgot the guidance scale. And we talked about

the CFG parameter moving from diffusion models that we know.

87:17 (Speaker A) Like, in stable diffusion, you can define how close to your prompt

should the model generate the image. Somebody decided, I think, an

illusion reaction. Somebody said, hey, can we have this control of

CFG to our LLM generation? CFG is a classifying guidance scale,

something like that.

87:37 (Speaker A) And they did it. The Chad GGR added this to Llama CPP. And so now you

can actually kind of pass a CFG control and fine tune.

87:48 (Speaker A) It's almost like a running fine tune to an extent. You can test the

model to be closer, farther away from the problem that you have.

Contrasting this with the stuff that we have on a GPD, four API,

which is temperature.

88:01 (Speaker A) And I think, Matt, you mentioned something to logic bias, logged

bias, something like that, right? Where you can ask it not to say

certain things. So contrasting CFG, it's like a different beast that

we now have a different control. And so GGML just merged into their

platform.

88:18 (Speaker A) Definitely worth checking out. And the second thing is, I need to

find the Tweet. Yesterday, Georgia was like, oh, yeah, by the way,

here's the 48% inference speed improval that somebody just merged in.

88:30 (Speaker A) Have you guys play and try this. For the 33 billion parameter model

of Llama, somebody just merged in a 50% increase on inference speed

just on the way. And I find this incredible because Gmail already

runs many stuff on Raspberry Pi or whatever, iPhones, and now

somebody's like, oh, yeah, here's a 50% increase in infinite speed.

88:41 (Speaker A) And then I think Nissan was here before he was talking about GGML

runs on the iPhone, because iPhones, even from three years ago, have

the same neuron chip that like the latest Max or some such, and that

this performance boost on GGML also applies to iPhones as well. So,

incredible stuff. And as we hear every week, we keep seeing leaps,

incredible leaps in speed and performance.

89:15 (Speaker A) Definitely worth checking out GGML and the five folks that work on

those stuff. GML comments, folks who use Llama, CCP, feel free to hop

up and raise your hand and give us more updates from that length. I

denied it.

89:28 (Speaker A) You are gay at the spaces, but sometimes as a guest as well. Other

than that, I think we'll move on to some more updates and then we

just have questions. No? Cool.

89:41 (Speaker A) So the next update that I have is from the diffusion side that we

sometimes cover. We don't cover it often, but we do cover it from

sometimes time to time. So two things from stability stable

diffusion.

89:46 (Speaker A) We talked about Sdxl, the new Excel model that can generate 1024

images. We've talked about last week about the 0.9 weights dropping.

90:01 (Speaker A) Sdxl 1.0 is now available in the Stable Diffusion discord. If you've

played with Me Journey before and you looked at Stable Diffusion,

it's like, it's not that great.

90:05 (Speaker A) Stable diffusion sdxl one is really impressive. And besides being

really impressive, they plan to release this open source. So we're

going to see a bunch of folks fine tune loras and specific versions

of the specific things.

90:16 (Speaker A) And I think it's like, incredible. If you want to play with those

models and you haven't yet, go to Stable Diffusion discord and hit up

that bot and then Netflix let us know how incredibly different that

is. And we're waiting for the wait for the Sdxl 1.

90:47 (Speaker A) 0 to drop. And I will mention this every day until the year mark.

It's been less than a year since table Diffusion.

90:57 (Speaker A) It's been less than a year. I remember I think it was August 22 when

they actually dropped the full open source model. Less than a year.

91:12 (Speaker A) And we've seen just such incredible progress. So, like Matt said

before, it's really hard to keep up, but it's also really hard to

internalize how far, just how far we're coming with those incredible

leaps and changes every week. And again, to just plug in this

Thursday I space.

91:21 (Speaker A) This is why we're here. Every thursdai talking about everything and

everything that's changed and updated. And the other thing that I

want to I see art in the audience with apart.

91:28 (Speaker A) If you play the list, the Excel, feel free to raise your hand to come

up. The other thing that they released, I don't know if you guys

familiar with Clip Drop. So Stable Diffusion bought Clip Drop as a

company and started implementing that interface compared to their

Dream Studio interface.

91:49 (Speaker A) So ClipDrop is like a way simpler interface day to day release,

something called Stable Doodle. Stable Doodle is I don't know if

folks in the audience remember this. Meme how to draw an owl.

91:51 (Speaker A) Step one, draw a circle. Step two, draw some eyes. And step three is

like, draw the rest of the f*****g owl.

92:06 (Speaker A) And then you have, like, a beautiful owl painting at the end of this.

This is now the go to test on how the Doodle models work. And I

pinned my attempt at this, but definitely check out ClipDrop Doodle

thing.

It's really fun to play with. So those are, like, the updates from

the diffusion world.

92:10 (Speaker D) Hey, real quick. I was just looking at the repository for Comfy UI,

and then I saw that I don't know how to say his name. Scousekip is in

here. So I just wanted to come on and say, like, hey, this is

incredible.

92:24 (Speaker D) This is what we've been talking about for months now, right? This

node based character codex, if you will, of like there's just

infinite possibilities. I just want to listen, but thanks.

92:35 (Speaker A) For bringing me up.

92:36 (Speaker D) This is really cool, man. I was just thanks for bringing up Comfy UI.

92:42 (Speaker A) I feel guilt at not being up to date on every single possible thing.

I know it's impossible. I really try, and Comfy I has been on my list

to try, but then Quad was released and Code Interpreter was released.

Comfy I seems like the thing we want, man.

92:42 (Speaker A) I think stabilization when they tried to bring up Dream Studio, they

talked about, like, a node based thing where you can pipe models to

other models, you can find filters, et cetera. Comfy UI for folks who

have tested it out, it looks like that's it. And I definitely want to

agree with Art.

93:16 (Speaker A) It's something to watch out and maybe try because automatic one on

one, even though it's, like, super advanced and has been there for a

beginning since Stable Diffusion, it's just like a s**t show of a UX.

Just like horrible, horrible. I'm sorry, guys.

93:30 (Speaker A) I've built a web UI before automatic. It's really hard to get Gradio

to play as much as you want. It's really hard to maintain a good UX

product with many, many people contributing, with many, many things

are changing under your feet.

93:45 (Speaker A) So it's really not their fault, but it's a s**t show to get started

with. And Comfy UI seems like a fresh, clean start. So definitely if

you're playing with this, test this out and let us know.

93:55 (Speaker A) Max, you have your hand raised and you play with the Excel. Give us

some of your thoughts.

94:01 (Speaker I) Yeah, I have played through the website in a studio, so I'm lately

working with a company that make toys for kids. They want to start

incorporating AI. And one of my concerns we're working with them is

like, okay, we want to generate images for kids. Something that is

going to probably freak them out is two things that diffusion models

have been lacking.

94:27 (Speaker I) One is the ability of painting things like complicated shapes or

intricate shapes like hands. SD. Excel is not better at it.

94:40 (Speaker I) Another one is this concept of what is named like concept bleeding,

which is this diffusion model tends to mix objects that are similar

in shape or form is not good at it, neither. Now, I was reading the

paper from Stability or the report. They claim they are outperforming

Mid Journey in five of seven categories now, mid Journey 5.

1, right?

95:12 (Speaker A) Just to make sure. Mid Journey since then released the new version

also because we're in same pace, but yeah, they've compared to Mid

Journey 5.1. Yeah.

95:20 (Speaker I) Well, now this is a report internal released by Stability. It's a

paper, it might have some credibility, I don't know. I like the

results. It's very close to me journey, but I think there is still

one or two steps behind, in my opinion.

95:36 (Speaker I) What is different is what you have mentioned, Alex. Once they release

the weight and we can see Lotus about this, I'm expecting to see the

results that we can get because probably that is what is going to

position this model like a step above Mid Journey, but not yet. This

is my opinion.

95:58 (Speaker A) Yeah, definitely. And thanks for that. And I love folks coming up and

sharing their opinion about these things. I will say on the top.

96:05 (Speaker A) Thanks Mike. Or I guess I know you're a new name, but I'm not sure if

I can if I should.

96:10 (Speaker I) Yeah, totally, totally have it, in my view. I'm Juan Spanish, living

in Mexico and I like these things.

96:17 (Speaker A) We appreciate you coming up here on the topic of UIs that we've

mentioned with somebody or somebody folks released Pinocchio. They

call this the AI browser. And I want to highlight this because I want

to give you practical tips. Janae, I think, is coming in with some

breaking news.

96:28 (Speaker A) I don't know if Janae wants to come up or can, but if you can, feel

free to come up and tell us there's some news from Bard. Until we

talk about Bard, the topic of UIs for those things, and you guys know

we're mostly focused on the LLM side and the Engineer side. Less than

there's a fusion, but we sometimes have love for both the above tool

that you can download and not deal with the terminal, not deal with

the bunch of stuff, unifies all of them.

97:08 (Speaker A) It's really nice. Check out the Nokio AI browser. I think it's open

source.

97:12 (Speaker A) You download this once, it's cross platform, Mac, PC, et cetera, and

then you're able to download Llama CPP, and then you're able to also

download table diffusion. And then fairly quickly, without knowing

how to code, without going through the terminal, without installing

packages, folks here know that installing the packages is like a

whole pain we all share and we all hate without doing all of that.

That's the promise that they have, you are able to pipe Llama outputs

into stable diffusion.

97:38 (Speaker A) So Yam previously mentioned kind of the model that can do, and Yam

and Method are talking about a method of generating prompts for LLMs,

but also we know that there's models prompts to actually generate

prompts for diffusions and they're trained on different and fine

tuned on different ways to generate diffusion prompts. Right, and

this Pinocchio browser is actually allowing you to run like an and

then pipe the output into stabilization model and then see the output

of that. I think it's incredible that this exists and is

downloadable.

98:07 (Speaker A) I haven't tried this yet. If you in the audience or somebody on stage

have tried Pinocchio, please raise your hand. I want to bring you up

and talk about Pinocchio and your experience with this.

98:19 (Speaker A) And if we haven't, I want to bring this to our attention so that next

week we're able to talk about this. This is added to my list of

things I like. The Comfy UI that I haven't tried it yet.

98:29 (Speaker A) Anybody use pinocchio yet? No? Cool. I wanted to get Cocktail Peanut.

The guy who wrote Cocktail Peanut.

98:36 (Speaker A) If you're in the audience, feel free to raise your hand. I don't

think you are, but feel free to follow the thread. He goes fairly

deep.

98:44 (Speaker A) And feel free to use and try Pinocchio by next week and then come up

next week and talk about the differences between this and running

automatic one one. All right, folks, thanks everyone for coming to

another Thursday. I space.

98:58 (Speaker A) Hope this has been helpful for a bunch of you. We tried a few new

things here. We tried to give updates, but also deep dive into a

conversation with Matt and looks from the reactions here that maybe

this is worth putting down on paper and sending out an email for

those of you who want to maybe sign up for this and not don't have

the time to listen to two hour spaces, so I'll definitely try at

least to do that.

99:19 (Speaker A) I want to thank a few folks on stage that have joined consistently

and providing a lot of signal yum follow Yam. He has great insights

into models and training and different things al in the audience.

Thanks always for coming up.

99:33 (Speaker A) Junaid is running the Denver meetup, and if you're in the Denver

area, feel free to join us next week. Thanks for coming. Haven't seen

you in a while, buddy.

99:45 (Speaker A) Juan sorry. Yeah, I think Juan great. Maxi and Lentos has recently

been joining us.

99:51 (Speaker A) It's been great. We have some more folks in the Evans who are

regulars, and we invite you to also be regulars and come up and talk

about Thursday. I will say this one thing, tag me in anything that's

new.

100:01 (Speaker A) I would love that. And help promote the message for other folks. If

you did like the space, this also really helps for more folks to get

to the bottom of this for those folks.

100:01 (Speaker A) I didn't get to their questions. I apologize. I'm trying to keep this

as a balance of a high signal thing versus letting everybody

questions as well.

100:22 (Speaker A) Last thing I'll say is about myself, a little bit consultant. I stay

up to date so you don't have to. That's my tagline.

100:29 (Speaker A) If you're in the company and needs consultancy for somebody who's up

to date on everything, I try to be that guy. Feel free to tap me in

the DMs. And, yeah, thursdai folks, keep tagging us everything that's

new. We're going to try to cover next week with that.

100:34 (Speaker A) I thank all of you. Thanks for coming. Thanks for giving us two and a

half hours of your attention.

100:34 (Speaker A) I really appreciate it. Attention is sparse and very important, and I

really thank everybody who gave us, like, two and a half hours. Thank

you, folks.

101:00 (Speaker A) Hey, Alex, we really appreciate you.

101:04 (Speaker B) Thanks, Alex.

101:05 (Speaker H) Thanks for doing a good space and keeping us on track, actually.

101:09 (Speaker A) Yeah, thank you.

101:10 (Speaker D) Yeah, alex definitely want to kind of.

101:13 (Speaker A) Give our thanks to you as well.

101:15 (Speaker E) For curating an awesome space.

101:17 (Speaker D) I think I'm definitely not the only one that gets a lot of good

signal out of this. And I know a lot of hard work goes into keeping

yourself up to.

101:27 (Speaker A) Date so that you can share it.

101:28 (Speaker E) With all of us.

101:29 (Speaker D) So just on my own behalf, thank you. And I'm sure that is echoed by.

101:34 (Speaker E) A lot of people on stage and in the audience.

101:36 (Speaker A) Humble man thank you. I appreciate you. Thank you, folks. Have a nice

Thursday and bye next week.

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

If you’d like to hear the whole 2 hour conversation, here’s the link to twitter spaces we had. And if you’d like to add us to your favorite podcatcher - here’s the RSS link while we’re pending approval from Apple/Spotify

Happy LLaMa day! Meta open sourced LLaMa v2 with a fully commercial license.

LLaMa 1 was considered the best open source LLM, this one can be used for commercial purposes, unless you have more than 700MM monthly active users (no 🦙 for you Google!)

Meta has released the code and weights, and this time around, also a fine-tuned chat version of LLaMa v2 to all, and has put them on HuggingFace.

There are already (3 days later) at least 2 models that have fine-tuned LLaMa2 that we know of:

* @nousresearch have released Redmond Puffin 13B

* @EnricoShippole with collaboration with Nous have released LLongMa, which extends the context window for LLaMa to 8K (and is training a 16K context window LLaMa)

* I also invited and had the privilege to interview the folks from @nousresearch group (@karan4d, @teknium1 @Dogesator ) and @EnricoShippole which will be published as a separate episode.

Many places already let you play with LLaMa2 for free:

* https://www.llama2.ai/

* HuggingFace chat

* Perplexity LLaMa chat

* nat.dev, replicate and a bunch more!

The one caveat, the new LLaMa is not that great with code (like at all!) but expect this to change soon!

We all just went multi-modal! Bing just got eyes!

I’ve been waiting for this moment, and it’s finally here. We all, have access to the best vision + text model, the GPT-4 vision model, via bing! (and also bard, but… we’ll talk about it)

Bing chat (which runs GPT-4) has now released an option to upload (or take) a picture, and add a text prompt, and the model that responds understands both! It’s not OCR, it’s an actual vision + text model, and the results are very impressive!

I’ve personally took a snap of a food-truck side, and asked Bing to tell me what they offer, it found the name of the truck, searched it online, found the menu and printed out the menu options for me!

Google’s Bard also introduced their google lens integration, and many folks tried uploading a screenshot and asking it for code in react to create that UI, and well… it wasn’t amazing. I believe it’s due to the fact that Bard is using google lens API and was not trained in a multi-modal way like GPT-4 has.

One caveat is, the same as text models, Bing can and will hallucinate stuff that isn’t in the picture, so YMMV but take this into account. It seems that at the beginning of an image description it will be very precise but then as the description keeps going, the LLM part kicks in and starts hallucinating.

Is GPT-4 getting dumber and lazier?

Researches from Standford and Berkley (and Matei Zaharia, the CTO of Databricks) have tried to evaluate the vibes and complaints that many folks have been sharing, wether GPT-4 and 3 updates from June, had degraded capabilities and performance.

Here’s the link to that paper and twitter thread from Matei.

They have evaluated the 0301 and the 0613 versions of both GPT-3.5 and GPT-4 and have concluded that at some tasks, there’s a degraded performance in the newer models! Some reported drops as high as 90% → 2.5% 😮

But is there truth to this? Well apparently, some of the methodologies in that paper lacked rigor and the fine folks at AI Snake Oil ( Sayash Kapoor and Arvind) have done a great deep dive into that paper and found very interesting things!

They smartly separate between capabilities degradation and behavior degradation, and note that on the 2 tasks (Math, Coding) that the researches noted a capability degradation, their methodology was flawed, and there isn’t in fact any capability degradation, rather, a behavior change and a failure to take into account a few examples.

The most frustrating for me was the code evaluation, the researchers scored both the previous model and the new June updated models on “code execution” with the same prompt, however, the new models defaulted to wrap the returned code with ``` which is markdown code snippets. This could have been easily fixed with some prompting, however, the researchers scored the task based on, wether or not the code snippet they get is “instantly executable”, which it obviously isn’t with the ``` in there.

So, they haven’t actually seen and evaluated the code itself, just wether or not it runs!

I really appreciate the AI Snake Oil deep dive on this, and recommend you all read it for yourself and make your own opinion and don’t give into the hype and scare mongering and twitter thinkfluencer takes.

News from OpenAI - Custom Instructions + Longer deprecation cycles

In response to the developers (and the above paper), OpenAi announced an update to the deprecation schedule of the 0301 models (the one without functions) and they will keep that model alive for a full year now!

Additionally, OpenAI has released “Custom Instructions for ChatGPT” which allows a chatGPT user to store custom instructions, information and custom prompt that will be saved on OpenAI server side, and will append to every new session of yours with chatGPT.

Think, personal details, preferred coding style (you love ruby and not python) and other incredible things you can achieve without copy-pasting this to every new session!

Don’t forget to enable this feature (unless you’re in the UK or EU where this isn’t available)

Thanks for tuning in, wether you’re a newsletter subscriber, twitter space participant, or just someone who stumbled onto this post, if you find this interesting, subscribe and tell your friends!

“We stay up to date so you don’t have to” is the #ThursdAI motto! 🫡

In other news this week:

LangChain has gotten some flack but they are looking ahead and releasing LangSmith, an observability framework for your agents, that does NOT required using LangChain!

It looks super cool, and is very useful to track multiple prompts and tokens across agent runs! And the results are share-able so you can take a look at great runs and share yours with friends!

Don’t forget to share this with your friends and come back next week 🫡

— Alex Volkov

Hey there, welcome to this special edition of ThursdAI. This episode is featuring an interview with Nous Research, a group of folks who fine-tune open source large language models to make them better. If you are interested to hear how finetuning an open source model works, dataset preparation, context scaling and more, tune in!

You will hear from Karan, Teknium, LBJ from Nous Research and Enrico who worked along side them.

To clarify, Enrico is going in depth into the method called Rope Scaling, which is a clever hack, that extends the context length of LLaMa models significantly and his project LLongMa which is an extended version of LLaMa with 8000 token context window.

The first voice you will hear is Alex Volkov the host of ThursdAI who doesn’t usually have a lisp, but for some reason, during the recording, twitter spaces decided to mute all the S sounds.

Links and acknowledgments:

* Nous Research - https://nousresearch.com/ (@nousresearch)

* Redmond Puffin 13b - First LLaMa Finetune

* LLongMa - LLaMa finetune with 8K context (by Encrico, emozilla and KaioKenDev)

* Nous-Hermes-Llama2-13b-GPTQ - Hermes Finetune was released after the recording 🎊

Psst, if you like this, why don’t you subscribe? Or if you are subscribed, consider a paid subscription to support #ThursdAI

Show transcription with timestamps:

Alex Volkov - targum.video (@altryne)[00:00:55] Yeah. That's awesome. So I guess with this, maybe, Karan, if you if you are able to, can you you talk about Nous research and how kind of how it started and what the what are you guys doing, and then we'll dive into the kind of, you know, Hermes and and Puffin and the methods and and all of it.

karan (@karan4d)[00:01:16] Absolutely. Nous research. I mean, I I myself and many other of us are just, like, enthusiasts that we're fine tuning models like, you know, GPTJ or GPT 2. And, you know, we all are on Twitter. We're all on Discord, and kind of just found each other and had this same mentality of we wanna we wanna make these models. We wanna kinda take the power back from people like OpenAI and anthropic. We want stuff to be able to run easy for everyone. And a lot of like minds started to show up.

karan (@karan4d)[00:01:50] I think that Technium's addition initially to Nous research, Jim, kinda showing up. And himself, I and human working on compiling the Hermes dataset was really what came to attract people when Hermes came out. I think we just have a really strong and robust, like, data curation thesis in terms of that. And I think that have just some of the most talented people who have come to join us and just volunteer and work with us on stuff. And I absolutely must say, I can see in the in the listeners is our compute provider, Redmond AI.

karan (@karan4d)[00:02:30] And, you know, none of this none of these models would be possible without Redmond's generous sponsorship for us to be able to deliver these things lightning fast, you know, without making us through a bunch of hoops just a a total total pleasure to work with. So I would I have to shell and say, you know, I highly recommend everyone check out Redmond as because they really make our project possible.

Alex Volkov - targum.video (@altryne)[00:02:52] Absolutely. So shout out to Redmond AI and folks give them a follow. They're the the only square avatar in the audience. Go take them out. And, Karan, thanks for that. I wanna just do a mic check for teknium. Teknium. Can you speak now? Can you? Can I hear you?

Teknium (e/λ) (@Teknium1)[00:03:08] Yeah. My phone died right when you were introducing me earlier.

Alex Volkov - targum.video (@altryne)[00:03:10] Yep. What's up, Eric? -- sometimes on Twitter basis. Welcome, Technium. So briefly, going back to question. I don't know if you heard it. What besides the commercial and kind of the the contact window, what kind of caught your eye in the llama, at least the base until you guys started, or have you also, like, the other guys not had a second to play with the base model and dove into fine tuning directly?

Teknium (e/λ) (@Teknium1)[00:03:35] Yeah. The only thing that really caught my eye was the chat model and how horribly RLHF it was.

Alex Volkov - targum.video (@altryne)[00:03:41] Yeah. I've seen some conversations about and kind of the point of Ira, RLHF as well. And okay. So so now that we've introduced Neus research, sorry, I wanna talk to you guys about what you guys are cooking. Right? The we've seen, the the Hermes model before this was, like, loved it as one of the, you know, the best fine tunes that I've seen at least and the the the most performing ones. Could you guys talk about the process to get to the Hermes model, the previous one? and then give us things about what coming soon?

karan (@karan4d)[00:04:16] Teknium, you got this one. man.

Teknium (e/λ) (@Teknium1)[00:04:22] Yeah. It was basically I saw Alpaca, and I wanted to make it like, remake it with GPT 4, and then from there and just pretty much exclusively included anything that was GPT 4 only, and that was the beginning of the thesis for that. Going forward, though, We still have a lot of low quality data, I think, in Hermes data set that can be cleaned out, and then there's a lot of new data sets that have come out that I wanna start merging into there. also wanna move to something like chat ML or even Vikura format so that we can do some multi turn stuff. It's not very great, long chat.

Alex Volkov - targum.video (@altryne)[00:05:03] Yeah.

karan (@karan4d)[00:05:03] Within within within the Hermes dataset, you know, a lot of it is public available stuff that's particularly GPT 4. Of course, Technium's massive GP teacher dataset. We also have a bunch of GPT 4 data we had generate that we didn't release necessarily just yet, as well as an instruction set that's particularly focused on tasks like Python, transformers, linguistics, very small dataset of that. That's inside Hermes that, you know, we don't really talk about much, but figure that we'll put some exposure to right now on the spaces. And yeah.

Alex Volkov - targum.video (@altryne)[00:05:42] That's awesome. And so the previous Hermes was released on top of LAMA 1, and for many folks, know, obviously, they couldn't use this for different commercial points. And now that this model relates, what the models that you guys release, are you thinking about the license of them? And could you talk about, like, the availability of folks using them in commercial standing now that, you know, the the back of it is commercially available.

LDJ (@Dogesator)[00:06:07] Mhmm. I think we have puffin licensed us MIT I'll have to doublecheck on our own own model. I think that's right, Curran, right, or Tech?

karan (@karan4d)[00:06:18] Yeah. I think so either that or Apache 2 point Like, if if the base model is commercially usable, you know, the stuff we put out is you're good to go. It's -- Yeah.

LDJ (@Dogesator)[00:06:29] So And, like, in our announcements, I put in kind of, you know, one of the main things. It's it's commercially available. the first Nous as far as I think yeah. I'm pretty sure it's the first commercially available Nous model that's released, and a big differential data from Hermes is the fact that, like tech was saying, Hermes is pretty much all single turn data. And it's surprisingly can do pretty decent at multiturn conversations when you actually use it. But then puffin is almost kind of, like, a 180 where it's a vast majority really on context multi turn data.

LDJ (@Dogesator)[00:07:09] And oh, I think can you guys hear me so? I can hear. Okay. It's just something's up with that. Okay. Yeah. So puffin is a vast majority, multi turn data, GPT 4 specifically, and a lot of it is actually real human conversations with GPT for that go on for some of them 4k 6 k context, like, even all the way up to the max 8 k context length of GPT 4. And then we took those few thousand conversations of real humans interacting with GPT 4. And now after that, I'm not sure if you've A lot of people probably heard of Camel AI.

LDJ (@Dogesator)[00:07:46] So they have the physics, biology, chemistry, and mathematics data set. And then within those, there's a bunch of subtopics that you can carry it through. And I just pretty much spent a good few days curating just handpicking the right subtopics, like differential geometry, logic problems, optimization problems, a bunch of different GPT, for examples, and responses from those different subtopics. And then I specifically added those in certain ways to the puffin dataset.

Alex Volkov - targum.video (@altryne)[00:08:17] Awesome. So just just looking for the audience maybe. The puffin model that I think the official name is the red redmon puffin 7B or, sorry, 13B. Yes. This is this is the model that you guys fine tuned, and one of the first is maybe not the first fine tune of llama v Two. that's now publicly available, like you said, maybe with MIT license on Huggingspace, and I think you even added the GGML quantized version. Correct? Mhmm. So and so folks can can go and download that and and already start playing with this. And so first of all, thank you for contributing to the open source. That's great to see. And the speed with which you guys are fine tuned on this is also great to see.

Alex Volkov - targum.video (@altryne)[00:08:55] And maybe now that we've introduced this, maybe this is like repeating a bit. So could you speak about the the difference so the difference is the in the data set, in the task that you fine tune? Like, what is the actual difference between the Hermes or the Hermes that's coming out and the Puffin model? What would people use them for differently? Is that like that? That's a question.

Teknium (e/λ) (@Teknium1)[00:09:21] The profit model definitely be better at multi turn stuff. That's for sure. Yeah.

nisten (@nisten)[00:09:28] So if you want to do anything like OpenAI I'll I'll paste the link above the GGML version of it because I I really I'm I'm gonna test it thoroughly, but I I really think because you guys have use GPT 4, high quality, multi turn conversations, then this can have actual, like, practical use for whoever else was to use it either as, like, something that tells you about the documentation on the site or walks a user through. In other words, this should be better than Hermes then in for, like, customer service stuff, which is just one example.

nisten (@nisten)[00:10:08] Anyway, yeah, I'm gonna try. I'll I'll paste the the link above.

karan (@karan4d)[00:10:14] It's it's likely better for production use alongside, like, stuff that you have with, like, a retrieval pipeline, like, with lang chain, etcetera. Like, I I would believe that without to get it, you know, or just talking, of course. But, you know, there is even though, you know, with this Lima tech unique of of small examples where we can get, like, a a really good really good model that does really well.

karan (@karan4d)[00:10:41] The thing about Hermes dataset and just its size and the various types of data and topics that are in there, I think you get a totally different like, role play or storytelling experience or completion experience with Hermes. Personally, I feel that way.

Alex Volkov - targum.video (@altryne)[00:11:01] Awesome.

Teknium (e/λ) (@Teknium1)[00:11:01] So and that. Another thing about Puffin Dataset is that it does go up to, like, 8K and Enrico here. has been doing a ton of work on extending Llama's context.

Alex Volkov - targum.video (@altryne)[00:11:13] Right. So I wanna I wanna give an introduction then introduce Enrique and and talk about this real quick. Right? LAMA version 1 was released with, again, 2,000 tokens in the contact window. And then many folks, including KaioKendev, and Emozhila. Right? And and some other folks, I think, were involved in bringing some of the quote on quote tricks about what eventually ended up being named rope, scaling, if I'm if I'm not mistaken. And we follow this, and we've talked about the previous news ThursdAI, I. And Llama V2 was released with 4000 tokens in the context window.

Alex Volkov - targum.video (@altryne)[00:11:52] And, you know, we're now still used to kind of Claude and the 16k GPT 3 that four didn't seem like a lot. And then many folks were wondering, and, meanwhile, Enrico was working, whether or not the rope scaling method would apply to the next plumber and look like it did. And so I wanna introduce Enrico uh Enrico Shippole. I hope on staying this right. Welcome to the state. Hopefully, you can unmute and and this place works with you. And The second finetune that I saw rest of the was also back with Nous, the Nouse research, and this was the extended version, what's called Longma.

Alex Volkov - targum.video (@altryne)[00:12:28] So Enrique will go out of the stage and feel free to introduce yourself, your affiliation with news and LlongMa with with the context window.

Enrico Shippole (@EnricoShippole)[00:12:38] Hello. So I'm actually a independent researcher. I'm sponsored by Stability AI, Eleuther AI, and a few other different organizations, including NewsNow. Awesome. I work with different people like Tanishq from Medark, Aaron Komatsusaki, who also is from a Luther and Duck AI. John Ney from Nomosai. So I I have a I have a lot of affiliation with a bunch of different organizations. including together. We're starting a project right now with them.

Alex Volkov - targum.video (@altryne)[00:13:13] That's that's so great to hear, and so welcome to Thursday. Welcome to this day. And can you talk to us a little bit about kind of the ROPE scaling method and and how how were you able to, like, find them like this quickly and how the results looked so far? I wasn't able to run this myself. But hopefully, yeah, talk to us about

Enrico Shippole (@EnricoShippole)[00:13:34] Okay. So initially, The the thing is I actually was hoping that both Emozilla, Bowen, and KaioKenDev would have been able to make it because It was kinda like a equal parts effort on, like, all fronts from each of us. Initially, I had trained some pathways models at 8,000 context length about 4 months ago based on the exposition paper, which did rotary embedding scaling initially. They were one of the first people did it. They based their methodology off of ofer presses alibi.

Enrico Shippole (@EnricoShippole)[00:14:11] I would imagine that most people are pretty familiar with Ofir Press in this work on the alibi positional bias that's been used in a wide range of models now. So Emozilla and I came into contact based off of the work that he had seen me doing with the Palm models scaling those to 8000 context length pretraining, not fine tuning. So what we had initially done is basically take a section of c 4 in different data sets that had examples that were all over 8000 context length that pretrained on them packed together.

Enrico Shippole (@EnricoShippole)[00:14:50] with a beginning of string and end of string token to help with, like, the attention masking portion of that. After he had seen that, Emozilla actually became into contact with kaikode dev I believe Kaiokendev is how you pronounce it. Kaiokendev had also been following Ofir Press's research. He had started working on his own version of scaling the rotary embeddings, I believe based off of both alibi and exposition.

Enrico Shippole (@EnricoShippole)[00:15:22] And what he found is that by scaling the max position all embeddings and the rotary embedding from something like 2048, which you would initially train with. He scaled it up to 8000 or 8192. And he found that by applying, like, in interpolation to the encoding by scaling basically like the the positional index in the rotary embedding, that you were able to essentially turn down the frequency window and rope by like a factor of 0.25.

Enrico Shippole (@EnricoShippole)[00:16:01] The scaling depends on the length that you're trying extrapolate to and the initial context length that the model was trained with. So if you were training with LAMA 2, which had an context window of 4096, and you wanted to do the linear interpolation positional scaling to something like 8192. then you would use a scaling factor of 0.5. If you were trying to do it from 2048, which is the original LAMA was trained with, and you wanted to scale it to 8192, then you would use a scaling factor of 0 point 25.

Enrico Shippole (@EnricoShippole)[00:16:39] So basically, after we had done all of this, Meta had released a paper around the same time that Kaiokendev had released his blog. They both found very similar finding. They had shown in the meta paper that you only had to fine tune for 1000 steps with the linear positional interpolation scaling to be able to get the benefit of doing a full pretrain at a context window of 8192.

Enrico Shippole (@EnricoShippole)[00:17:13] So this is actually like a a big step because it shows that you no longer need to pre train right off the bat at a longer context length. Then you're able to do the fine tuning on essentially a a lower resource like, computational budget and still be able to get the, like, greater results of the longer context window. I know a lot of the major AI companies had been doing just for my work in in personal research with many of them had been doing staged scaling of the context window during training.

Enrico Shippole (@EnricoShippole)[00:17:46] So they would pre train basically, when pre training, they would separate the initial examples from a dataset into multiple stages.

Enrico Shippole (@EnricoShippole)[00:17:54] So anything that is under the window of 2048, you'd separate from the initial dataset then you take things between 2048 4096, then 4096, and 8192, and you would basically chunk the data sets into those different parts you'd first initially train on the 2048 chunk of the data, then you would train on the data between 2048 and 4096, and then you would do the same thing from 4096 to 8192, or if you want to scale that to 16k or 32k context length. But what we have shown now with both the meta paper and this thing, you don't even need to go through that extensive pretraining and staged process, you can just go from a context length of 2048 to 8192.

Enrico Shippole (@EnricoShippole)[00:18:47] scale the rotary embeddings by whatever type of factor that you want to use. So like I was saying, if you're going from 2048 to 8192, you'd be using a scaling factor of 0.25. It only needs 2 lines of code to be able to do that. In the LLongMa post, I had provided an example of scaling the rotary embeddings. The the code was written by Emozilla or Jeff.

Enrico Shippole (@EnricoShippole)[00:19:15] We also came into contact with after all these experiments we then came into contact with Bowen, who had worked a lot about the dynamic NTK scaling with Emozilla, and he had also done NTK by parts which we're we're currently training a lot of models on. So we have the Longma 1 models trained on the open llama series, like the suite of those models that use the linear interpolation scaling.

Enrico Shippole (@EnricoShippole)[00:19:45] We now have the llama 2 models or the longma 2 suite, which is what we're calling it, again, trained on the linear interpolation scaling And then we have another suite of models coming out very soon that uses the the NDK by parts dynamic scaling. That was really specialized by Bowen, so I do not wanna speak on his behalf. It'd it'd probably be good to get him to talk about it in another one of these.

Alex Volkov - targum.video (@altryne)[00:20:14] Absolutely. So let's get in touch after this and and and and set it up. So Thank you for the a very in-depth kind of explanation because we did cover the the the kind of the RoPE killing and how Kaioken in the image boards are ready to wherever he started this in his blog post, and then how it's gonna rotate it. So it's great to to actually hear from the folks who are doing this. I just for the audience, I've attached Enrico's tweet about LLongMA 2, which is now currently trained at AK contact length.

Alex Volkov - targum.video (@altryne)[00:20:47] And and Rico, you told us that we may see even double from the So could you think about the next the next version?

Enrico Shippole (@EnricoShippole)[00:20:56] Okay. So the the initial training process of doing this up to a context, like length of 8192, can be due with be done, basically, with deep speed, 02. and activation checkpointing. And you're able to fit the model on a A100 80 gigabyte node. Now, we are working on the process of scaling it both to 16 k and 32 k. This requires a different methodology during training, you either need to use deep speed 0.3 or fully sharded data parallelism.

Enrico Shippole (@EnricoShippole)[00:21:35] Both of those are are very similar for people who aren't aware. Basically, you're just sharding the optimizer states. The model states across, like, different nodes. You can also use things like tensor parallelism to help with the scaling as well. And then we're going to be basically just adjusting the scaling factor again, collecting a large we've already collected large quantity of data at 16k context length, and we're going to be doing the fine tuning to 16k and be releasing those models Soon, all of this computing is sponsored by stability AI.

Enrico Shippole (@EnricoShippole)[00:22:12] They've been very generous what helping with a lot of the independent research.

Alex Volkov - targum.video (@altryne)[00:22:17] That so I wanna shout out Stability AI for not only given, you know, the world's stability diffusion, also participating in this kind of next wave of AI. Many folks kinda coined the stability AI moment when released the the stable diffusion of the. I wanna say 1.4 back then almost a year ago now, and many folks are saying the about the Llama 2 release now this commercially open source, and and folks can start, like, doing things for you know, for profit companies can join So we definitely wanna shout out stability for for the effort here. And, Enrico, thank you. And, folks, please follow Enrico, and and we'll stay tuned.

Alex Volkov - targum.video (@altryne)[00:22:56] I wanna ask Karan and and Teknium, and other folks from Nous the efforts that that Enrico was talking about. the longer context windows. How would they kinda interplay with the stuff that you're working on with Hermes with with Pufin? Are are kind of the efforts interchangeable? We're gonna see building a top of each other?

karan (@karan4d)[00:23:16] So I I think LDJ can definitely speak to this, but I'd like to happily say that once we did Longbow 1 on the 1st Llama generation of models, we already had puffin 2k, 4k, and 8 for that -- Yeah. -- already prepared and ready. So as the LLongMa models for 13B are released, we will also be doing equivalent, puff in fine tunes, and Potentially Hermes fine tunes. We can talk a little bit more about the future of Hermes at a a little bit later, though.

LDJ (@Dogesator)[00:23:51] Yeah. I mean, I was pretty much going to say the same thing, but kind of elaborate on that about how before when LLongMa V1 and everything. And during the development of LLongMa, there was actually, like you know, of course, me Enrico who are usually just called concepts of mind and and and Emozilla. Like, we've all kinda, like, been butting shoulders a lot together and just kinda working closely, you know, in the same Discord and whatnot. And it's like, hey. Like, you know, working on this, like, experimental LLongMa with thing. Like, hey. You wanna try, like, fine tuning, and then the plan just kind of ended up being like, okay. Just gonna have this Puffin thing.

LDJ (@Dogesator)[00:24:31] that Puffin dataset is already containing a ton of high context conversational data. from GPT 4 and, like, human high quality data. So it's like it's like the perfect fit to have something that's high context capable will be fine tuned on that. And then LLaMa 2 came out, and it's like, oh, Yeah. Let's let's get this out ASAP, and then we'll figure out what we're gonna do later.

Alex Volkov - targum.video (@altryne)[00:24:58] Yeah. Great. And it's just great to see, you know, how many opportunities is like this where with open source can the stuff that we're able to now run and gonna iterate on are building on top of each other. They're just incredible. and this is maybe a watershed moment. And I I wanna thank all of you for being here. I wanna kind of let the other folks who usually hear on Thursday, I need to ask you a question or 2 for Nous visitors. Yam and Nisten, if you if you have a question for news or for Enrico, go ahead. I I will stay young.

Alex Volkov - targum.video (@altryne)[00:25:29] I know you if you have to ask the super deep technical stuff, and the audience will, like it will fly over their I I won't be using the DM with LBJ and and Rico. But yeah. Of course, the stuff that we haven't covered and interesting tough news. Feel free as it pertains to LAMA 2 is gonna be very interesting, I think, for everyone.

nisten (@nisten)[00:25:47] Just to quickly clarify, you guys fine tuned the plain model. Right? Not the chat 1.

Teknium (e/λ) (@Teknium1)[00:25:55] Yep. Okay. Yep. The base model. We wouldn't fine that model. The chat 1 at all.

Alex Volkov - targum.video (@altryne)[00:26:00] Actually, to -- Yeah. The -- -- to maybe continue this stratigram for interrupting. Just one sec. To continue this question, the there are models they were released by Meta, and you have to, like, register and get the email and everything. And then they put some stuff on Hugging Face. And then the the those models were delineated with, like, dash HF. Have you guys use the HuggingFace or the Meta 1, and do you guys know the difference? I felt somebody that, like, maybe doesn't work as well and to inform her Yeah.

Teknium (e/λ) (@Teknium1)[00:26:30] The one on Hugging phase is an FP 16 and the original Llama 2 models in bf16, but we tested the difference between the two models at Carper, and there's such a negligible difference in their quality that it's irrelevant, but we trained on the Hug and Face f P Sixteen ones, but in the f Sixteen ask them.

Alex Volkov - targum.video (@altryne)[00:26:52] Sorry. Yeah. Goran, for interrupting. Go ahead.

karan (@karan4d)[00:26:56] No. All good.

Alex Volkov - targum.video (@altryne)[00:26:58] I I totally forgot what -- That's not it. interrupted today. Yes, Randall. Okay. Nispen, if you have a question for Kiran to follow-up with feel free, and And if not, then, Yum, if you have anything that you wanna ask the the fine folks from Nous, feel feel free as well.

Yam Peleg (@Yampeleg)[00:27:17] Yeah. Sure. First, thank you for what you're doing, guys. You're really making a difference for anyone. There aren't many demos online, so anyone that didn't try Hermes, I highly encourage you to try. I don't know why there aren't them. Okay. I know why there aren't demos that cost money, but just try it. Okay? And now I got a question because from my experience, if you train on the open datasets of Hermes, you get a significantly less quality of a model. No. Now I'm fine I'm fine if you don't release datasets. Don't don't get me wrong.

Yam Peleg (@Yampeleg)[00:27:54] Just I wanted to ask, is there anything else besides the data that is different? What what tips can you give for, I don't know, someone else that want to train high quality model besides having high quality data.

Teknium (e/λ) (@Teknium1)[00:28:08] Everyone understands this. Yeah. The hyperparameters can make key difference. LBJ knows very well because we had to do a ton of different tests. We don't have our freight owners for puffin model. But I'm not sure if those are on the model card for Hermes. If they're not, I can put them And Karen your card can probably talk about the Nous datasets that weren't made public.

karan (@karan4d)[00:28:38] Yeah. We've got, like, maybe around, like, 50 k items of data, like, versus, like, total 300 k instructions there that are not released. And to be frank with you about 45 k of them is just more GPT 4, like, alpaca style instructions. The 5000 or so, the, like, 4500 them compose this dataset we have we've been working on that, you know, at this point, I'm pretty comfortable talking about a we call it the p dactyl dataset.

karan (@karan4d)[00:29:14] I won't speak on everything that's in it, but, essentially, And I don't know if this is the thing that made the big difference, but it's, like, the the one place where I guess you deviate from just using the open datasets more GPT 4 instructions, but it's got some transformers instructions, some linguistics instructions, some calculus 1, instructions, etcetera. It seems to be pretty good.

Teknium (e/λ) (@Teknium1)[00:29:41] Also, Yam, do you have links or anything to the models that tried it with just the makeup of the datasets that we're public from Hermes because I haven't actually seen that before.

Yam Peleg (@Yampeleg)[00:29:57] And again, can you repeat that?

Teknium (e/λ) (@Teknium1)[00:29:58] didn't hear. Do you have any links to the models that trained with just the open datasets from Hermes that you could share with me later?

Yam Peleg (@Yampeleg)[00:30:06] No. No. It's just it's just from my experiments -- Oh, okay. -- on training. Pretty much following the same idea of let's take only GPT 4 from all the open datasets, and the the model that you get is is different. for sure. And and it might be that hyperparameters, you know.

Teknium (e/λ) (@Teknium1)[00:30:25] Another thing that we did too is pretty extensive, like, cleaning. We did do deduplication. We removed things like a URL. Like, any response that had a URL in it, we removed in case it was gonna like, hallucinated URLs. Instead of, like, maybe 8 different filtering processes too that might have made our data quality higher.

LDJ (@Dogesator)[00:30:48] So as an AI language model?

nisten (@nisten)[00:30:51] For anybody -- What do you say? -- for anybody in the audience that hyperparameter meters are are just like the settings in the oven. So it it looks here, like, the ingredients were all okay, but yam mess something up, and before selling as a token -- Yeah. -- came out half baked at the model.

LDJ (@Dogesator)[00:31:08] So we're gonna have to check that out.

LDJ (@Dogesator)[00:31:10] I'm a big proponent personally of hyperparameter optimization being underrated right now, like, in -- Yeah. -- the current space. And that's something I've kind of focused on a lot specifically for things like puffin and just trying to help others around and use some stuff like trying to optimize they're doing, and even just something like like what you just said about the settings for the oven, I mean, double the amount of time you're putting something in the oven, and it's not gonna come out twice as good. It's not even gonna come out 10% as good. It's gonna come worse. You know?

LDJ (@Dogesator)[00:31:45] And although it depends, like, what is your baseline for how how much time you're putting it in the oven and all these different variables that kind of are dependent on each other and affect each other. So it's definitely something you kind of have to build an intuition about to some degree. And then the other end is really I feel like there has to be more investment and more time and energy invested into actual tools that make hyperparameter optimization easier for people that are doing these things.

Yam Peleg (@Yampeleg)[00:32:13] Yeah. Yeah. And the thing is that the models are are really big, so it's really expensive to run them. So you have you have a trade off of how many how much computer you're investing in searching hyperparameters rather than actually using it for training. But but I completely agree So one one last question, actually, too.

Teknium (e/λ) (@Teknium1)[00:32:33] Actually, one thing before we go on. Something great about the puffin dataset is that it's just like, 3000 or so examples, I believe. And so it makes tuning a lot less expensive because you can finish the whole training in just a couple of hours. So, like, with Hermes, if we wanted to try full ablations and dozens of them, it would take weeks weeks to do.

LDJ (@Dogesator)[00:32:55] Yeah. Yeah. Well, to be fair, it's not like it only takes a couple hours on one GPU. We use a a 100 80 gigabytes. So Yeah. Yeah.

Teknium (e/λ) (@Teknium1)[00:33:04] Courtesy of Redman.

Alex Volkov - targum.video (@altryne)[00:33:05] Thank you, Redman.

Enrico Shippole (@EnricoShippole)[00:33:08] Mhmm. I should also probably clarify that when doing the context length, extrapolation, We're doing it on 1,000,000,000 tokens and 64, 80 gigabyte a 100.

Yam Peleg (@Yampeleg)[00:33:20] OOf Mhmm.

Alex Volkov - targum.video (@altryne)[00:33:23] Yeah. Yam is getting over excited. Alright, folks. I wanna -- Yeah. Yeah. -- maybe maybe ask her on this one less and we'll move on to the the the regular ThursdI update camera cadence. But I will say that, like, folks from Nous research and and Rick and and some other here. Thank you so much for coming up and giving us kind of the insights into how this actually happens. Lama2 just released, you know, a few days ago, and you guys are already pumping out, like, open source fine tuned models. And it's great to see. And just so you know, there's always a stage for you here to come in and and announce things.

Alex Volkov - targum.video (@altryne)[00:33:53] And If you do wanna announce, like, a release or something, maybe just, you know, right now, Karan and and Teknium and some folks, I would love to hear like, when the next Hermes is coming?

karan (@karan4d)[00:34:06] Before we say that, I just would like to clarify something about Hermes. So we have the original Hermes dataset on LAMA 2 as something that we will release, but also a sequel to the Hermes dataset, Hermes 2. There will be a distinction between these 2, and you'll see you'll see the the the prior come out first and the latter come out after. But as for release, etcetera, I will absolutely let Technium take the stage with those final words.

Teknium (e/λ) (@Teknium1)[00:34:36] So the training is nearly done. At least it was about 2.8 epochs out of 3 a few hours ago. So it might be done already. Before I release it though, unlike puffin, I didn't we wanted it puffing out, like, same day that llama 2 came out, so we didn't run any benchmarks. And we had to put all the compute we had on Hermes immediately after we were done with that. So we don't have any compute to do any benchmarks or puffing until Hermes is done.

Teknium (e/λ) (@Teknium1)[00:35:06] But before I release Hermes, I do wanna do, like, a full range of benchmarks and stuff like that to make sure everything's good and have a pretty detailed model card, but that should probably only take the rest of tonight at the most. So probably tomorrow morning would be when Hermes comes out.

Alex Volkov - targum.video (@altryne)[00:35:22] That's some folks. And you you heard it here first and definitely follow Teknium, Karan, Enrico, LDJ, and the rest of, like, Nous Research folks, and stay tuned. Enrico, go ahead.

Enrico Shippole (@EnricoShippole)[00:35:34] Yes. I just wanted to to piggyback off of Teknium comment a little bit. So we did do pretty sense of the valuation of the Lauma 2 AK models. We had run different things on perplexity using Gov Report in a couple different other data sets to make sure that the length extrapolation in the context was working properly. We did passkey retrieval. We also did a lot of extensive human evaluation, which took a little bit. I had wanted to get the LAMA 2 AK models out yesterday, but we decided to push it back one day.

Enrico Shippole (@EnricoShippole)[00:36:08] So and what we were doing is we were feeding in research papers and seeing if it could pull out even, like, relevant pieces of information from the context length. And so far, it has been quite successful. So we're we're still running more evals, but the ones so far have shown that there's been, like, no performance degradation, no matter what context length that you're basically using with these extended models.

Alex Volkov - targum.video (@altryne)[00:36:32] That sounds great. and now that this this, you know, LLongMa lies out and the next versions are gonna come out as well. I'm sure that some other folks who also contribute to this research and tell you, like, from their own experiences and vibe. So, yeah, I wanna thank folks. Again, this has been very illuminating, and very glad to have you. And, obviously, the stage is yours whenever you want to come here, and we appreciate you. And you guys are welcome to stay tuned and kinda chime in to the rest of the updates. And with that, I think, for folks in the audience, we're moving to the next thing.

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

⏰ Breaking news, ThursdAI is now on Apple Podcasts and in this RSS ! So use your favorite pod-catcher to subscribe or his this button right here:

Our friends at Zealous have provided an incredible platform for us to generate these awesome video podcasts from audio or from twitter spaces so if you prefer a more visual format, our deep thanks to them!

P.S - You can find the full 2 hour space with speakers on our Zealous page and on Twitter

Here’s a summary of the main things that happened in AI since last ThursdAI:

🧑‍🎨 Stability.ai releases SDXL1.0

* Generates 1024px x 1024x stunning images

* High high photorealism

* Supports hands and text

* Different (simpler?) prompting required

* Fine-tunes very well!

* Supports LORAs, ControlNet in-painting and outcropping and the whole ecosystem built around SD

* Refiner is a separate piece that adds high quality detail

* Available on Dreamstudio, Github, ClipDrop and HuggingFace

* Also, is available with incredible ComfyUI and can be used in a free Colab!

Image Credit goes to Thibaud

Superconductors on Hugging Face? What?

Honestly, this has nothing immediate to do with AI updates, but, if it pans out, it’s so revolutionary that it will affect AI also!

Here’s what we know about LK-99 so far:

* 2 papers released on arXiv (and hugging face haha) in the span of several hours

* First AND second paper both claim extraordinary claims of solving ambient superconductivity

* Ambient pressure and room temp superconductive material called LK-99

* Straightforward process with a clear replication manual and fairly common materials

* Papers lack rigor, potentially due to rushing out or due to fighting for credit for nobel prize

* The science is potentially sound, and is being “baked and reproduced in multiple labs” per science mag.

Potential effects of room temperature superconductivity on AI:

While many places (All?) can benefit from the incredible applications of superconductors (think 1000x batteries) the field of AI will benefit as well if the result above replicates.

* Production of GPU and CPU is power-constrained and could benefit

* GPU/CPUs themselves are power-constrained while running inference

* GPT-4 is great but consumes more power (training and inference) than previous models making it hard to scale

* Local inference is also power-restricted, so running local models (and local walking robots) could explode with superconductivity

* Quantum computing is going to have a field day if this is true

* So will fusion reactors (which need superconductors to keep the plasma in place)

As we wait for labs to reproduce, I created a twitter list of folks who are following closely, feel free to follow along!

AI agents protocol, discussion and state of for July 2023

* Participated in an e2b space with tons of AI builders (Full space and recap coming soon!)

* Many touted AI agents as a category and discussed their own frameworks

* Folks came up and talked about their needs from the agent protocol proposed by e2b

* Agents need to be able to communicate with other agents/sub agents

* Tasks payloads and artifacts and task completion can be async (think receiving a response email from a colleague)

* The ability to debug (with timetravel) and trace and reproduce an agent run

* Deployment, running and execution environment issues

* Reliability of task finish reporting, and evaluation is hard

Frontier model forum

* OpenAI, Anthropic, Google, and Microsoft are forming the Frontier Model Forum to promote safe and responsible frontier AI.

* The Forum will advance AI safety research, identify best practices, share knowledge on risks, and support using AI for challenges like climate change.

* Membership is open to organizations developing frontier models that demonstrate safety commitment.

* The Forum will focus on best practices, AI safety research, and information sharing between companies and governments.

* Some have expressed concern that this could enable regulatory capture by the “Big LLM” shops that can use the lobbying power to stop innovation.

StackOverflow AI - “The reports of my death have been greatly exaggerated”

Stack overflow has been in the news lately, when a graphic of it’s decline in traffic has become viral.

They have publicly disputed that information claiming they have moved to a different measuring and didn’t update the webpage, but then also… announced Overflow AI!

* AI search and aggregation of answers + ability to follow up in natural language

* Helps drafting questions

* AI answers with a summary, and citations with the ability to “extend” and adjust for your coding level

* VSCode integration!

* Focusing on “validated and trusted” content

* Not only for SO code, stack overflow for teams will also embed other sources (like your company confluence) and will give you attributed answers and tagging abilities on external content

This has been an insane week in terms of news (👽 anyone?) and superconductors and AI releases! As always, I’m grateful for your attention! Forward this newsletter to 1 friend as a favor to me if you learned something new? Or alternatively, retweet us on twitter for bigger reach!

Thank you! See you next ThursdAI (and on Sunday when I release the State Of Agents recap 😅 )

ThursdAI - Get in on this, and share w/ 1 friend 🫡

First of all, let me address this from the get go, I’m not a material scientist! I am pretty good at finding information in twitter’s incredibly noisy info stream. (hey, this is how I bring you AI updates every ThursdAI)

Since LK-99 is potentially groundbreaking and revolutionary, I’ve compiled a twitter list of everyone who I found credible, interested and a source of new information, and there’s now over 1.5K followers to this list alone!

Since this clearly is interesting to a lot of you, I reached out to a few prominent people on this list, and asked them to join a twitter space, to try and stitch together an update on the current state of LK-99, replication attempts, history and lore, as it stands a week after the original papers release.

If you found this interesting, you’re the type of person who wants to stay up to date, feel free to subscribe and keep this Substack alive!

First of all, let’s do some level setting. Superconductors are real, we’ve used them in MRI machines for example, but the currently available superconductors need extremely low temperature and high pressure to well.., and the promise of a room temperature and ambient pressure superconductor is the holy grail of energy use.

For a breakdown on what superconductors are, and what they can mean for the world, I strongly recommend this thread from Andrew Cote (published presciently a full two weeks before the LK-99 paper) or watch this incredible breakdown:

July 22nd, the LK-99 arXiv day!

On July 22nd, two papers describing “worlds first room temperature superconductor” were uploaded to arXiv:

2307.12008 - Sukbae Lee, Ji-Hoon Kim, Young-Wan Kwon (submitted by Kwon)

and after 2 hours and 20 minutes another paper was uploaded

2307.12037 - Sukbae Lee, Jihoon Kim, Hyun-Tak Kim, Sungyeon Im, SooMin An, Keun Ho Auh (Submitted by Hyuntak Kim)

You may notice that the first two authors on both papers are Sukbae Lee and Ji-Hoon Kim, and in fact LK stands for Lee and Kim and 99 in the LK-99 name stands for the year 1999 they have started research on this.

You may also notice that YW Kwon who submitted the first paper, is not included on the second one, and in fact, is no longer part of the Quantum Energy Research Institute (Aka QCentre) where he was a CTO (he’s no longer listed on the site)

If this shakes out, and SC is replicated, there’s definitely going to be a Netflix series on the events that led to YW Kwon to release the paper, after he was no longer affiliated with QCentre, with limited information so let’s try to connect the dots (a LOT of this connecting happened on the ground by Seo Sanghyeon and his friends, and translated by me. Their original coverage has a LOT of details and is available in Korean here

Let’s go back to the 90s

On the LinkedIn page of Ji-Hoon Kim (the page turned blank shortly before me writing this), JH Kim showed that he started working on this back in 1999, and they estimated they have a material that contained “very small amount of superconductivity” and together with Sukbae Lee, in 2018 they have established QCentre to complete the work of their Professor Emeritus of Chemistry at Korea University, the late Choi Dong-Sik (1943-2017) who apparently first proposed the LK-99 material (following the 1986 bonanza of the discovery of high temperature superconductors by IBM researchers).

Fast forward to 2017, a wish expressed in a last will and testament starts everything again

Professor Choi passed away, and in this will requested follow-up research on ISB theory and LK-99 and Quantum Energy Research Institute is now established by Lee and Kim (LK) and they continue their work on this material.

In 2018, there’s a potential breakthrough, that could have been an accident that led to the discovery of the process behind LK-99?

Here’s a snippet of Seo Sanghyeon explaining this:

Kwon Young-Wan the ex-CTO

Kwon is a Research Professor at Korea University & KIST, is the third author on the first arXiv paper, and the submitter, was previously the CTO, but at the time of the paper to arXiv he was not affiliated with QCentre for “some months” according to an interview with Lee.

He uploads a paper, names only 3 authors (Lee, Kim and Himself) and then surprisingly presents LK-99 research at the MML2023 international conference held in Seoul a few days later, we haven’t yet found a video recording, however a few reports mention him asking for an interpreter, and talking about bringing samples without demonstration and proper equipment.

Important to note, that

Enter Hyun-Tak Kim

H.T Kim is probably the most cited and well-known professor in academia among the folks involved. See his google scholar profile, with a D-index of 43 and has 261 publications and 11,263 citations.

He’s a heavy hitter, and is the submitter and listed as the author of paper number 2 submitted to arXiv, 2 hours and 20 minutes after paper number 1 above.

In the second paper, he’s listed as the third author (and the submitter to arXiv) and his contribution is acknowledged like so:

An author, Hyun-Tak Kim (H. T. Kim),’s knowledge on mechanisms of both superconductivity and the metal-insulator (gap-nogap) transition highly contributed to writing the mechanism part. The knowledge was acquired over 20 years by processes of performing national projects including project [Grant 2017-0-00830] funded by Institute for Information and Communications Technology Promotion (IITP) in MSIT of Korea government in ETRI. H. T. Kim left ETRI on Nov. of 2022.

In the first paper H.T. is not acknowledged, and is only mentioned in in reference no. 52 to his paper from 2021.

Ok enough about the people Alex! Does the rock levitate?

In January, QCentre youtube channel uploaded an unlisted video that showed magnetic properties of LK-99 and another video, with partial levitation is widely shared on social media.

The partial levitation shown is attributed to the Meissner Effect and is a supposed proof of room temperature super conductivity. However, these two videos are inconclusive and are not enough for us to take QCentre claims at face value.

The scientific community, having been stung by a recent incident surrounding a supposed room temp superconductor, where the evidence was apparently falsified (Dais et. al.) are not so easily swayed.

Adding to that, the mess around the multiple papers, showing different theories, the lack of peer review, or independent replication, the surprised publication, and a rushed follow up publication, all makes people wonder, what is going on here? This doesn’t seem like a fabricated attempt.

Summary of replication attempts so far (Sun, Jul 20)

Given the importance of this discovery, and the “relative” triviality of replication, common enough materials, the process is not extremely complex (but kids, do not try this at home) so we can bet that “furnaces in solid-state materials labs around the world have been cooking yesterday and today to try to reproduce” [Science Magazine]

We have reports from China that supplies of Led Apatite are running dry as many are trying to replicate quietly?

Additional reports from India where Dr. VPS. Awana, the Chief scientist at CSIR-NPL and team are trying to replicate, with results expected as early as tomorrow (Monday, Jul 31) and has been emailing with Lee

In addition to this, we’ve had Andrew McCalip from Varda space who has been live-tweetin, twitch streamin his “Meissner effect or bust” campaign to reproduce LK-99, while the world watches (Andrew joined the space as well) and provides ideas, materials and an outpour of support for this gung-ho, almost cowboy effort.

We’ve also had folks from MIT who claimed that professors who want to remain anonymous, and went to MML2023 are also in contact with the team and are trying to test the material.

Replication failure is … not a failure?

Discussing the replication attempts with experts on stage, we all concluded that there are likely 2 ways for the world to know wether LK-99 is a superconductor.

* Replication succeeds and scientists analyze the replicated sample

* QCentre team provides a sample, and some very smart independent folks put it under a microscope, a magnetism analysis and a bunch of other measurements and confirm that it’s a superconductor at room temperature.

While we wait for either of those, I encourage you to check out the resources, the space recording, and the list of folks I’ve collected to stay in the loop!

Here’s a list of relevant links:

* Paper 1 DOI

* Paper 2 Arxiv

* Paper 3 Arxiv

* New Scientist Interview

* ChosunBiz Interview (Korean)

* Yonhap Interview (Korean)

* Twitter List

And the list of folks who participated in the space, give them a follow:

* Alex Volkov (@altryne)

* Seo Sanghyeon (@sanxiyn)

* Ate-a-Pi (@8teAPi)

* Andrew McCalip (@andrewmccalip)

* Andrew Cote (@Andercot)

* Ely Rabani (@radsci)

* Robotbeat (@Robotbeat)

* Marsh Ray (@marshray)

* Ben (@BenShindel)

* Ken Condon (@KenCondon1)

* Jesus (@jesuslares_me)

* Danielle Fong (@DanielleFong)

For your convenience, attached is an AI transcription of the space with speakers and timestamps (may be off by a few minutes) :

[00:02:40] Alex Volkov (@altryne): Hello. Hello, everyone. There's a lot of you here, and I wanna welcome a shoot for up on stage while we wait for a few more guests, and then we can get started. Thank you so much for taking the time joining us. as you're as interested as all of us in this very exciting, very confusing, very potentially groundbreaking news. So I wanna introduce 2 folks up on stage 2 folks up on stage already, and bringing up another one just now. And hey, Andrew. Hey.

[00:03:18] Alex Volkov (@altryne): Hey, How are you guys?

[00:03:23] Ben (@BenShindel):

Doing well. How are you?

[00:03:27] Alex Volkov (@altryne): A little bit you know, the palms are a little bit sweaty. This is a insane turnout. Twitter is indeed a public space on because that we have. And, hopefully, spaces or two spaces, whatever they call it now, will hold. And I only invite Sam here to speak as well. Hey, Tobias. How are you?

[00:03:51] Ate-a-Pi (@8teAPi):

I'm good. I'm good. So good to good to, you know, hear from you guys in person, Alex. Thanks for putting the space together.

[00:04:00] Alex Volkov (@altryne): Thirdly. Andrew, we're gonna introduce Andrew, but many folks who are here already follow you and and follow your work. How how's your evening going, Andrew?

[00:04:12] Andrew McCalip (@andrewmccalip):

Lee, this has been a wild ride. Thanks for putting all this together. It's gonna be great to get all the information in one place for the first time. This is my first time experiencing the full volume of the Internet, and just been a a lot of fun to see all the positivity around the progress.

[00:04:29] Alex Volkov (@altryne): That's great. So I'll do my best that, you know, Mother think this. I will maybe preface this that I am not a scientist. Many of the terms that we'll hear today in the space I've heard for the first time a couple of days ago. What I am is a Twitter for many, many years, and I have collected a a list of folks who I I personally wanted to follow to kinda see the updates as they roll out, and we've seen many, many things roll out very quick. with a lot of confusion and different replication attempts from different places. And I just compiled the list for myself. I started following.

[00:05:08] Alex Volkov (@altryne): 8 to buy had incredible incredible content diving into the the timeline. I found I I wanna introduce thank you. Am I saying this right? I think you need to hit the the mute button in a mute. If this is your first time talking on RESTASIS. let me know if you're able to do that. And if not, we'll try to solve this. And out as I was collecting folks, And I I started seeing that Andrew started doing their application attempts and even doing Twitch.

[00:05:46] Seo Sanghyeon (@sanxiyn):

Can you hear me?

[00:05:47] Alex Volkov (@altryne): Can you hear me? We can hear you. Hey, Sam Kim. How are you?

[00:05:57] Seo Sanghyeon (@sanxiyn):

It it it's the noon in South Korea, and I'm fine.

[00:06:01] Alex Volkov (@altryne): the afternoon. Right?

[00:06:03] Seo Sanghyeon (@sanxiyn):

It's 1. Yes. Yes. It's the 1 PM.

[00:06:06] Alex Volkov (@altryne): Awesome. And so I was just doing an introduction maybe as you were telling up, you maybe not heard some of it. However, folks in the audience who followed this kind of thread and how we came to be here I have a a thread that I'll post on top here that has all the folks from the Twitter list that I forgot. And San Kyung and his his team is basically the reason for the space. Me and Nathan kind of found Sunqun. Am I saying Sunqun correctly? Is that is that the right way to say this?

[00:06:41] Seo Sanghyeon (@sanxiyn):

My name is. Your your, yeah, your pronunciation is not actually not.

[00:06:48] Alex Volkov (@altryne): Okay. I'll I'll turn my best to put months at the at the right names. And so we both me and 8 to 5, a a 34 in Saint Kyung, who's in Seoul currently, and definitely speaks the language we don't speak, and so there's a lot of insight and translation. And so, yeah, I guess we'll will get started, so feel free to present yourself, and then talk a little bit about your last few days and how you came around getting in this topic. and then how kinda what you found so far.

[00:07:28] Seo Sanghyeon (@sanxiyn):

I I didn't really expect to to speak.

[00:07:30] Alex Volkov (@altryne): That's okay. That's okay.

[00:07:32] Seo Sanghyeon (@sanxiyn):

That's put me put me on the spot. Yeah.

[00:07:34] Alex Volkov (@altryne): I don't wanna put you on the spot, but give us maybe a brief summary.

[00:07:44] Ate-a-Pi (@8teAPi):

Maybe maybe do you do you want me to help Sanyon?

[00:07:47] Seo Sanghyeon (@sanxiyn):

Yes, please. Okay. You you have read my right top, so maybe maybe you can explain what's going on.

[00:07:57] Ate-a-Pi (@8teAPi):

Okay. So I'm I'm just gonna I'm just gonna just to preface everything, I I'm writing a work of fiction. So all of you guys are just participating in an experiment. So but I'm trying to keep everything to kinda, like, factual and trying to interpret what what is kind of happening on the ground. Right? Shyam is much more factual, and he he has actually been doing a primary source work. So he's been actually digging up the actual Korean language science papers. He's been sitting down with friends They've kinda, you know, summarized and kind of tried to understand what's going on.

[00:08:36] Ate-a-Pi (@8teAPi):

And he's really the one that's, you know, put together this that that the you know, the the the mentor, you know, whose name, I think, in some transliterations comes out to TS's chair, some Donsick He the mentor was basically in superconductors in this idea of this kind of 1 dimensional super and he had this theory.

[00:09:00] Seo Sanghyeon (@sanxiyn):

That so the name is che. che. Oh, sure. Yeah. Yeah. Yeah. He was a a professor in the Korean University's Department of Chemistry.

[00:09:13] Ate-a-Pi (@8teAPi):

Yeah. And and so he he had this idea, this theory, and he had graduate students. and one of those graduate students was Lee, and Lee kind of took up the mantle of this this theory. And then they, you know, tied up with who was an experiment list.

[00:09:37] Ate-a-Pi (@8teAPi):

And then they kinda discovered this trace this coast of a trace of a material in 1990 And at that point, what happens is having discovered this trace, their path kind of diverge this, and Kim, the experimentalist, goes on to do a masters, not in superconductors. So he does his masters in something else, and then he does the battery materials kind of PhD, and he graduates in 2008.

[00:10:12] Ate-a-Pi (@8teAPi):

while Lee continues on the superconductor path, does experimental any when he publishes his PhD. It's both a theory and synthesis of superconductors. And then he graduates, and then he he goes to work as a science adjunct professor, which we which we just found out. Like, a computer science adjunct professor, and he's there for about, you know, 4, 5 5 years. He doesn't publish. And and I'm guessing at this point, he kinda gets, like, you know, cashier out of out of academia completely, and he sets up a consulting firm, basically, Q Center.

[00:10:50] Ate-a-Pi (@8teAPi):

And they start taking on consulting work. And and then, again, the timeline is a little bit unclear on whether or not they continue to work on on the on on the product on what they discovered. And what happens then is in 2017, Chey Dongksik passes.

[00:11:18] Ate-a-Pi (@8teAPi):

And as he passes, he he gets his former students together, and he asked them to finish off what they started to find this superconducting material that they saw a ghost of a trace of in 1999. And he passes, and they have no money. basically. Song Young has done, again, primary source research, and, you know, the the office space is basically, like, like, a two story building, you know, somewhere in the you know, in in Seoul. It's a very modern kind of office. They don't have much money.

[00:11:57] Ate-a-Pi (@8teAPi):

My guess my guess is that they need Kim. because KIM is the experimentalist, and I'm guessing also that none of the theory works at this point. The only thing that they have to go on is that they actually did find something in 1999. And Kim, I'm guessing, is also quite practical because he didn't do he didn't pursue the superconductors for the PhD. Right? Because he's quite practical, he's like, dude, you get me money. I'll join you. You don't have money. I'm not joining you for your wild goose, Jason. Right?

[00:12:36] Ate-a-Pi (@8teAPi):

So Lee goes out and he recruits Kwan. And Kwan is kind of like you know, he's he's a US PhD. He has a research university, you know, position. recruit them, and they get funding. And I think I think Sam Young, you were you were saying that Kwon is the one on the, you know, National Science Foundation of Korea's like you know, list, like, grant. Right? I I think that's what you said.

[00:13:08] Seo Sanghyeon (@sanxiyn):

So the paper mentions the public grant from South Korea. called the National Resource Foundation, which is like National Science Foundation in United States. And Korn is listed as a primary invest mitigate our PI, if then.

[00:13:25] Ate-a-Pi (@8teAPi):

Right?

[00:13:26] Alex Volkov (@altryne): Mhmm.

[00:13:27] Ate-a-Pi (@8teAPi):

Yeah. Yeah. That's right. Okay. So he he's the PI. So they recruit him as the PI, and Jade Kim, who is, you know, Lee's partner, basically leaves his very comfortable position as a research director in a hearing aid test.

[00:13:44] Seo Sanghyeon (@sanxiyn):

Yeah.

[00:13:44] Alex Volkov (@altryne): Yeah. Yes.

[00:13:45] Seo Sanghyeon (@sanxiyn):

Yes. Yeah. Hearing aid Yeah. I Or the eye test there? Yeah. Yeah. For the ISER tech and in manufacture, the battery is specialized for the hearing aid. code. It is a medical device. They have a different standard from other batteries. And company a small business in South Korea, but seems competitive worldwide.

[00:14:13] Alex Volkov (@altryne): So he leaves his let me let me -- Yeah. Go ahead. Just real quick and to give folks a quick summary. The main paper that we saw the explosion from that was published on July 22nd, so a week and and almost a day we're, like, almost 8 days into this. The three people that you you just said, besides the first professor, Choi or chair or Troy and or several places write it separately. So the the three people, SoftBank, Jihoon Kim, which is the LK in LK 99, right, Lee and Kim. And the third person you just mentioned is Young Wan, Kwan. Yes.

[00:14:52] Alex Volkov (@altryne): Those are the the 3 authors on the paper that kind of was published on our side out of the blue. 8 days ago. Please continue.

[00:15:03] Ate-a-Pi (@8teAPi):

Right. And then so at this at this point, they're in 2017, And, you know, Lee goes out and does the fundraising. He recruits Kwan, who's the research professor, Kwon is basically he's on the paper. He he's he's the principal investigator on the grant, but he's still a professor at university. So he's basically, I'm guessing, like, a day a day in the, you know, in the office at Q Center, very modest place. I think the grand size is pretty small, and they get this ESR machine.

[00:15:41] Ate-a-Pi (@8teAPi):

And again, from what I can tell, the ESR machine only came knows how to use it. Because none of the other people are actually synthetic, you know, synthesis people. They're all like theory guys, Kuan is a physicist. And Kim himself, JH Kim himself, he's looking for something which you have to know what you're looking for, right? Because that's what he says in his LinkedIn. He's like, I'm looking for some if you don't know what you're looking for, then forget about it. Right?

[00:16:19] Ate-a-Pi (@8teAPi):

But he he knows what he's looking for, and they refine, they refine, and they refine, and he keeps doing experiments. He keeps refining the experiment, and he goes through, like, a 1000 iterations. And somehow, starting in 2018, somehow, By the middle of 2018, they find it. So that that's a surprising thing for me because they've I I I suspect they they've been working on it you know, before or, you know, Jay and Lee had a breakthrough on their theory, so they knew how to narrow the workspace down. But somehow in at the end of the day, Kim is the one grinding.

[00:16:58] Ate-a-Pi (@8teAPi):

Through that 1000 experiments, finally, to get, you know, a sample that works.

[00:17:03] Seo Sanghyeon (@sanxiyn):

And then they start by -- No. No.

[00:17:05] Alex Volkov (@altryne): No.

[00:17:05] Ate-a-Pi (@8teAPi):

No.

[00:17:05] Alex Volkov (@altryne): No.

[00:17:05] Seo Sanghyeon (@sanxiyn):

No. No. No. No. No. No? So so besides the two papers, there is a paper published in April returning query. And In their own words, they describe what what prompted their breakthrough in 2018.

[00:17:27] Seo Sanghyeon (@sanxiyn):

and it said that so so they are putting the material in a quartz tube And because they called it to best courts to cancel and Brooke, And the material left after the breaking of the glass was had the property they wanted. So so it was an accidental discovery.

[00:18:02] Ate-a-Pi (@8teAPi):

So can can you repeat that? Like, they what what happened? They put it in the quartz tube, and the quartz tube broke accidentally?

[00:18:10] Seo Sanghyeon (@sanxiyn):

Yes.

[00:18:10] Alex Volkov (@altryne): Yes. Yes.

[00:18:11] Seo Sanghyeon (@sanxiyn):

I see. And and And that what's the breakthrough in 2018? I see. It's what I'm saying.

[00:18:19] Alex Volkov (@altryne): Yeah. I just wanna confirm what I hear. The breaking of the course you led to the incidental discovery. This is this is the the breakthrough as it's written in the first paper in Korea? Yes. Yes. Okay. So I'll just call ASAP, I'll just give it back for some logistics. Folks, if you look up on on top of the space, there's a few tweets we're pinning. And as we go along, we're gonna add some information on top of this. The 3rd the third we pin from dystopian breaker has a link to the original kind of Korean paper. So please go ahead, Datapai.

[00:18:54] Seo Sanghyeon (@sanxiyn):

So so quick -- Okay. point.

[00:18:56] Alex Volkov (@altryne): Yeah.

[00:18:56] Ely Rabani (@radsci):

Go ahead. Go ahead. This this could be important because, you know, as as soon as you expose it to the atmosphere, your getting hydration. And hydration, you know, might be harmful, might be helpful. From this, like, little account, it seems like it it it either didn't do anything or was helpful. But, like, no what temperature it was at when it broke, and and things like that could could actually be really pertinent.

[00:19:30] Ate-a-Pi (@8teAPi):

Yeah. So, absolutely, like so it's not they he does do the 1000 experiments, but the 1000 experiments, whether that gets him there or not, at one point in the experiment, the quartz tube breaks, that gets them there. They get lucky. Right? So they get they get lucky. And then after that, things proceed pretty quick They isolate they isolate it, and then they they get the crystallization. They start working on the papers. They start on the patents, and they start also trying to figure out the chemical vapor deposition process. They seem to have made some way some headway on the chemical vapor deposition process.

[00:20:06] Ate-a-Pi (@8teAPi):

And then, you know, sometime around September 2021, something start happening. Quant takes a position, sabbatical at, I think, Seoul University at that point. I'm not sure whether that means he's putting more time in the office or not. And then that fast forwards to yeah. Go go ahead, Sunggham.

[00:20:33] Seo Sanghyeon (@sanxiyn):

No. No.

[00:20:33] Alex Volkov (@altryne): No.

[00:20:33] Ate-a-Pi (@8teAPi):

You go ahead. Okay. So that fast forward about March 2023 when basically the international patent has been filed. And Kuan leaves the team at this time. I'm not sure when Kim comes on board. That's not very to me at what point Yum Tuck comes on board.

[00:20:57] Ate-a-Pi (@8teAPi):

So I'm guessing it's after the nature, the nature paper gets dinged in 2020, And and and, you know, the the other thing that strikes me also is that every single person on the team is very aware of every single hoax in superconductors to date. Right? They they they all know the space well, They've seen every single hoax before. They know they know what the hoaxes look like. They know what to look for. They know what diamagmatism is. So I I I don't think yeah.

[00:21:29] Seo Sanghyeon (@sanxiyn):

Go ahead. So the date is So the day before the yesterday, Andrew McCully posted on his Twitter the translation of the Korean paper at Doctor Lloyd. Is that correct? And can can you so so how did you translate and can Can you say something about it?

[00:21:59] Alex Volkov (@altryne): Andrew, I think he's Frank to you. So I can just ring to you. You posted a translated paper also. Right?

[00:22:08] Andrew McCalip (@andrewmccalip):

Yes. Now that was just a machine translation from Google. That was just a very cursory translation.

[00:22:19] Seo Sanghyeon (@sanxiyn):

Okay.

[00:22:19] Ate-a-Pi (@8teAPi):

So in basically, quantity is team in March, and then you have the kind of papers being released, you know, haphazardly. The next the next point that of them is that they had started releasing the papers app as early, like, late last week.

[00:22:42] Alex Volkov (@altryne): And and then and then we have -- And by the way, I think it's it's important to highlight by Kwan, the guy who's no longer affiliated with with QCenter. Like, this this sole endeavor a business venture that's funded for for this for this purpose. Kwan is no longer affiliated with that. We've seen Sankyo posted an interview in Korea from Friday where I think both of the and Kim say that Kwan, the guy who published the first paper, is no longer affiliated.

[00:23:12] Alex Volkov (@altryne): there were some speculation as to maybe the limit of three people on the paper is the limit of the Nobel Prize or 2 or 3 authors. I don't have this confirmed, but this is speculation going around. And it's important to note like, both of them say that the paper was not ready when it was released, and it was released by Juan, the guy who left the first paper. 2 hours later, 2 than 20 minutes later, another paper gets released in the in the same archive with, I wouldn't say, 5 authors. not including Kwan. Right?

[00:23:48] Ate-a-Pi (@8teAPi):

So Lee -- Yeah. And -- The user the the user name is TumTuk team, the the college professor from, you know, Virginia is the username who who pushes the r archive paper at that Yeah.

[00:24:04] Seo Sanghyeon (@sanxiyn):

Chantakim is a big name with the 18 days of 45, and If you look at the paper, there is an error message in Korean saying that Bloomberg could not be found. It is a neutral error message when you did the some of the typesetting wrong.

[00:24:27] Seo Sanghyeon (@sanxiyn):

And You just don't probably see the room temperature, sugar conductor paper with the error deaths that had to bookmark cannot be found if you are following if you are in not in emergency.

[00:24:52] Alex Volkov (@altryne): So so it does feel to us at least from the summary so far that the paper that Quang released has different information than than the second paper, and the second paper feels like it was released in the Harry and included more people that currently work at Q Center, including Hyundai Kim. And Sonja, owner, you this question. You mentioned his h h score or something score. Can can you explain the importance of that score for him talking?

[00:25:20] Seo Sanghyeon (@sanxiyn):

creates someone else to the explanation.

[00:25:24] Ate-a-Pi (@8teAPi):

Okay. So so the h score is, you know, because we have a web web savvy audience here. It's kind of like a page rank for, you know, researchers. It shows you how influential how influential the researcher was, and so a higher score means that more people have been citing your paper.

[00:25:45] Ben (@BenShindel):

Go ahead, Ben. Yeah. More precisely. So, like, an h index of, say, 40 means you have 40 papers that each have 40 citations or more. That's a little tricky to understand. So, like, if I get another paper that has only 30 citations, it won't affect my h index at all. I have to get a 41st paper that has 41 citations to to to make it rise.

[00:26:07] Alex Volkov (@altryne): So I think it's it's safe to say HUNTAKIM, the guy who submitted the second paper, potentially haphazardly. Correct? Like, we're we're we're saying there's 2 hours after the first one. So likely prompted by these events is a well well sighted very well sighted scientist with a very high kind of confidence score. It's not like a random person of the street that decide that there's now a superconductor of room temperature and, you know, verified it.

[00:26:41] Seo Sanghyeon (@sanxiyn):

Okay. Sorry for being side tracked, but I just checked the the motion related to Korean paper or not to talk through it by Andrew. And on the page 5, we clearly said that the quartz tube was destroyed due to internal pressure during rapid cooling of reaction and etcetera. So I think, in fact, nobody really read ready carefully. It is it is just there about the quartz tube once destroyed.

[00:27:19] Ate-a-Pi (@8teAPi):

Yeah. So I think I think it's yeah. Definitely, like, probably the the rest of us are are are not very close readers. of of that paper.

[00:27:29] Seo Sanghyeon (@sanxiyn):

So so We can we can continue on after the upload to the archive.

[00:27:42] Ate-a-Pi (@8teAPi):

Indeed. So okay. So they they they it goes into our our archive, and then all of the events of the last week happen you know, I don't think any of us expected any of the events to happen. So we've all just been kind of, like, following along and seeing what happens next. I had no idea that there was a metallics conference in South Korea, and I I definitely had, like, no idea that you know, one of the authors would show up there, and it gets posted on Twitter. And so and then and then Seung Young points it out on the FM Korea Football message board.

[00:28:20] Ate-a-Pi (@8teAPi):

And so we translate, you know, what the audience reaction was in in in a bad translation to get -- So -- -- whatever message was across.

[00:28:30] Alex Volkov (@altryne): -- mind let me interject here because this is around the that I found out about this. Alex, frozen coffee. Alex, I forgot his nickname. We invited him here. He posted a a very long Twitter thread that got the attention of the algorithm and then boosted of this room template ambin pressure, superconductor paper from Korea. I think he only started talking about the first paper, and then after the second paper also came out. And I think at this point, or somewhere around there. Andrew, you found out about this. What what did you first hear about, you know, Twitter drama around LK 90 Right?

[00:29:08] Alex Volkov (@altryne): And, Andrew, feel free to at least produce you know, introduce yourself officially and BARDA and how you're interacting with this.

[00:29:16] Andrew McCalip (@andrewmccalip):

Yeah. So I was just cruising the Internet at night, and this came across. I think my my Twitter feed And so I I'm incredibly curious. This is something that has been a bit of a a hobby for me. And so I was always interested in superconductors, so it it caught my attention. I'm a mechanical engineer. So full disclosure. I am not a subject matter expert. I am simply an aerospace engineer that has a lot of curiosity and some assets at his disposal.

[00:29:50] Andrew McCalip (@andrewmccalip):

And so reading this paper, it it struck me just the simplicity of of the process. And so I realized that I probably had the ability to replicate with full fidelity, the process that was described in the paper. And so that within about 30 minutes, I I realized I should simply start down this road that Twitter was already picking up at the time.

[00:30:21] Andrew McCalip (@andrewmccalip):

There's some conversations going back and forth and the it was the classic scenario where on every superconductor discussion, there is the same conversation that happens over and over again. And this synthesis appeared so simple that it seemed that the most expedient thing was to simply test it physically. And so my my work is very receptive of of after hours projects. I'm I'm known as the the guy that has really aggressive hobbies, let's say.

[00:30:57] Andrew McCalip (@andrewmccalip):

And so I'm always in the back doing something interesting with materials or automation. So within 30 minutes of reading the paper, I had ticked off orders to various chemical suppliers. I've reached out to overseas vendors. to try to procure a couple of the the elements. And so it was just kind of an offhand comment that I made on Twitter and and then the ball really started rolling, and I realized that everyone wanted to see this this made.

[00:31:32] Andrew McCalip (@andrewmccalip):

And so it was just supposed to be a a a fun little project, but I was really overwhelmed by the the response. Everyone wanted to to see this done. I think there's this incredible curiosity, there's this incredible drive. People wanna see, like, incredible things happen for the the the human race. And so something if this magnitude pops up, everyone's motivated to drop everything and investigate. And I think that's where we're at.

[00:32:08] Alex Volkov (@altryne): And I think you met the algorithm at the right place where folks were excited about the future and think this could bring a lot of changes around the future, and you started saying, hey. You know? Here's a here's a direct approach. Let's try to replicate this. And I I wanna just highlight the fact the the materials involved in creating this. And the process, some folks say and please talk about this. Some folks say that has been an attempt at a hoax, it wouldn't be as simple. They wouldn't have released a simple instruction manual kind of quote, unquote simple that many labs around the work they replicate given the materials and and the right equipment. Right?

[00:32:48] Ely Rabani (@radsci):

So -- Yeah.

[00:32:48] Alex Volkov (@altryne): So -- -- straightforwardness of this potentially shows some stuff.

[00:32:51] Ely Rabani (@radsci):

So this this is a good time for for a PSA. I mean, I know that that Andrew is well aware of this, and and and many of peep of the people who've been following it. But in case anybody who's listening isn't. The these compounds in vapor form at any rate are are highly talked music, and you you have to know lab safety. If you're gonna start trying to experiment with them, you need things like, a glove box and, you know, all kinds of PPE, a fume hood, everything else. Taking risks with this kind of thing is just really not worth it.

[00:33:31] Alex Volkov (@altryne): I I I can't stress that. Absolutely. Don't try this at home.

[00:33:36] Andrew McCalip (@andrewmccalip):

kids definitely. Yeah. Absolutely. There's a lot of chatter in the beginning in the first couple hours about this can be replicated in a garage And, you know, I thought it was interesting. I thought maybe we've got the opportunity to to do it safely. we've got all the right equipment. We've got, you know, the the 1,000,000 of dollars of equipment that support our spacecraft business. that allow us to do some of these things safely. And so I thought Twitter wants to live vicariously through somebody why not do this?

[00:34:12] Andrew McCalip (@andrewmccalip):

I ended up being in sort of an interesting middle ground because I'm not in academia. I'm also not trying to commercialize any part of this tech. really just doing it for fun because it's incredibly interesting. So I've got no skin in the game except for making this work in a transparent manner. and then getting the materials into the hands of the experts.

[00:34:34] Andrew McCalip (@andrewmccalip):

So I thought if we can leverage some of our equipment and some of our, you know, very smart people that we have, to speed this timeline up, I didn't see anybody in the United States being vocal about trying to do replication there are so many stories coming out of other parts of the world that all the labs, there must be thousands of furnaces burning right now trying to replicate this. But I wanted to get material into the hands of some local experts in California.

[00:35:09] Andrew McCalip (@andrewmccalip):

And so that's really our our goal is, you know, can we can we sort of be the face of of the Internet do this experiment in a safe manner and then help advance the science and be sort of a forcing function to to doing this replication.

[00:35:27] Alex Volkov (@altryne): So, Andrew, just before just a a small pause before you continue, I want to ask the other, Andrew, here. The Andrew code, if if you're able to unmute and and and talk us if you're available about the potential reasons why all of Twitter jumped on this. Andrew Kot, you had a thread on room temperature superconductors. About 2 weeks before this, like, almost a permanent is kind of a threat. And could you give us some summary first of all, feel free to introduce yourself, but also some summary of what this means if this replicates, what this means for the world.

[00:36:07] Alex Volkov (@altryne): Applications, you know, give us, like, some excitement of what happens if this is an actual ambient pressure in room temperature superconductor? Andrew? Does not look like Andrew is Oh, hey.

[00:36:33] Andrew Cote (@Andercot):

Sorry. My my audio cut out for a second. I I missed the prompt. Oh, here you are. Let you only -- Sure. Yeah. Thanks. Thanks very much.

[00:36:44] Alex Volkov (@altryne): So so folks so so I I explained to folks your thread about MBN, you know, pressure room temperature superconductors that you've offered, what, 2 weeks before the paper came out. And then suddenly, this dropped. And I wanted you to highlight some of the potential applications of superconductors and give us some of the highlights of what happens in this replicating. This is an actual, you know, real thing.

[00:37:08] Andrew Cote (@Andercot):

Yeah. Sure. So it's kind of a funny thing. Yeah. I put that thread out there 7 weeks before this story broke. You know, just I have worked with this kind of stuff in in a few different areas now, so it's very, you know, superconducting radio frequency cavities are standard technology in accelerator physics to fill these to work in.

[00:37:31] Andrew Cote (@Andercot):

Like, my first job in physics was actually in a condensed matter lab using a a scanning tunneling microscope to look at, you know, electronic structures of potential high temperature superconductors So this has always been sort of like a holy grail of material science, like sort of a holy grail of applied physics. It's one of these properties it's one of these materials where the bulk properties come from its quantum mechanical behavior. And and, you know, when quantum mechanics and its effects escape the realm of the very tiny, it can really manifest as as magical phenomenon at our scale in the world of the kind of the bulk matter or the big stuff.

[00:38:10] Andrew Cote (@Andercot):

So, you know, superconductors are used currently today, You know, it's it's they've reached engineering applicability through decades of continuous refinements and improvements. And and some of the biggest things to think about in what lets these things get used in industrial applications is their ability to superconducts at higher and higher temperatures And, also most also importantly, is to operate at higher and higher background magnetic field strengths. And so the way to think about this is that a superconductor, it's allowing current to move through it with zero resistance, but it also perfectly spells magnetic fields.

[00:38:48] Andrew Cote (@Andercot):

And there's an operating point of these materials where it's basically the current density and the temperature and the magnetic field kind of put the bounds or the performance envelope on the material. So some conductors can carry tons of current, but they can't exist in a very high field. And so, you know, those are hard to make as useful. You can use them for carrying, like, electricity, which is awesome, but often what you really wanna do is generate very strong magnetic fields. So I think maybe the most familiar to the most people here would be, like an MRI machine. Right?

[00:39:27] Andrew Cote (@Andercot):

Magnetic resonance imaging. So the idea there is you're generating very high strength field, and magnetic fields are measured in Tesla, for example. So just for just for context, you know, 3 Tesla is a is a pretty strong field, and that's what is about the strength using an MRI. So, you know, MRIs use these cryogenically cooled magnets, or or they're not I don't think cryogenically cooled. They're actually often just copper, but they do have cooling. But they generate this high strength field, and then, you know, it kind of sets all these little protons in your body spinning and dancing in a little, you know, kind of radiating energy.

[00:40:03] Andrew Cote (@Andercot):

And then you have a pickup coil, which is like an antenna, and the antenna is trying to pick up that energy and kinda reconstruct what's going on in your body. And this is how we can get, like, a really high detailed, high fidelity, three-dimensional image of what's going on inside someone without any invasive surgery. So it's, like, you know, MRIs are a real kind of amazing breakthrough in medical imaging. Superconductors if they could work without cryogenics would really simplify and make cheaper and more available, high resolution, high fidelity, three d images of people's bodies.

[00:40:35] Andrew Cote (@Andercot):

not just for making the magnetic fields, but also for picking up the signal emitted by the protons that get put into motion by the field in the first place. So it's kind of, like, one sort of off the shelf example. I think another one that's kind of under the radar, we don't think about it's not just in carrying electricity without resistance, which is useful for long range, like energy transmission, that kind of stuff. But if you look at the national grid, I mean, only 5, 7 percent of energy total, which is still significant, but it's, you know, single digit percentage ends up, you know, burning as weight You're suddenly muffled.

[00:41:11] Alex Volkov (@altryne): I don't think yeah. You're suddenly a voice like your -- Oh, better.

[00:41:18] Andrew Cote (@Andercot):

Now it's better. Okay. Sorry about that. Yeah. So just gonna say so, you know, National Grid Scale Energy Production. Right? So trans transmitting the energy to its endpoint consumption, there's a bit of waste heat along the way. But what's what's also important to think about is how that energy is produced. It's produced also using high strength magnetic fields. And I was looking into this. There's a a experiment where these guys used sort of more modern high temperature superconducting tape to, you know, retrofit a large DC generator then it had, like, a 36 percent power improvement, right, which is pretty substantial. That's that's a that's a serious win.

[00:41:58] Andrew Cote (@Andercot):

Yeah. So there's there's, you know, sort of thousands of places this stuff could be used that would really just, like you know, it would either greatly improve the performance efficiency, reduce the cost, increase the accessibility of what we think of as, like, high technology like MRIs or particle accelerators. But it would also just decrease the cost of basic things like electricity generation and distribution And that's just the beginning. Right? So, you know, this kind of stuff there's a really good analogy here actually with the transistor, you know, for for years, scientists, then electrical engineers and physicists, they had this idea of a transistor. Right?

[00:42:35] Andrew Cote (@Andercot):

If only we could have some kind of simple, reliable, current model supplier. We could design all these wonderful things. We could design all these different kinds of logic functions and so forth. And so there was this search for the transistor people were searching for something that could do that, and they had anticipated all the places it could be used ahead of time. And it wasn't until at Bell labs, you know, a very kind of funny crossover here. One of the guys that's on the patent for the transistor is John Bardine. and John Bardeen's actually the only guy to win 2 Nobel Prizes. 1 was for the transistor. The other was for the theory of superconductivity, right, which is Barting Cooper Schiffer Theory, BCS.

[00:43:14] Andrew Cote (@Andercot):

So, again, it's one of it's one of those things where, you know, physicists, scientists, engineers kinda thought about this for a long time, realize this be amazing. And there's been this, you know, really complicated random walk through the configuration space of possible materials, right, which is so high dimensional. There's so many things you can construct. So I think it's I'm very optimistic about the field in general. I think one thing to think about with this particular result there's so much artisanal craft and and mastery that goes into producing these materials in a reliable, consistent way You know, science people don't often recognize. It's a lot of art involved too. Right?

[00:43:52] Andrew Cote (@Andercot):

Like like, things that are reduced to expert practice us and know how. And so I'd I'd just be cautious on, you know, jumping to conclusions either on this particular result, if it's if it's valid right now. But, also, if some labs can't fail to reproduce it, it doesn't actually rule it out entirely. I I think there's scientists that have traveled to Korea to work with the original authors. I look closely at that. You know, I'd also you know, I my internal odds are kind of like a 1 in 6 chance, this pans out, and it and it could be big.

[00:44:21] Andrew Cote (@Andercot):

But that doesn't mean that it's the end of the search or the end of the race, and I'm and I'm also optimistic that Getting people to understand what the massive long term and large scale social benefits of this kind of discovery could be could help direct a lot more basic science research towards this field. You know, I think we spend a lot of things on, like, how to make smartphone cameras better and not a lot of things on and not as much as we could spend on things like high temperature superconductors. And this is a final example.

[00:44:48] Andrew Cote (@Andercot):

I mean, so right now, you know, I work as a accelerator engineer, accelerator is a type of magnetic confinement fusion reactor The reason the company I work for can't exist, and and the reason there is this current burn and boom in nuclear fusion, is because we've engineered these high temperature superconductors to work in higher and higher magnetic fields, at at higher and higher temperatures. And and the big economic breakthrough there came when we can have these superconductors that can work at liquid nitrogen temperatures, right, which is 77 kelvin. And it's a lot cheaper to make liquid nitrogen and run that kind of cryogenics than it like liquid helium at, like, 4 Kelvin.

[00:45:24] Andrew Cote (@Andercot):

So, you know, we're already reaping some of the benefits of this sort of tech stack maturing over time. And I think really just getting started in terms of, like, the hunt for promising materials. I mean, I'm hoping this results in positive publicity and more effort, more energy, put into the field. I think if this doesn't pan out as the thing, you know, don't give up hope. Right? I mean, this is a long term game. Science sees by starts and stops. There's no fundamental physics here that's impossible. Right? There's no physical principle that says this can't work. Right? This isn't like a a momentumless or massless propulsion drive like the EM drive.

[00:46:04] Andrew Cote (@Andercot):

isn't, like, superluminal neutrinos. Right? Those things kind of break laws of physics. This is very much in the realm of, yeah, physically possible. seems seems very you know, in my mind, seems likely there could be something out there given the complexity of state space of electronic structures and given how you know, how large that space of exploration can be. And, yeah, so I think I'm just kind of you know, this is a great time to be interested in material science to appreciate basic science research and educating ourselves on on how good the future can be. You know, I think there's a lot of narratives right now in society and cultural in general. that kinda say, like, you know, you know, we we can't solve our way out of our biggest problems today. Right?

[00:46:43] Andrew Cote (@Andercot):

And and I'm very much on the other side of that debate. I think we can. I think it's through efforts like this. I think it's through people like Andrew at Varda that are willing to do stuff in their backyard or their garage or their fact or their their work workplace on their extra time. You know? I mean, this is the kind of this is the the let's build mentality. Right? And so I think we can build our way out of the world's greatest problems, and I its fundamental scientific advances like this discovery could be that that kind of paved the way out of there too. So, yeah, overall, very optimistic.

[00:47:11] Andrew McCalip (@andrewmccalip):

Andrew? That that's incredibly well said. That is an incredibly well balanced viewpoint. So how would you advise people to absorb the the next week of the new cycle? I mean, we're very much on a you know, we're we're back dead. We're back type of hype cycle. So how do you advise people to think about the results that they're seeing knowing that this is a a very difficult thing to replicate when it just because it a negative result is shown in a lab that doesn't mean it's not physically possible.

[00:47:49] Andrew McCalip (@andrewmccalip):

It's very difficult to prove the negative here. So tell us how we should absorb the new cycle coming up in the next few days.

[00:47:59] Ate-a-Pi (@8teAPi):

So I I I I I I might I might say something about that. I think I think this is basically tacit knowledge transfer, and you Kim Kim seems to have been this kind of, like, artisanal, like, you know, experiment list. So you need people to actually sit there in the lab with this guy, and he needs to demonstrate to them. And they need to pick up and and there might be things that he does, which he didn't write down. That that's the like, my my take on it given that He is the experiment list. He's the synthesis on on the team.

[00:48:38] Ate-a-Pi (@8teAPi):

Given that the team seems to have been only, like, 5 or 6 people, is that this guy is the maybe the only person in the world as of, like, you know, 18 months ago. I'm guessing that, you know, he managed to transfer some of that to the JungTux team. So I'm guessing that at at least one more one more team on on earth has this now. And I'm guessing that this knowledge transfer is now happening to a couple more people. So so you need to see this progress maybe 2 or 3 cycles for, like, a bunch of other people to have learned the skill, and then that's when that's when things get interesting.

[00:49:14] Seo Sanghyeon (@sanxiyn):

I mean, you don't really need to replicate to to verify this. There, the the team can just the team has the working samples. they can adjust the samples to the laps around the world.

Hey, the rest of the episode is for paid subscribers to thursdai.

I encourage you to subscribe or upgrade your subscription to access it, there’s almost 2 more hours of in depth conversation, stitching of facts, experts on material science, physics, electrical engineering and MIT folks chiming in. It’s really a great space, around 25K folks have listened to it on twitter so far.

Hi, today’s episode is published on a Friday, it’s been a busy week with at least 4 twitter spaces, countless DMs and research!

OpenAI announces UX updates

* Example prompts: No more staring at a blank page!

* Suggested replies: ChatGPT automatically synthesizes follow up questions. Then you just click a button

* GPT-4 by default: When starting a new chat as a Plus user, ChatGPT will remember your previously selected model!

* 4. Uploading multiple files is now supported in the Code Interpreter beta for all Plus users.

* 5. Stay logged in: You’ll no longer be logged out every 2 weeks and if you do, we have a sweet new welcome page!

* 6. Keyboard shortcuts: Work faster with shortcuts, Try ⌘ (Ctrl) + / to see the complete list.

ThursdAI - I stay up to date so you don’t have to

Alibaba releases Qwen7b

* Trained with high-quality pretraining data. Qwen-7B pretrained on a self-constructed large-scale high-quality dataset of over 2.2 trillion tokens. The dataset includes plain texts and codes, and it covers a wide range of domains, including general domain data and professional domain data.

* Strong performance. In comparison with the models of the similar model size, outperforms the competitors on a series of benchmark datasets, which evaluates natural language understanding, mathematics, coding, etc.

* Better support of languages. New tokenizer, based on a large vocabulary of over 150K tokens, is a more efficient one compared with other tokenizers. It is friendly to many languages, and it is helpful for users to further finetune Qwen-7B for the extension of understanding a certain language.

* Support of 8K Context Length. Both Qwen-7B and Qwen-7B-Chat support the context length of 8K, which allows inputs with long contexts.

* Support of Plugins. Qwen-7B-Chat is trained with plugin-related alignment data, and thus it is capable of using tools, including APIs, models, databases, etc., and it is capable of playing as an agent.

This is an impressive jump in open source capabilities, less than a month after LLaMa 2 release!

GTE-large a new embedding model outperforms OPENAI ada-002

If you’ve used any “chat with your documents” app or built one, or have used a vector database, chances are, you’ve used openAI ada-002, it’s the most common embedding model (that turns text into embeddings for vector similarity search)

This model is ousted by an OpenSource (nee. free) one called GTE-large with improvements on top of ada across most parameters!

OpenOrca 2 preview

Our friends from AlignmentLab including Teknium and LDJ have discussed the release of OpenOrca 2! If you’re interested in the type of finetuning things these guys do, we had a special interview w/ NousResearch on the pod a few weeks ago

OpenOrca tops the charts for the best performing 13B model 👏

Hyper-write releases a personal assistant

You know how much we love agents in ThursdAI, and we’re waiting for this field to materialize and I personally am waiting for an agent to summarize the whole links and screenshots for this summary, and… we’re not there yet! But we’re coming close, and our friends from HyperWrite have released their browser controlling agent on ThursdAI. Talk about a full day of releases!

I absolutely love the marketing trick they used where one of the examples of how it works, is “upvote us on producthunt” and it actually did work for me, and found out that I already upvoted

Superconductor continues

I was absolutely worried that I won’t make it to this thursdAI or won’t know what to talk about because, well, I’ve become a sort of host and information hub and a interviewer of folks about LK-99. Many people around the world seem interested in it’s properties, replication attempts and to understand this new and exciting thing.

We talked about this briefly, but if interests you (and I think it absolutely should) please listen to the below recording.

ThursdAI - See ya next week, don’t forget to subscribe and if you are already subscribed, and get value, upgrading will help me buy the proper equipment to make this a professional endeavor and pay for the AI tools! 🫡

Hey everyone, welcome to yet another ThursdAI update! As always, I’m your host, Alex Volkov, and every week, ThursdAI is a twitter space that has a panel of experts, guests and AI enthusiasts who join to get up to date with the incredible fast pace of AI updates, learn together and listen to subject matter experts on several of the topics.

Pssst, this podcast is now available on Apple, Spotify and everywhere using RSS and a new, long form, raw and uncut, full spaces recording podcast is coming soon!

ThursdAI - Is supported by readers, and I promised my wife I’d ask, if you find this valuable, why not upgrade your subscription so I can keep this going? Get better equipment and produce higher quality shows?

I started noticing that our updates spaces are split into several themes, and figured to start separating the updates to these themes as well, do let me know if the comments if you have feedback or preference or specific things to focus on.

LLMs (Open Source & Proprietary)

This section will include updates pertaining to Large Language Models, proprietary (GPT4 & Claude) and open source ones, APIs and prompting.

Claude 1.2 instant in Anthropic API (source)

Anthropic has released a new version of their Claude Instant, a very very fast model of Claude, with 100K, a very capable model that’s now better at code task, and most of all, very very fast!

Anthropic is also better at giving access to these models, so if you’ve waited in their waitlist for a while, and still don’t have access, DM me (@altryne) and I’ll try to get you API access as a member of ThursdAI community.

WizardLM-70B V1.0 tops OSS charts (source)

WizardLM 70B from WizardLM is now the top dog in open source AI, featuring the same License as LLaMa and much much better code performance than base LLaMa 2, it’s now the top performing code model that’s also does other LLMy things.

Per friend of the pod, and Finetuner extraordinaire Teknium, this is the best HumanEval (coding benchmark) we’ve seen in a LLaMa based open source model 🔥

Also from Teknium btw, a recent evaluation of the Alibaba Qwen 7B model we talked about last ThursdAI, by Teknium, actually showed that LLaMa 7B is a bit better, however, Qwen should also be evaluated on tool selection and agent use, and we’re waiting for those metrics to surface and will update!

Embeddings Embeddings Embeddings

It seems that in OpenSource embeddings, we’re now getting state of the art open source models (read: require no internet access) every week!

In just the last few months: - Microsoft open-sourced E5 - Alibaba open-sourced General Text Embeddings - BAAI open-sourced FlagEmbedding - Jina open-sourced Jina Embeddings

And now, we have a new metric MTEB and a new leaderboard from hugging face (who else?) to always know which model is currently leading the pack. With a new winner from this week! BGE (large, base and small (just 140MB) )

Embedding models are very important for many AI applications, RAG (retrieval augmented generation) products, semantic search and vector DBs, and the faster, smaller and more offline they are, the better the whole field of AI tools we’re going to get, including, much more capable, and offline agents. 🔥

Worth noting that text-ada-002, the OpenAI embedding API is now ranked 13 on the above MTEB leaderboard!

Open Code Interpreter 👏

While we’re on the agents topic, we had the privilege to chat with a new friend of the pod, Shroominic who’s told us about his open source project, called codeinterpreter-api which is an open source implementation of code interpreter. We had a great conversation about this effort, the community push, the ability of this open version to install new packages, access the web, run offline and have multiple open source LLMs that run it, and we expect to hear more as this project develops!

If you’re not familiar with OpenAI Code Interpreter, we’ve talked about it at length when it just came out here and it’s probably the best “AI Agent” that many folks have access to right now.

Deepfakes are upon us!

I want to show you this video and you tell me if you saw this not in an AI newsletter, would you have been able to tell it’s AI generated.

This video was generated automatically, when I applied to the waitlist by HeyGen and then I registered again and tried to get AI Joshua to generate an ultra realistic ThursdAI promo vid haha.

I’ve played with many tools for AI video generation and never saw anything come close to this quality, and can’t wait for this to launch!

While this is a significant update for many folks in terms of how well deepfakes can look (and it is! Just look at it, reflections, HQ, lip movement is perfect, just incredible) this isn’t the only progress data point in this space.

Play.ht announced version 2.0 which sounds incredibly natural, increased model size 10x and dataset to more than 1 million hours of speech across multiple languages, accents, and speaking styles and emotions and claims to have sub 1s latency and fake your voice with a sample of only… 3 seconds! 🤯

So have you and your loved ones chosen a code word to authenticate over the phone? Or switched to a verifiable communication style? While those of us with multiple accents don’t yet have to worry, everyone should stop believing any video or voice sample from now on, it’s just inevitable that all of that will be deepfaked and we should start coming up with ways to authenticate content.

If you made it this far, and any of the above was new/important to you, why not support this pod/newsletter/community? If you’d like to sponsor us more directly, please ping me at altryne [at] gmail.com , I’m also open to consulting, and if you’re a great company, Developer Relations positions :)

Finally, we’ve talked for a whopping 2 hours on the spaces, and that whole conversation can be heard on our Zealous page which has transcripts, AudioGrams of key moments, and space summarizations!

And the Long form space recordings can be added to your podcatcher separately if you’d prefer the “ThursdAI raw feed” by using this RSS link, and will come as it’s own podcast very soon! Thanks to our friends at Zealous

Thank you,

Alex Volkov.

Host ThursdAI - Recaps of the most high signal AI weekly spaces

CEO @ Targum.video

AI Consultant with free slots (Lets Talk)

Hey everyone, this is Alex Volkov, the host of ThursdAI, welcome to yet another recap of yet another incredibly fast past faced week.

I want to start with a ThursdAI update, we now have a new website http://thursdai.news and a new dedicated twitter account @thursdai_pod as we build up the ThursdAI community and brand a bit more.

As always, a reminder that ThursdAI is a weekly X space, newsletter and 2! podcasts, short form (Apple, Spotify) and the unedited long-form spaces recordings (RSS, Zealous page) for those who’d like the nitty gritty details (and are on a long drive somewhere).

Open Source LLMs & Finetuning

Honestly, the speed with which LLaMa 2 finetunes are taking over state of the art performance is staggering. We literally talk about a new model every week that’s topping the LLM Benchmark leaderboard, and it hasn’t even been a month since LLaMa 2 release day 🤯 (July 18 for those who are counting)

Enter Platypus 70B (🔗)

Platypus 70B-instruct is currently the highest ranked open source LLM and other Platypus versions

We’ve had the great pleasure to chat with new friends of the pod Arielle Lee and Cole Hunter (and long time friend of the pod Nataniel Ruiz, co-author of DreamBooth, and StyleDrop which we’ve covered before) about this incredible effort to finetune LLaMa 2, the open dataset they curated and released as part of this effort and how quick and easy it is possible to train (a smaller 13B) version of Platypus (just 5 hours on a single A100 GPU ~= 6$ on Lambda 🤯)

We had a great interview with Garage BAIND the authors of Platypus and we’ll be posting that on a special Sunday episode of ThursdAI so make sure you are subscribed to receive that when it drops.

Open Orca + Platypus = OrctyPus 13B? (🔗)

We’ve told you about OpenOrca just last week, from our friends at @alignment_lab and not only is Platypus is the best performing 70B model, the open source community comes through with an incredible merge and collaborating to bring you the best 13B model, which is a merge between OpenOrca and Platypus.

This 13B model is now very close to the original LLaMa 70B in many of the metrics. LESS THAN A MONTH after the initial open source. It’s quite a remarkable achievement and we salute the whole community for this immense effort 👏 Also, accelerate! 🔥

Join the skunksworks

Speaking of fast moving things, In addition to the above interview, we had a great conversation with folks from so called SkunksWorks OS discord, Namely Far El, Prateek Yadav, Alpay Ariak, Teknium and Alignment Labs, and our recurring guest hosts Yam Peleg and Nisten covered two very exciting community efforts, all happening within the SkunksWorks Discord.

First effort is called MoE, Open mixture of experts, which is an Open Source attempt at replicating the Mixture of Experts model, which is widely attributed to why GPT-4 is so much better than GPT-3.

The second effort is called Ablation studies, which is an effort Teknium is leading to understand once and for all, what is the best, cheapest and most high quality way to finetune open source models, whether it's Qlora or a full finetune or Loras.

If you're interested in any of these, either by helping directly or provide resources such as GPU compute, please join the SkunksWorks discord. They will show you how to participate, even if you don't have prior finetuning knowledge! And we’ll keep you apprised of the results once they release any updates!

Big Co LLMs + API updates

In our Big CO corner, we start with an incredible paper from MetaAi, announcing:

Self-Alignment w/ Backtranslation method + Humpback LLM - MetaAI

Summarized briefly (definitely listen to the full episode and @yampeleg detailed overview of this method) it’s a way for an LLM to be trained on a unsupervised way of creating high quality datasets, for itself! Using not a lot of initial “seed” data from a high quality dataset. Think of it this way, fine-tuning a model requires a lot of “question → response” data in your dataset, and back-translation proposes “response → question” dataset generation, coming up with novel ways of saying “what would a potential instruction be that would make an LLM generate this result”

This results in a model that effectively learns to learn better and create it’s own datasets without humans (well at least human labelers) in the loop.

Here are some more reading material on X for reference.

OpenAI new JS SDK (X link)

OpenAI has partnered with StainlessAPI to released a major new version 4 of their TS/JS SDK with the following incredible DX improvements for AI engineers

* Streaming responses for chat & completions

* Carefully crafted TypeScript types

* Support for ESM, Vercel edge functions, Cloudflare workers, & Deno

* Better file upload API for Whisper, fine-tune files, & DALL·E images

* Improved error handling through automatic retries & error classes

* Increased performance via TCP connection reuse

* Simpler initialization logic

The most exciting part for me is, this is now very easy to get started with AI projects and get streaming on the incredible Cloudflare workers platform (Targum is part of the first Cloudflare workers launchpad but is not affiliated, we’re just superfans 🫶)

Vision & Multi Modality

There’s been some really cool stuff happening in computer vision and multi-modal AI recently. First up, a new method called 3D Gaussian Splatting that shows an incredibly clear and smooth way to generate 3d scenes from just a few images.

Compared to neural radiance fields (NeRFs), Gaussian splatting produces much smoother results without the grainy voxel artifacts NeRFs often have. However, it achieves this improved quality without sacrificing the speed and performance of NeRFs. So Gaussian splatting gives a big boost in realism compared to NeRF renderings, while maintaining real-time speeds in cleaning up those “clouds”

Supervision from Roboflow (and Piotr)

Btw our own friend of the pod and AI Vision expert @skalskiP (who reviewed Gaussian Splatting for us) is also having a crazy ThursdAI week, with their open source library called SuperVision, which is a computer vision toolkit, and is trending #2 on Github 👏

Apple stepping in their Vision (not the headset) Transformer game

Apple has open sourced ml-fastvit, which is their general purpose Vision Transformers model, which they claim runs at ~1ms on mobile devices, including code and pre-train weights available on Github 🔥

This is great to see from Apple ML teams, not only them open sourcing, but also them preparing all of us to the world of spatial computers (Vision Pro coming remember?) and many new Computer Vision heavy apps will be available at those incredible speeds.

This is also great for on device inference running these models in node / on edge (as Friend of the pod @visheratin demonstrated with WebAI)

Additional updates included Nvidia releasing a web playground for NeVa, which is their MLLM (Multimodal LLM, get used to seeing this term everywhere) and you can play with that here ), and Link-Context learning for MLLMs

Agents

OpenAi is also announced that Global Illumination joining OpenAI, that team is CEOd by the creator of Instagram stories algorithm and feed contributor and the team is behind a massive open world minecraft clone. Will we see OpenAI release agents into that world? We know that they are working on agents

A16Z - AI Town (🔗)

Speaking of agents roaming free and interacting, we covered the open sourcing of SmallVille just last week ↴ and now we see a new open source framework called AI Town of running letting agents roam and interact with each other from Andreessen Horowitz AI division.

AI Town (Github) is a web framework, written in TypeScript and is built to run, get customized and run with different LLMs (even Open source ones) in mind and you can see the AI agents running around in a live demo here

This ThursdAI was so packed with great information, that it’s really worth listening to the whole recording, you can do this on our Zealous page, RSS and on twitter (all those links can always be found on thursdai.news )

If you found this valuable, join our community and let your friends know? This is a great way to support us, as well as participate in the discussion on social, tag #thursdAI on anything you feel is worthwhile for us to summarize and

Brief outline for your convenience:

[00:00] Introduction by Alex Volkov[06:00] Discussing the Platypus models and data curation process by Ariel, Cole and Nathaniel[15:00] Merging Platypus with OpenOrca model by Alignment Labs* Combining strengths of Platypus and OpenOrca* Achieving state-of-the-art 13B model[40:00] Mixture of Experts (MOE) models explanation by Prateek and Far El[47:00] Ablation studies on different fine-tuning methods by Teknium

Full transcript is available for our paid subscribers 👇 Why don’t you become one?

Here’s a list of folks and models that appear in this episode please follow all of them on X:

* ThursdAI cohosts - Alex Volkov, Yam Peleg, Nisten Tajiraj

* Garage Baind - Ariel, Cole and Nataniel (platypus-llm.github.io)

* Alignment Lab - Austin, Teknium (Discord server)

* SkunkWorks OS - Far El, Prateek Yadav, Alpay Ariak (Discord server)

* Platypus2-70B-instruct

* Open Orca Platypus 13B

I am recording this on August 18th, which marks the one month birthday of the Lama 2 release from Meta. It was the first commercially licensed large language model of its size and quality, and we want to thank the great folks at MetaAI. Yann LeCun, BigZuck and the whole FAIR team. Thank you guys. It's been an incredible month since it was released.

We saw a Cambrian explosion of open source communities who make this world better, even since Lama 1. For example, LLaMa.Cpp by Georgi Gerganov is such an incredible example of how open source community comes together and this one guy in the weekend Took the open source weights and made it run on CPUs and much, much faster.

Mark Zuckerberg even talked about this, how amazing the open source community has adopted LLAMA, and that Meta is also now adopting many of those techniques and developments back to run their own models cheaper and faster. And so it's been exactly one month since LLAMA 2 was released.

And literally every ThursdAI since then, we have covered a new state of the art open source model all based on Lama 2 that topped the open source model charts on Hugging Face.

Many of these top models were fine tuned by Discord organizations of super smart folks who just like to work together in the open and open source their work.

Many of whom are great friends of the pod.

Nous Research, with whom we've had a special episode a couple of weeks back Teknium1 seems to be part of every orgm Alignment Labs and GarageBaind being the last few folks topping the charts.

I'm very excited not to only bring you an interview with Alignment Labs and GarageBaind, but also to give you a hint of two additional very exciting efforts that are happening in some of these discords.

I also want to highlight how many of those folks do not have data scientist backgrounds. Some of them do. So we had a few PhDs or PhD studies folks, but some of them studied all this at home with the help of GPT 4. And some of them even connected via ThursdAI community and space, which I'm personally very happy about.

So this special episode has two parts. The first part we're going to talk with Ariel. Cole and Natniel, currently known as GarageBaind, get it? bAInd, GarageBaind, because they're doing AI in their garage. I love it.

🔥 Who are now holding the record for the best performing open source model called Platypus2-70B-Instruct.

And then, joining them is Austin from Alignment Labs, the authors of OpenOrca, also a top performing model, will talk about how they've merged and joined forces and trained the best performing 13b model called Open Orca Platypus 13B or Orctypus 13B

This 13b parameters model comes very close to the Base Llama 70b. So, I will say this again, just 1 month after Lama 2 released by the great folks at Meta, we now have a 13 billion parameters model, which is way smaller and cheaper to run that comes very close to the performance benchmarks of a way bigger, very expensive to train and run 70B model.

And I find it incredible. And we've only just started, it's been a month. And so the second part you will hear about two additional efforts, one run by Far El, Prateek and Alpay from the SkunksWorks OS Discord, which is an effort to bring everyone an open source mixture of experts model, and you'll hear about what mixture of experts is.

And another effort run by a friend of the pod Teknium previously a chart topper himself with Nous Hermes models and many others, to figure out which of the fine tuning methods are the most efficient. and fast and cheap to run. You will hear several mentions of LORAs, which stand for Low Rank Adaptation, which are basically methods of keeping the huge weights of LAMA and other models frozen and retrain and fine tune and align some specific parts of it with new data, which is a method we know from Diffusion World.

And it's now applying to the LLM world and showing great promise in how fast, easy, and cheap it is to fine tune these huge models with significantly less hardware costs and time. Specifically, Nataniel Ruiz, the guy who helped Ariel and Cole to train Platypus, the co-author on DreamBooth, StyleDrop and many other diffusion methods, mentioned that it takes around five hours on a single A100 GPU to fine tune the 13B parameter model. That, if you can find an A100 GPU, that's around $10.

That's incredible.

I hope you enjoy listening and learning from these great folks, and please don’t forget to checkout our website at thursdai.news for all the links, socials and podcast feeds.

Brief outline for your convinience:

[00:00] Introduction by Alex Volkov

[06:00] Discussing the Platypus models and data curation process by Ariel, Cole and Nathaniel

[15:00] Merging Platypus with OpenOrca model by Alignment Labs

* Combining strengths of Platypus and OpenOrca

* Achieving state-of-the-art 13B model

[40:00] Mixture of Experts (MOE) models explanation by Prateek and Far El

[47:00] Ablation studies on different fine-tuning methods by Teknium

Full transcript is available for our paid subscribers 👇 Why don’t you become one?

Hey everyone, this week has been incredible (isn’t every week?), and as I’m writing this, I had to pause and go check out breaking news about LLama code which was literally released on ThursdAI as I’m writing the summary! I think Meta deserves their own section in this ThursdAI update 👏

A few reminders before we dive in, we now have a website (thursdai.news) which will have all the links to Apple, Spotify, Full recordings with transcripts and will soon have a calendar you can join to never miss a live space!This whole thing would have been possible without Yam, Nisten, Xenova , VB, Far El, LDJ and other expert speakers from different modalities who join and share their expertise from week to week, and there’s a convenient way to follow all of them now!

TL;DR of all topics covered

* Voice

* Seamless M4T Model from Meta (demo)

* Open Source LLM

* LLaMa2 - code from Meta

* Vision

* IDEFICS - A multi modal text + image model from Hugging face

* AI Art & Diffusion

* 1 year of Stable Diffusion 🎂

* IdeoGram

* Big Co LLMs + API updates

* GPT 3.5 Finetuninng API

* AI Tools & Things

* Cursor IDE

Voice

Seamless M4t - A multi lingual, mutli tasking, multimodality voice model.

To me, the absolute most mindblowing news of this week was Meta open sourcing (not fully, not commercially licensed) SeamlessM4T

This is a multi lingual model that takes speech (and/or text) can generate the following:

* Text

* Speech

* Translated Text

* Translated Speech

In a single model! For comparison sake, I takes a whole pipeline with whisper and other translators in targum.video not to mention much bigger models, and not to mention I don’t actually generate speech!

This incredible news got me giddy and excited so fast, not only because it simplifies and unifies so much of what I do into 1 model, and makes it faster and opens up additional capabilities, but also because I strongly believe in the vision that Language Barriers should not exist and that’s why I built Targum.

Meta apparently also believes in this vision, and gave us an incredible new power unlock that understands 100 languages and does so multilingually without effort.

Language barriers should not exist

Definitely checkout the discussion in the podcast, where VB from the open source audio team on Hugging Face goes in deeper into the exciting implementation details of this model.

Open Source LLMs

🔥 LLaMa Code

We were patient and we got it! Thank you Yann!

Meta releases LLaMa Code, a LlaMa fine-tuned on coding tasks, including “in the middle” completion tasks, which are what copilot does, not just autocompleting code, but taking into account what’s surrounding the code it needs to generate.

Available in 7B, 13B and 34B sizes, the largest model beats GPT3.5 on HumanEval, which is a metric for coding tasks. (you can try it here)

In an interesting move, they also separately release a specific python finetuned versions, for python code specifically.

Additional incredible thing is, it supports 100K context window of code, which is, a LOT of code. However it’s unlikely to be very useful in open source because of the compute required

They also give us instruction fine-tuned versions of these models, and recommend using them, since those are finetuned on being helpful to humans rather than just autocomplete code.

Boasting impressive numbers, this is of course, just the beginning, the open source community of finetuners is salivating! This is what they were waiting for, can they finetune these new models to beat GPT-4? 🤔

Nous update

Friends of the Pod LDJ and Teknium1 are releasing the latest 70B model of their Nous Hermes 2 70B model 👏

* Nous-Puffin-70B

We’re waiting on metrics but it potentially beats chatGPT on a few tasks! Exciting times!

Vision & Multi Modality

IDEFICS - a new 80B model from HuggingFace, was released after a years effort, and is quite quite good. We love vision multimodality here on ThursdAI, we’ve been covering it since we say that GPT-4 demo!

IDEFICS is a an effort by hugging face to create a foundational model for multimodality, and it is currently the only visual language model of this scale (80 billion parameters) that is available in open-access.

It’s made by fusing the vision transformer CLIP-VIT-H-14 and LLaMa 1, I bet LLaMa 2 is coming soon as well!

And the best thing, it’s openly available and you can use it in your code with hugging face transformers library!

It’s not perfect of course, and can hallucinate quite a bit, but it’s quite remarkable that we get these models weekly now, and this is just the start!

AI Art & Diffusion

Stable Diffusion is 1 year old

Has it been a year? wow, for me, personally, stable diffusion is what started this whole AI fever dream. SD was the first model I actually ran on my own GPU, the first model I learned how to.. run, and use without relying on APIs. It made me way more comfortable with juggling models, learning what weights were, and we’ll here we are :) I now host a podcast and have a newsletter and I’m part of a community of folks who do the same, train models, discuss AI engineer topics and teach others!

Huge thank you to Emad, Stability AI team, my friends there, and everyone else who worked hard on this.

Hard to imagine how crazy of a pace we’ve been on since the first SD1.4 release, and how incredibly realistic the images are now compared to what we got then and got excited about!

🎂

IdeaoGram joins the AI art race

IdeoGram - new text to image from ex googlers (announcement) is the new kid on the block, not open source (unless I missed it) it boasts significant text capabilities, and really great quality of imagery. It also has a remix ability, and is availble from the web, unlike… MidJourney!

Big Co LLMs + API updates

Open AI pairs with ScaleAI to let enterprises finetune and run finetuned GPT3.4 models!

This is an interesting time for OpenAI to dive into fine-tuning, as open source models inch closer and closer to GPT3.5 on several metrics with each week.

Reminder, if you finetune a GPT3.5 model ,you need to provide your own data to OpenAI but then also you have to pay them for essentially hosting a model just for you, which means it’s not going to be cheap.

Use as much prompting as humanly possible before you consider doing the above fine-tuning and you may be able to solve your task much better and cheaper.

Agents

The most interesting thing to me in the world of agents actually came from an IDE!

I installed Cursor, the new AI infused VsCode clone, imported my vscode settings, and off we went! It can use your own GPT-4 keys if you don’t want to send them our code or pay, it embeds your whole repo for easy import and code understand and does so much more, like adding a button to every error in console to “debug” and has an “new AI project” feature, which builds you a template just by typing a few words!

Our friends Alessio and Swyx have interviewed the founder of Cursor on their podcast, a strong recommendation to check that episode out!

After using Cursor for just a few days, I don’t want to go back to VSCode and even consider … maybe pausing my copilot subscription 🤯

That’s all for today folks! I wish you all a great week, and we’ll see you in the next ThursdAI 🫡

Thank you for reading ThursdAI - Recaps of the most high signal AI weekly spaces. This post is public so feel free to share it with a friend? Let’s get to 1K readers 🔥

Hey ya’ll, welcome to yet another ThursdAI, this is Alex coming at you every ThursdAI, including a live recording this time!

Which was incredible, we chatted about Falcon 180B,had a great interview in the end with 3 authors of the YaRN scaling paper and LLongMa 128K context, had 3 breaking news! in the middle, MOJO🔥 has been released and Adept released a LLaMa comparable OSS model (and friend of the pod) @reach_vb showed an open ASR leaderboard on hugging face! We also covered an incredible tiny model called StarCoder 1B that was finetuned by friend of the pod (who joined the space to talk to us about it!)

As always, you can listen to the whole 3 hour long form conversation (raw, unedited) on our Zealous page (and add it to your podcatcher via this RSS) and this short-form pod is available on Apple, Spotify and everywhere.

ThursdAI - Hey, if you enjoy these, how about subscribing for real? Would love to do this full time! Every paid subscriber is like a dear friend 🧡

TL;DR of all topics covered

* Open Source LLM

* Falcon 180B announced by TIIUAE (Announcement, Demo)

* YaRN scaling paper - scaling LlaMa to 128K context (link)

* OpenHermes-13B from @teknium1 (link)

* Persimmon-8B from Adept.AI (link)

* Starcoder-1B-sft from @abacaj (link)

* Big Co LLMs + API updates

* OpenAI first ever Dev conference (link)

* Claude announces a $20/mo Claude Pro tier (link)

* Modular releases Mojo🔥 with 68,000x improvement over python (Link)

* Vision

* Real time deepfake with FaceFusion (link)

* HeyGen released AI avatars and AI video translation with lipsync (link, translation announcement)

* Voice

* Open ASR (automatic speech recognition) leaderboard from HuggingFace (link)

* Tools

* LangChain Hub (re) launched

* Open Interpreter (Announcement, Github)

Open Source LLM

🦅 Falcon 180B - The largest open source LLM to date (Announcement, Demo)

The folks at the “Technology Innovation Institute” have open sourced the huge Falcon 180B, and have put it up on Hugging Face. Having previously open sourced Falcon 40B, the folks from TIIUAE have given us a huge model that beats (base) LLaMa 2 on several evaluations, if just slightly by a few percentages points.

It’s huge, was trained on 3.5 trillion tokens and weights above 100GB as a file and requires 400GB for inference.

Some folks were not as impressed with Falcon performance, given it’s parameter size is 2.5 those of LLaMa 2 (and likely it took a longer time to train) but the relative benchmarks is just a few percentages higher than LLaMa. It also boasts an embarrassingly low context window of just 2K tokens, and code was just 5% of it’s dataset, even though we already know that more code in the dataset, makes the models smarter!

Georgi Gerganov is already running this model on his M2 Ultra because he’s the Goat, and co-host of ThursdAI spaces, Nisten, was able to run this model with CPU-only and with just 4GB of ram 🤯 We’re waiting for Nisten to post a Github on how to run this monsterous model on just CPU, because it’s incredible!

However, given the Apache2 license and the fine-tuning community excitement about improving these open models, it’s an incredible feat. and we’re very happy that this was released!

The complete open sourcing also matters in terms of geopolitics, this model was developed in the UAE, while in the US, the export of A100 GPUs was banned to the middle easy, and folks are talking about regulating foundational models, and this release, size and parameter model that’s coming out of the United Arab Emirates, for free, is going to definitely add to the discussion wether to regulate AI, open source and fine-tuning huge models!

YaRN scaling LLaMa to 128K context window

Last week, just in time for ThursdAI, we posted about the release of Yarn-Llama-2-13b-128k, a whopping 32x improvement in context window size on top of the base LLaMa from the folks at Nous Research, Enrico Shippole, @theemozilla with the help of Eluether AI.

This week, they released the YaRN: Efficient Context Window Extension of Large Language Models paper which uses Rotary Position Embeddings to stretch the context windows of transformer attention based LLMs significantly.

We had friends of the pod Enrico Shippole, theemozilla (Jeff) and Bowen Peng on the twitter space and an special interview with them will be released on Sunday, if you’re interested in scaling and stretching context windows work, definitely subscribe for that episode, it was incredible!

It’s great to see that their work is already applied into several places, including CodeLLaMa (which was released with 16K - 100K context) and the problem is now compute, basically, context windows can be stretched, and the models are able to generalize from smaller datasets, such that the next models are predicted to be released with infinite amount of context window, and it’ll depend on your hardware memory requirements.

Persimmon-8B from AdeptAI (announcement, github)

AdeptAI, the company behind Act-1, a foundational model for AI Agent that does browser driving, and has a few co-founders that are the original transformers paper authors, have dropped a ThursdAI surprise, a fresh (read, not a LLaMa clone) model!

Releasing an completely open source model called Persimmon-8B, with a full Apache 2 license, 16K context window (using custom RoPE scaling methods) and some interesting inference speedups with C++.

A very interesting 8B model that can fit on most consumer hardware, with additional tricks and a huge context window, is definitely welcome!

Additional interesting point is, they have 70K unused embeddings for multimodal extensions! Can’t wait to see what’s that about!

Starcoder-1B-sft - tiny model that’s great at code

Anton Bacaj (@abacaj) has finetuned StarCoder, to achieve some incredible results, for such a tiny model! Remember the first item, a whopping 180B parameter Falcon? We’ll, this is just 1B parameters model, finetuned on 65K sampled dataset of code, that’s outperforming Falcon, LLaMa2, Palm-2 (and Persimmon) on coding tasks, and runs on your device, so fast, that it’s hard to read!

It boasts an incredible 39% on HumanEval task and 31% on MBPP! (Anton reran and updated the MBPP score later) and can run locally. Friend of the pod Xenova has already ported this model to transformers.js and it’ll soon run in your browser!

OpenHermes-13B from @teknium1 (link)

Our friend Teknium1 (who we’ve interviewed a few weeks ago) releases OpenHermes on top of LLaMa2, but this time it’s a completely open model and datasets, marking this the first time that Hermes models have been open!

OpenHermes was trained on 242,000 entries of primarily GPT-4 generated data, from open datasets across the AI landscape, including:

* GPTeacher - General Instruct,

* Roleplay v1, Roleplay v2, and Code Instruct Datasets, by Teknium

* WizardLM (v1, evol_instruct 70k), by WizardLM

* Team/nlpxucan Airoboros GPT-4 (v1.0), by JonDurbin

* Camel-AI's domain expert datasets, by the Camel-AI Team

* CodeAlpaca, by Sahil2801

* GPT4-LLM and

* Unnatural Instructions, by Microsoft

Check it out folks!

Big Co LLM + API updates

Modular finally ships Mojo 🔥 (Announcement)

I just knew it, that Mojo will finally be shipped during ThursdAI, and in fact, this was a great #BreakingNews moment on twitter spaces!

Modular, and it’s co-founder Chris Lattner (author of LLVM, MLIR, Swift and many other things) have finally released their Mojo 🔥 language, for AI.

Mojo 🔥 is like Python++, includes strong types, full interoperability with python ecosystem but is able to run basic vanilla python, and has so so much more in it, but the main thing Modular is claiming is a whopping 68,000x improvement over vanilla python!

You didn’t misread this, 68,000 improvement, when using all the Modular inference compilers, and Mojo virtualization tricks and compilation improvements. It’s incredible.

The beauty of Mojo is that it meets developers where they are and allows them to adopt new features to achieve high performance gradually. By combining the best of dynamic and static languages, Mojo can deliver performance up to 68,000 times faster than Python today. That's quite a leap! If you want to delve deeper into Mojo's origin story, you can find more information in their documentation. But for now, let me highlight a few key benefits that Mojo offers:

Firstly, Mojo allows you to write everything in one language, merging the usability of Python with the systems programming features that typically require developers to rely on C, C++, or CUDA. This means that both research and deployment teams can work within a common codebase, streamlining the workflow from research to production.

Secondly, Mojo unlocks Python's performance potential. While Python is widely used, it may not be the best tool for high-performance or specialized hardware tasks. However, Mojo bridges that gap by enabling high performance on CPUs and providing support for exotic accelerators like GPUs and ASICs. With Mojo, you can achieve performance levels on par with C++ and CUDA.

Thirdly, and this is a big one, Mojo seamlessly integrates with the entire Python ecosystem. You can leverage the extensive library collection available in Python while making use of Mojo's features and performance benefits. This means you can easily combine libraries like NumPy and Matplotlib with your Mojo code – talk about flexibility!

Finally, Mojo allows you to upgrade your AI workloads effortlessly. By tightly integrating with the Modular AI Engine, Mojo empowers you to extend your AI workloads with custom operations. This includes pre-processing and post-processing operations, as well as high-performance mathematical algorithms. You can even integrate kernel fusion, graph rewrites, shape functions, and more. Mojo is all about expanding the possibilities!

Mojo’s playground has been around since May and I have a deep dive here but you should really watch for over 3 hours on everything from Why they chose to be a python superset, to why he thinks the community will pick it up, it’s an incredible watch and will make you excited about Mojo!

WebGPU ships with support for FP16 in Chromium

Chrome has shipped with WebGPU back in April of 23’, after years of development, it allows high performance 3D graphics (and of course, transformers inference) in the browser and on the web!

However, for inference of models, GPU access is not enough, you also need to be able to run smaller models. Well, one way to make models smaller is to run them in fp16 format. Essentially cutting the precision of the weights numbers by half, we can use much smaller (read compressed) models with a slight loss in accuracy.

Friends of the pod Nisten and Xenova (transformers.js author) have given us an update that a new, updated fp16 support has shipped in nightly of chromium, allowing for much much smaller models to be run on clientside!

OpenAI first dev conference (Announcement)

OpenAI has announced their first developer focused conference, to happen in SF during November 6th!

In person only (with the keynote being streamed to all) and they also said that they won’t do any model announcement like GPT-5 😂

But we'll all expect at least a few API updates!

Vision

FaceFusion 1.1.0 - a deepfake faceswapper (Announcement, Github)

We all know deepfakes are here, I mean, don’t we? But did you know that it’s now super easy to face swap your face into an image or a video?

FaceFusion does just that, an incredibly fast way to deepfake someone’s face into an image or a video with a few clicks, works on CPU (I couldn’t make it work on GPU but it’s possible) and shows incredible results!

Enjoy Steve Buschemi dance around as Harry Styles? 3 clicks and 10 minutes and you get this 🔥

Friend of the pod CocktailPeanut, has made it incredible easy to install with just 1 click with his pinokio.computer app, which I use and love!

Facefusion also has a webcam mode that is able to deepfake any image onto a webcam stream for a lot of fun on zoom calls! (which I wasn’t able to test for some reason)

HeyGen launches their deep AI face creator

Many of us used 11Labs to clone voices, but what if you can clone a voice AND an image of a person? With just 2 minutes of their recording?

That’s what HeyGen are claiming to be able to do, and we’ve previously reported that their incredible realistic AI avatar generation from videos/images + voice really blew us away.

Heygen just launched their service and you can sign up and get a few minutes for free, here’s a sample (with the CEO avatar, they couldn’t make my own for some launch day errors)

The video you see on top of just that, the CEO of HeyGen, thanking you for reading this weeks ThursdAI!

Voice

ASR leaderboard + New top ASR model from Nvidia

I love doing ThursdAI, and one of the things I love most, is folks sending me stuff they worked on, and then coming to ThursdAI to chat about it.

Friend of the pod Vaibhav (VB) Srivastav, who’s an incredible dev rel at HuggingFace, focusing on Audio, has shipped a new Open-ASR (automatic speech recognition) leaderboard on huggingface!

Showing the top ASR models like Whisper and a new comer, Nvidia FastConformer, which I didn’t even know existed, and now it’s topping Whisper for english speech to text tasks!

HuggingFace leaderboards like these are definitely a boon for the Open Source industry as they allow all of us to easily select open source models, but also allow the open source community to start racing towards the top, while we all benefit!

Tools

Open Interpreter (Announcement, Github)

One tool that I’ve used this week, and is incredible, is OpenInterpreter from @heyitskillian

It’s incredibly easy to install and run, and behaves like OpenAI Code Interpreter (renamed to Advanced Data Analytics) but on your computer, and is able to do things like control your apps, lower volume, edit images/files and tons more

pip install open-interpreter

And that’s it!

Give it a try (and you have to approve each command that it runs)

It’s a great agent, and hopefully we’ll get Killian to chat with us about it on next ThursdAI!

LangChain hub has launched (link)

If you’re into LangChain, and even if you aren’t, it’s undeniable the weight LangChain has in the ai engineer industry! They have a connector for everything, tons of folks use them, and they have raised a bunch of funding.

They have just launched their new LangChain Hub and it’s exciting! Many folks are sharing their best prompts on there, and ways to work with langchain, with upvotes and sharable links!

Also, worth nothing that our friends swyx and Alessio from Latent Space have recently released an episode with Harrison on Latent space, and it’s WELL worth listening (and reading) as swyx did a deep dive into Landchain, it’s nay-sayers and everything in between!

Check it out below :

Thank you, see you next time (with some incredible personal news I’ll have to share)

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Happy Sunday everyone, I am very excited to bring you this interview with the folks who took LLaMa 2 and made it LLoooooongMa!

Extending LLaMa 2 context window from 4,000 to a whopping 128,000 tokens (Yarn-Llama-2-13b-128k on Hugging Face), these guys also came up with a paper called YaRN (Efficient Context Window Extension of Large Language Models) and showed that YaRN is not only requires 10x less tokens to create these long contexts, but also 2.5x less training steps!

And, the models generalize so there’s now no need to collect extremely long sequences (think books length sequences) for the models to understand those context lengths.

I have decided also to do something different (which took me half of Sunday so I can’t promise and am not committing to this format, but for the premium subscribers, you can now watch this interview with running Karaoke style subtitles and improved audio! This will be uploaded to Youtube in a week but aren’t you glad you subscribed and is getting this first?)

Here’s a teaser preview:

And here’s the chapter for your convenience (the only thing that’s ai generated 😂)

0:00 - Introduction

3:08 - Discussion of extending LLAMA2's context length from 4,000 tokens to 128,000 tokens using the YaRN method

8:23 - Explanation of rope scaling for positional encodings in transformers

13:21 - How the rope scaling idea allows for longer context through positional interpolation

18:51 - Using in-context learning to train models on shorter sequences but still handle long contexts

25:18 - Sourcing long-form data like books to train 128k token models

31:21 - Whether future models will natively support longer contexts

37:33 - New model from Adept with 16k context using rope scaling

42:46 - Attention is quadratic - need better algorithms to make long context usable

49:39 - Open source community pushing state of the art alongside big labs

52:34 - Closing thoughts

As always, full (manually edited) transcription (and this time a special video version!) is reserved for the premium subscribers, I promise it’ll be worth it, so why not .. y’know? skip a cup of coffee from SB and support ThursdAI?

Hey, welcome to yet another ThursdAI 🫡

This episode is special for several reasons, one of which, I shared a personal life update (got to listen to the episode to hear 😉) but also, this is the first time I took the mountainous challenge of fixing, editing and “video-fying” (is that a word?) our whole live recording! All 3 hours of it, were condensed, sliced, sound improved (x audio quality is really dogshit) and uploaded for your convenience. Please let me know what you think!

Premium folks get access to the full podcast in audiogram format, and a full transcription with timestamps and speakers, here’s a sneak preview of how that looks, why not subscribe? 😮

TL;DR of all topics covered

* Open Source LLM

* Microsoft Phi 1.5 - a tiny model that beats other 7B models (with a twist?) (Paper, Model)

* Baichuan 7B / 13B - a bilingual (cn/en) model with highly crafted approach to training (Paper, Github)

* Big Co LLMs + API updates

* Nothing major this week

* Voice & Audio

* Stable Audio 🎶 - A new music generation model from Stability AI. (Website)

* Coqui XTTS - an open source multilingual text to speech for training and generating a cloned voice (Github, HuggingFace)

* AI Art & Diffusion

* Würstchen v2 - A new super quick 1024 diffusion model (Announcement, Demo, Github)

* DiffBIR - Towards Blind Image Restoration with Generative Diffusion Prior (Annoucement, Demo, Github)

* Tools

* Nougat from Meta - open-source OCR model that accurately scans books with heavy math/scientific notations (Announcement, Github, Paper)

* GPT4All Vulkan from Nomic - Run LLMs on ANY consumer GPUs, not just NVIDIA (Announcement)

* Nisten’s AI ISO disk - Announcement

And here are timestamps and chapter/discussion topics for your convenience:

[00:05:56] Phi 1.5 - 1.3B parameter model that closely matches Falcon & LLaMa 7B

[00:09:08] Potential Data Contamination with Phi 1.5

[00:10:11] Data Contamination unconfirmed

[00:12:59] Tiny models are all the rage lately

[00:16:23] Synthetic Dataset for Phi

[00:18:37] Are we going to run out of training data?

[00:20:31] Breaking News - Nougat - OCR from Meta

[00:23:12] Nisten - AI ISO disk

[00:29:08] Baichuan 7B - an immaculate Chinese model

[00:36:16] Unique Loss Terms

[00:38:37] Baichuan ByLingual and MultiLingual dataset

[00:39:30] Finetunes of Baichuan

[00:42:28] Philosophical questions in the dataset

[00:45:21] Let's think step by step

[00:48:17] Is breath related text in the original dataset?

[00:50:27] Counterintuitive prompting for models with no breath

[00:55:36] Idea spaces

[00:59:59] Alex - Life update about ThursdAI

[01:04:30] Stable Audio from Stability AI

[01:17:23] GPT4ALL Vulkan

[01:19:37] Coqui.ai releases XTTS - an open source TTS - interview With Josh Meyer

[01:30:40] Summary

Here’s a full video of the pod, and a full transcription, and as always, 🧡 thank you for bring a paid subscriber, this really gives me the energy to keep going, get better guests, release dope podcast content, and have 3 hours spaces and then spend 7 hours editing 🔥

Hey! Welcome to this special ThursdAI Sunday episode. Today I'm excited to share my interview with Killian Lucas, the creator of Open Interpreter - an incredible new open source project that lets you run code via AI models like GPT-4 or local models like Llama on your own machine.

Just a quick note, that while this episode is provided for free, premium subscribers enjoy the full write up including my examples of using Open Interpreter, the complete (manually edited) transcript and a video form of the pod for easier viewing, search, highlights and more. Here’s a trailer of that in case you consider subscribing

If you haven’t caught up with GPT-4 Code Interpreter yet (now renamed to Advanced Data Analytics), I joined Simon Willison and swyx when it first launched and we had a deep dive about it on Latent Space and even at the day of the release, we were already noticing a major restricting factor, Code Interpreter is amazing, but doesn’t have internet access, and can’t install new packages, or use new tools.

An additional thing we immediately noticed was, the surface area of “what it can do” is vast, given it can write arbitrary code per request, it was very interesting to hear what other folks are using it for for inspiration, and “imagination unlock”.

I started a hashtag called #codeinterpreterCan and have since documented many interesting use cases, like comitting to git, running a vector DB, convert audio & video to different formats, plot wind rose diagrams, run whisper and so much more.

I personally have all but switched to Code Interpreter (ADA) as my main chatGPT tab, and it’s currently the reason I’m still paying the 20 bucks!

Enter, Open interpreter

Just a week after open sourcing Open Interpreter, it already has over 20,000 stars on GitHub and a huge following. You can follow Killian on Twitter and check out the Open Interpreter GitHub repo to learn more.

Installing is as easy as pip install open-interpreter. (but do make sure to install and run it inside a venv or a conda env, trust me!)

And then, you just.. ask for stuff! (and sometimes ask again as you’ll see in the below usage video)

Specifically, highlighted in the incredible launch video, if you’re using a mac, Code Interpreter can write and run AppleScript, which can run and control most of the native apps and settings on your mac.

Here’s a quick example I recorded while writing this post up, where I ask Open Interpreter to switch system to Dark mode, then I use it to actually help me extract all the chapters for this interview and cut a trailer together!

Hey dear ThursdAI friends, as always I’m very excited to bring you this edition of ThursdAI, September 21st, which is packed full of goodness updates, great conversations with experts, breaking AI news and not 1 but 2 interviews

ThursdAI - hey, psst, if you got here from X, dont’ worry, I don’t spam, but def. subscribe, you’ll be the coolest most up to date AI person you know!

TL;DR of all topics covered

* AI Art & Diffusion

* 🖼️ DALL-E 3 - High quality art, with a built in brain (Announcement, Comparison to MJ)

* Microsoft - Bing will have DALL-E 3 for free (Link)

* Big Co LLMs + API updates

* Microsoft - Windows Copilot 🔥 (Announcement, Demo)

* OpenAI - GPT3.5 instruct (Link)

* OpenAI - Finetuning UI (and finetuning your finetunes) (Annoucement, Link)

* Google - Bard has extensions (twitter thread, video)

* Open Source LLM

* Glaive-coder-7B (Announcement, Model, Arena)

* Yann Lecun testimony in front of US senate (Opening Statement, Thread)

* Vision

* Leak : OpenAI GPT4 Vision is coming soon + Gobi multimodal? (source)

* Tools & Prompts

* Chain of Density - a great summarizer prompt technique (Link, Paper, Playground)

* Cardinal - AI infused product backlog (ProductHunt)

* Glaive Arena - (link)

AI Art + Diffusion

DALL-E 3 - High quality art, with a built in brain

DALL-E 2 was the reason I went hard into everything AI, I have a condition called Aphantasia, and when I learned that AI tools can help me regain a part of my brain that’s missing, I was in complete AWE. My first “AI” project was a chrome extension that injects prompts into DALL-E UI to help with prompt engineering.

Well, now not only is my extension no longer needed, prompt engineering for AI art itself may die a slow death with DALL-E 3, which is going to be integrated into chatGPT interface, and chatGPT will be able to help you… chat with your creation, ask for modifications, alternative styles, and suggest different art directions!

In addition to this incredible new interface, which I think is going to change the whole AI art field, the images are of mind-blowing quality, coherence of objects and scene elements is top notch, and the ability to tweak tiny detail really shines!

Additional thing they really fixed is hands and text! Get ready for SO many memes coming at you!

Btw, I created a conversational generation bot in my telegram chatGPT bot (before there was an API with stability diffusion and I can only remember how addicting this was!) and so did my friends from Krea :) so y’know… where’s our free dall-e credits OpenAI? 🤔

Just kidding, an additional awesome thing that now, DALL-E will be integrated into chatGPT plus subscription (and enterprise) and will refuse to generate any living artists art, and has a very very strong bias towards “clean” imagery.

I wonder how fast will it come to an API, but this is incredible news!

P.S - if you don’t want to pay for chatGPT, apparently DALL-E 3 conversational is already being rolled out as a free offering for Bing Chat 👀 Only for a certain percentage of users, but will be free for everyone going forward!

Big Co LLM + API updates

Copilot, no longer just for code?

Microsoft has announced some breaking news on #thursdai, where they confirmed that Copilot is now a piece of the new windows, and will live just a shortcut away from many many people. I think this is absolutely revolutionary, as just last week we chatted with Killian from Open Interpreter and having an LLM run things on my machine was one of the main reasons I was really excited about it!

And now we have a full on, baked AI agent, inside the worlds most popular operating system, running for free, for all mom and pop windows computers out there, with just a shortcut away!

Copilot will be a native part of many apps, not only windows, here’s an example of a powerpoint copilot!

As we chatted on the pod, this will put AI into the hands of so so many people for whom opening the chatGPT interface is beyond them, and I find it incredibly exciting development! (I will not be switching to windows for it tho, will you?)

Btw, shoutout to Mikhail Parakhin who lead the BingChat integration and is now in charge of the whole windows division! It shows how much dedication to AI Microsoft is showing and it really seems that they don’t want to “miss” this revolution like they did with mobile!

OpenAI releases GPT 3.5 instruct turbo!

For many of us, who used GPT3 APIs before it was cool (who has the 43 character API key 🙋‍♂️) we remember the “instruct” models where all the rage, and then OpenAI basically told everyone to switch to the much faster and more RLHFd chat interfaces.

Well now, they brought GPT3.5 back, with instruct and turbo mode, it’s no longer a chat, it’s a completion model, that is apparently much better at chess?

An additional interesting thing is, it includes logprobs in the response, so you can actually build much more interesting software (by asking for several responses and then looking at the log probabilities), for example, if you’re asking the model for a multiple choice answer to a question, you can rank the answers based on logprobs!

Listen to the pod, Raunak explains this really well!

FineTune your finetunes

OpenAI also released a UI for finetuning GPT3.5 and upped the number of concurrent finetunes to 3, and now, you can finetune your finetunes!

So you can continue finetuning already finetuned models!

Bard extensions are like chatGPT plugins but more native.

While we wait for Gemini (cmon google, just drop it!) the multi modal upcoming incredible LLM that will beat GPT-4 allegedly, Google is shoving new unbacked features into Bard (remember Bard? It’s like the 5th most used AI assistant!)

You can now opt in, and @ mention stuff like Gmail, Youtube, Drive and many more Google services and Bard will connect to them, do a search (not a vector search apparently, just a keyword search) and will show you results (or summarize your documents) inside Bard interface.

The @ ui is really cool, and reminded me of Cursor (where you can @ different files or documentation) but in practice, from my 2 checks, it really didn’t work at all and was worse than just a keyword search.

Open Source LLM

Glaive-coder-7B reaches an incredible 63% on human eval

Friends of the pod Anton Bacaj and Sahil Chaudhary have open sourced a beast of a coder model Glaive-coder-7B, with just 7B parameters, this model achieves an enormous 63% on HumanEval@1, which is higher than LLaMa 2, Code LLaMa and even GPT 3.5 (based on technical reports) at just a tiny 7B parameters 🔥 (table from code-llama released for reference, the table is now meaningless 😂)

Yann Lecun testimony in front of US senate

Look, we get it, the meeting of the CEOs (and Clem from HuggingFace) made more waves, especially on this huge table, who wasn’t there, Elon, Bill Gates, Sundar, Satya, Zuck, IBM, Sam Altman

But IMO the real deal government AI thing was done by Yann Lecun, chief scientist at Meta AI, who came in hot, with very pro open source opening statements, and was very patient with the very surprised senators on the committee. Opening statement is worth watching in full (I transcribed it with Targum cause… duh) and Yann actually retweeted! 🫶

Here’s a little taste, where Yann is saying, literally “make progress as fast as we can” 🙇‍♂️

He was also asked about, what happens if US over-restricts open source AI, and our adversaries will … not? Will we be at a disadvantage? Good questions senators, I like this thinking, more of this please.

Vision

Gobi and GPT4-Vision are incoming to beat Gemini to the punch?

According to The Information, OpenAi is gearing up to give us the vision model of GPT-4 due to the hinted upcoming release of Gemini, a multi modal model from Google (that’s also rumored to be released very soon, I’m sure they will release this on next ThursdAI, or the one after that!)

It seems to be the case for both DALL-E 3 and the leak about GPT-4 Vision, because apparently Gemini is multi modal on the input (can take images and text) AND the output (can generate text and images) and OpenAI maybe wants to get ahead of that.

We’ve seen images of GPT-4 Vision in the chatGPT UI that were leaked, so it’s only a matter of time.

The most interesting thing from this leak was the model codenamed GOBI, which is going to be a “true” multimodal model, unlike GPT-4 vision.

Here’s an explanation of the difference from Yam Peleg , ThursdAI expert on everything language models!

Voice

Honestly, nothing major happened with voice since last week 👀

Tools

Chain of Density

The Salesforce AI team has developed a new technique for improving text summarization with large language models. Called Chain of Density (CoD), this prompting method allows users to incrementally increase the informational density of a summary.

The key insight is balancing the right amount of details and main ideas when summarizing text. With CoD, you can prompt the model to add more detail until an optimal summary is reached. This gives more control over the summary output.

The Salesforce researchers tested CoD against vanilla GPT summaries in a human preference study. The results showed people preferred the CoD versions, demonstrating the effectiveness of this approach.

Overall, the Salesforce AI team has introduced an innovative way to enhance text summarization with large language models. By tuning the density of the output, CoD prompts can produce higher quality summaries. It will be exciting to see where they take this promising technique in the future.

RememberAll - extend your LLM context with a proxy

We had Raunak from rememberAll on the pod this week, and that interview is probably coming on Sunday, but wanted to include this in tools as it’s super cool.

Basically with 2 lines of code change, you can send your API calls through RememberAll proxy, and they will extract the key information, and embed and store it in a vectorDB for you, and then inject it back on responses.

Super clever way to extend memory, here’s a preview from Raunak (demo) and a more full interview is coming soon!

Cardinal has launched on ProductHunt, from my friends Wiz and Mor (link)

Quick friendly plug, Wix and Mor are friends of mine and they have just launched Cardinal, an AI infused product backlog, that extracts features, discussion about feature requests, and more, from customer feedback, from tons of sources.

Go give them a try, if you’re looking to make your product backlog work better, it’s really really slick!

Hey, if you arrived here, do me a quick favor? Send me a DM with this emoji 🥔 , and then share this newsletter with 1 friend who like you, loves AI?

Thanks, I expect many potatoes in my inbox! See you next ThursdAI 🫡

Here’s the full transcript (no video this time, I’m finishing this up at 10:30 and video will take me at least 3 more hours, apologies 🙇‍♂️)

[00:10:21] Alex Introduces Yam Peleg

[00:10:57] Alex Introduces Nisten Tahiraj

[00:11:10] Alex Introduces Far El

[00:11:24] Alex Introduces Xenova

[00:11:44] Alex Introduces Roie S. Cohen

[00:11:53] Alex Introduces Tzafrir Rehan

[00:12:16] DALL-E 3 - An AI art model with a brain, coming to chatGPT plus

[00:20:33] Microsoft c launches Windows CoPilot

[00:30:46] Open AI leaks, GPT-4 Vision, Gobic

[00:38:36] 3.5 instruct model from OpenAI

[00:43:03] Raunak intro

[00:43:25] Bard Extensions allow access to GMail, Youtube, Drive

FULL transcript:

[00:00:00] Alex Volkov: So, Thursday I is this wonderful thing that happened and happened organically as well.

[00:00:26] And basically what happens is we have this live recording every Thursday, every ThursdAI on Twitter spaces. I am I'm very grateful to share the stage with experts in their fields, and we all talk about different things, because AI updates are so multidisciplinary right now. It's really hard for even experts in their one field to follow everything.

[00:00:51] I find this mixture of experts type model on stage very conducive because we all go and find the most up to date things from the last week. And then we have folks who, it's their specification, for example, to comment on them. And you guys in the audience get the benefit of this. And it just happened organically through many conversations we had on, on Spaces since GPT 4 was launched.

[00:01:16] Literally the day, March 14th, 2023 aka Pi Day. It was the first day we started these spaces, and since then the community has grown to just... An incredible amount of people who join quality experts, top of their field people. I'm, I'm just so humbled by all of this. And since then, many folks told me, like Roy here in the audience, that, Hey, Alex, you're doing this in this weirdest hour.

[00:01:42] Thursday a. m. in San Francisco, nobody's gonna come. It's really hard to participate in the actual live recording. And so, I started a newsletter and a podcast for this. And so, if you aren't able to make it, I more than welcome you to register to the newsletter. You know what? Even if you are here every week, register to the newsletter, because why not?

[00:02:03] Because, share it with your friends. We're talking about everything AI related. Hopefully, hopefully no hype. And I have friends here to reduce the hype when I'm getting too hypey. Definitely none of the, Hey, here's a new AI tool that will help you fix the thing you don't need fixing.

[00:02:18] And I think that's, that's been resonating with the community. And so, as you now are here, you're also participant in this community. I welcome everybody to Tag Thursday AI on their news about ai or #thursdAI, or just like the Thursday iPod, which probably should join this so people get some more visibility. but you are part of the community. Now, those of you who come back, those of you who listen in, those of you who share all of them. All of these things are very helpful for the community to grow and for us to just know about more stuff.

[00:02:49] It's actually an incredible signal when two or three or more of you react under a piece of news and say, hey, we probably should cover this in Thursday. It really helps, truly. I think with that, yeah, I think this intro is enough intro. Welcome. What's up, Tzafrir? How are you?

[00:03:06] Tzafrir Rehan: All's well. Thank you very much. I wanted to, to strengthen your point about the time factor. So we expand. So anyone here who wants to be a little bit interested in generative technologies and breaking news and have some things to do in the meanwhile, and also looking to actually build something cool from all of this.

[00:03:31] Time is the limiting factor here. That's like the, the hardest resource here. Having this group and having everyone explore everything together. It's a lifesaver. It's like a order of magnitude improvement on our ability to move forward each one individually. And that's a group together just to give examples.

[00:03:53] So I'm interested in generative images, videos, and audio. And for each of these, there are hundreds of models right now available. With the availability to make fine tunes on specific datasets for some of these generating a single asset like a video can take hours. Training takes hours. If you want to explore a little bit like the effect of different prompts, just generating hundreds of samples takes hours.

[00:04:26] So without this group, it would be impossible to even know. Where to go and where to invest my time And the name of the game right now is to just choose where you invest your time on To actually get things done and keep up. So thank you. Thank you. Thank you for you and for this group And let's have fun.

[00:04:46] Alex Volkov: Thank you. Thank you everyone. I definitely feel super powered by the people in this group who can like back me up on, I read one tweet and then I saw some people react to this tweet, but I didn't have the time or the capability or the experience to dive in.

[00:05:00] And then there's folks here who did, and then we're going to complete each other. And I think our model, I haven't shared since we started, but our motto is we stay up to date. So you don't have to and have to, I think is the operating word. You want to stay up to date and you're welcome to stay up to date and you're welcome to tag us and talk with us and leave comments here in the chat as well, but you don't have to anymore because, there's a, there's a newsletter that will update you and there's folks on stage who will talk about this.

[00:05:26] I want to briefly cover one tiny thing that I did on the podcast that I think I will start doing as well. So, so far editing this hour and a half, two hours that we have here live was a pain, but I just decided to lean into this because. The conversation we're having here is so much more informative and interesting that any type of summary that I want to do or wanted to do is not going to do it justice.

[00:05:50] And so I had some different feedback from different folks about the length of the podcast. Some people said, yeah, 25 minutes, just the updates is like the right spot. And yeah, the podcast is moving towards. This is going to be the live recording. I'm going to edit this don't worry.

[00:06:04] But besides that, the podcast will be this conversation. Going forward as much as I'm able to edit this, and ship both the newsletter and the podcast in time on Thursday But with that Tzafrir thank you for the kind words, man. I appreciate you being here and sharing with us your expertise

[00:06:20] I want to say hi to Zenova and Arthur.

[00:06:22] We'll start with Zenova. Welcome Josh. How are you?

[00:06:27] Xenova: Yeah. Hey Yeah, pretty good. Been busy, busy, busy

[00:06:33] for those who Don't know. I'll just quickly introduce myself. I am the creator of Transformers. js, which is a JavaScript library for running HuggingFace Transformers directly in the browser, or Node, or Deno, or maybe Bunsoon.

[00:06:49] Who knows when that gets sorted out properly, but any JavaScript environment that you're, that you're looking for. And, yeah, I recently joined HuggingFace, which is exciting. Now I'm able to sort of work on it basically full time. And yeah, lots of, lots of exciting things are, are in the pipeline.

[00:07:06] Alex Volkov: It's been incredible to have you here and then see your progress with Transformer.

[00:07:10] js and then you joining Hug and Faceman. I appreciate the time here.

[00:07:13] Arthur, thank you for joining. Please feel free to introduce yourself.

[00:07:18] Arthur Islamov: Okay. So, my name is Arthur and I'm fixing and making WebAssembly to work with big models.

[00:07:25] So, soon you will be able to run anything huge in the browser, and I'm particularly interested in diffusion models, so right now I'm making the Staple Diffusion 2. 1 to work in the browser, and then have some plans to make SDXL, and maybe as well as Lama and other models too. With all that work done.

[00:07:50] Alex Volkov: That's awesome. Thank you for joining.

[00:07:52] Far El: Yo, what's up? Yeah, I'm my name is Farouk. I'm like founder of Nod. ai where we build autonomous agents and also working on skunkworks. ai, which is an open source group where we are pushing the boundaries of what we can do with LLMs and AI as a whole, really.

[00:08:10] Our first, like, major project is this open source MOE architecture that we've been tinkering around with for the last couple months. We're also exploring even more, exotic AI arcs to try to get, to GPT 4 level capability for open source.

[00:08:28] Alex Volkov: Awesome. Awesome. Awesome. And Nistan, welcome brother.

[00:08:33] Yam Peleg: Yeah. Hey everyone, I'm Nistan Tahirai and I'm terminally online. That's the introduction. Thank you. Yeah, I, I'm also, I'm a dev in Toronto. I worked on the first doctor wrapper which is still doing pretty well. Like no complaints so far, six months later, knock on wood. And yeah, recently started doing a lot more open source stuff.

[00:09:03] Put out a bunch of open source doctor models on, on HuggingFace, which I still need to write a benchmark for because there is no safety benchmarks that are public. And yeah, lately been working with Farouk to make the whole Sconcrooks AI mixture of experts model more usable because it's still, it's not even bleeding edge.

[00:09:26] And this one is more like hemorrhaging edge technology. It takes like three people to get it to work. And yeah, I've been extremely interested on the web GPU side ever since Zenova on a random tweet just gave me the command to start Chrome Canary properly. And then I was able to load it. Whole seven B model.

[00:09:48] And yeah, I'm thinking next for the future, if, if things go okay. I mean, my goal that I've set myself is to have some kind of distributed. Mixture of experts running via WebGPU and then having Gantt. js encrypts the connections between the, the different nodes and experts. And we'll see how that plays out because everything is changing so quickly.

[00:10:14] But yeah, it's, it's good to be here. And I'm glad I found this Twitter space randomly way back in

[00:10:21] Alex Introduces Yam Peleg

[00:10:21] Alex Volkov: Yeah, for a long time. I just want to welcome Yam to the stage. And Yam doesn't love introducing himself, but I can do it for you Yam this time if you'd like.

[00:10:31] All right. So, I will just run through the speakers on stage just real quick. Yam, thank you for joining us. Folks, Yam is our, I could say, resident... Machine learning engineer extraordinaire everything from data sets and training large language models understanding the internals of how they work and baking a few of his own definitely The guy who if we found the interesting paper, he will be able to explain this to us

[00:10:57] Alex Introduces Nisten Tahiraj

[00:10:57] Alex Volkov: Nisten. I call you like The AI engineer hacker type, like the stuff that you sometimes do, we're all in awe of being able to run stuff on CPU and doing different, like, approaches that, like, nobody thought of them before.

[00:11:10] Alex Introduces Far El

[00:11:10] Alex Volkov: Far El you're doing, like, great community organizing and we're waiting to see from the MOE and Skunkworks.

[00:11:15] And folks should definitely follow Far El for that and join Skunkworks OS. It's really hard for me to say. Skunks. Works OS efforts in the discord.

[00:11:24] Alex Introduces Xenova

[00:11:24] Alex Volkov: Zenova is our run models on the client guy so Transformers. js, everything related to ONNX and everything related to quantization and making the models smaller.

[00:11:35] All of that. All models, all modularities, but I think the focus is on, on the browser after you're new, but obviously you introduce yourself, WebGPU stuff.

[00:11:44] Alex Introduces Roie S. Cohen

[00:11:44] Alex Volkov: We have Roy, who's a DevRel in Pinecon, who he didn't say, but Pinecon and VectorDB is in Context Windows and, and discussion about RAG, like all of these things Roy is our go to.

[00:11:53] Alex Introduces Tzafrir Rehan

[00:11:53] Alex Volkov: And Tzafrir also introduced himself, everything vision, audio, and excitement. So a very well rounded group here. And I definitely recommend everybody to follow. And now with that, now that we are complete, let's please start with the updates because we have an incredible, incredible Thursday, literally every week, right folks?

[00:12:12] Literally every week we have an incredible Thursday

[00:12:16] DALL-E 3 - An AI art model with a brain, coming to chatGPT plus

[00:12:16] Alex Volkov: so we'll start with, with two big ones. I want to say the first big update was obviously DALL-E 3. So I will just share briefly about my story with DALL-E and then I would love folks on stage also to chime in. Please raise your hand so we don't talk over each other. DALL-E when it came out, When the announcement came out for DALL-E 2, I want to say it was a year ago in, a year and a half ago, maybe, in January, February or something, this blew me away.

[00:12:47] I have something called aphantasia, where, I don't know if you saw this, but like, I don't have like the visual mind's eye, so I can't like visually see things, and it's been a thing with me all my life, and then here comes the AI tool that can draw. Very quickly, then I turned my, I noticed stable diffusion, for example, and I just like.

[00:13:04] It took away from there. Everything that I have, all my interest in AI started from DALL-E basically. And DALL-E 3 seems like the next step in all of this. And the reason I'm saying this is because DALL-E 3 is visually incredible, but this is not actually like the biggest part about this, right? We have mid journey.

[00:13:22] I pinned somebody's comparison between DALL-E and mid journey. And Midrani is beautiful and Gorgias is a way smaller team. DALL-E 3 has this beautiful thing where it's connected to ChatGPT. So not only is it like going to be not separate anymore, you're going to have the chat interface into DALL-E 3.

[00:13:41] ChatGPT will be able to help you. As a prompt engineer, and you'd be able to chat with the creation process itself. So you will ask for an image, and if you don't know how to actually define what you want in this image, which types, you'd be able to just chat with it. You will say, you know what, actually make it darker, make it more cartoony, whatever.

[00:14:01] And then chatGPT itself with its brain is going to be your prompt engineer body in the creation. And I think. Quality aside, which quality is really, really good. The thing they're highlighting for, for DALL-E 3 is the ability to have multiple. Objects and subjects from your prompt in one image because it understands them.

[00:14:23] But also definitely the piece where you can keep talking to an image is changing the image creation UI significantly where, mid journey. With all, all the love we have for Midjourney is still stuck in Discord. They're still working on the web. It's, it's taking a long time and we've talked about Ideogram to lead them from the side.

[00:14:44] We know that Google has multiple image models like Imogen and different ones. They have like three, I think at this point, that they haven't yet released. And DALL-E, I think is the first. Multimodal on the output model that we'll get, right? So multimodal on the output means that what you get back towards you is not only text generation and we saw some other stuff, right?

[00:15:06] We saw some graphs, we saw some code interpreter can run code, etc. But this is a multimodal on the output. And Very exciting. I, I, DALL-E 3 news took Twitter by storm. Everybody started sharing this, including us. We can't wait to play with DALL-E 3. I welcome folks on stage. I want to start with Zafreer reaction, but definitely to share what we think about this.

[00:15:26] And the last thing I'll say... Say is that now that the community community is growing, suddenly people dmm me. So first of all, you're all welcome to DM me about different stuff. I see I see somebody in the audience with DM me. I think she's still here. So shout out about joining the better test for DALL-E three, which now they, they're able to share about Funny tidbit, it will, it's right now baked into the UI.

[00:15:48] So Dally 3 is going to be baked into ChatGPT and ChatGPT Enterprise UIs. However, when they tested this, they tested it via a plugin. So OpenAI actually built a plugin and had like a restricted access to this plugin. And folks who like talked with this plugin, the plugin ran the Dally ChatGPT version behind the scenes.

[00:16:06] And we don't have access to it yet. I don't know if anybody on stage has access. Please tell me if you do. The access is coming soon, which is interesting from OpenAI. And I think that's most of the daily stuff that I had. And I want to, please, please, buddy, I want to hear from Zafira, please.

[00:16:23] And please raise your hand. I really need us to not talk over each other.

[00:16:30] Thank you.

[00:16:31] Tzafrir Rehan: So yeah, DALL-E 3 is looking amazing. I did see some examples that people with early

[00:16:38] access were

[00:16:38] generating, and it's far more detailed and coherent than the things we are used to seeing from stable diffusion. And much less randomness, I would say. And what's exciting here is a few changes in the paradigm of how it works.

[00:16:56] For example, like you said,

[00:16:59] it doesn't expect you to know all the intricacies. You can describe in

[00:17:03] your natural language what you want to see

[00:17:05] and it will use

[00:17:07] GPT, however much they are powering the, for generating a prompt to make the whole image. That's the one thing. The other thing is that it's not

[00:17:19] text to image.

[00:17:21] It's more a conversation. Similar to how chat GPT is a conversation between you and the assistant. DALL-E 3 is a chat. So you can see in the video that they released. You generate one image and then you discuss if you want to make changes to it, if you want to make more variations, and that would be very interesting to see the flow.

[00:17:44] From the AI artist perspective, I think it will be met with a little bit hesitation, at least not knowing how much fine control they are providing. If they are letting away... to influence all these various parameters that the model uses. That is a lot of the workflow for generating AI art.

[00:18:06] And when you want to make a piece for release as an artist, you spend a lot of time fine tuning it.

[00:18:13] And today with Stable Diffusion, and with Mid Journey, we have a lot of fine grained control over changing the parameters by a little bit, adding one more word, That's one thing, and another thing is that artists usually actually want to have that control over the prompt. For example, this week I saw an interesting example, I'll try to find it for you, where the artist adds the words Event horizon to an image.

[00:18:44] Now the image is not of space, but the model does take that idea of the event horizon shape, and makes the image more shaped like an event horizon. So those are the kinds of tricks that right now prompt engineers use to make very specific changes in the image. So I'm interested to knowing if DALL-E 3 will allow that kind of control.

[00:19:08] And most of all, finally, we had DAL E2 very early in the game, before Stable Diffusion even gave the first clunky models, before everything, and there was so much work and mid journey. And so many much interesting things coming out in image generation and open AI will always like hanging back.

[00:19:30] We have this very basic value too, which sometimes works and usually doesn't gives you very weird results. So yeah, good to see that they are still working on actually

[00:19:43] innovating

[00:19:44] and thinking of the next step and how we can combine all of these technologies. To make something that's much more fun to the user experience.

[00:19:53] Alex Volkov: Absolutely. And I will remind some folks the internals behind kind of diffusion models, like stable diffusion, et cetera. OpenAI actually made the whole field happen, I think, with some was it VIT? Vision Transformer that they released and,

[00:20:05] Yam Peleg: they released the first diffusion. The first diffusion model.

[00:20:08] Alex Volkov: Yes. And so like the whole field is all to open the eye and it's great. I, it's a fair, I joined you in the, it's super great to see them innovate and give us some new UIs for this because. I heard from multiple people who have access to this, that this, you can get lost in just chatting to a picture, to the creation process.

[00:20:26] It's like a whole new creation process, basically, like prompting, but chatting. I'm very excited about this, very excited.

[00:20:31] , so we'll definitely talk more about this.

[00:20:33] Microsoft c launches Windows CoPilot

[00:20:33] Alex Volkov: I want to move on to the next thing, which is exciting. And so. Until today, basically, the word co pilot meant GitHub co pilot, at least for those of us with VS Code, those of us who write code. GitHub co pilot obviously is the auto complete engine that, gives you code abilities.

[00:20:50] And many of us use it, many of us don't use it. But, today, I think, Microsoft who owns GitHub and who is very close with OpenAI has announced Copilot for Windows. And it's coming soon with the Windows update. And we've seen some previews about this in some discussions. And I find it very interesting that Microsoft is innovating in AI, whereas we're waiting for Google to come up with Gemini.

[00:21:18] We're waiting for Google to, we're going to talk about Bard updates as well. But Copilot for Windows will be able To be just like a shortcut away. I think windows C is the new shortcut and you'd be able to ask it like he asked you for different things. And for those of us in the audience who didn't join us in the previous ThursdAIs, we.

[00:21:40] Talked with Killian from this open source called Open Interpreter. And one of the things that we all like in Open Interpreter is that it runs on my machine and it generates code, and some of that code could be AppleScript. And so it's very easy to run stuff on the Mac using AppleScript. You can open Calendar, you can send emails, you can do a bunch of stuff.

[00:21:58] And so it was beautiful to see that, like, even an open source agent like Open Interpreter is able to Run code and then, activate stuff on your computer. Having, and I think Kilian mentioned, like, Microsoft's Copilot is coming. And not just a week later, exactly a week later after that discussion, we now have Windows Copilot.

[00:22:16] Which is going to be able to run Windows for you. It's going to be able to open apps and shut down apps. It's going to be able to just like... Be a, chat GPT, but living inside windows. And I think it's going to be based on GPT 4. It only makes sense with the Microsoft OpenAI collaboration. And like I can't understate this for a second.

[00:22:38] GPT 4 was released on March, right? Chat GPT was released less than a year ago on November something. And now the next version of world's probably most. Common operating system, Windows, is going to have AI built in as a companion. How insane is this, folks? I, I, I, I have a Windows machine, because I have an NVIDIA GPU, blah, blah, blah, and not only I'm not only on the Mac and I'm really excited to, like, play with this.

[00:23:09] An additional thing that they've announced together with this update is connecting to the previous thing that we said, which is Bing, Chat, and Windows Copilot will both have DALL-E 3 built in for free. So DALL-E 3 is going to be possible on GPT Plus subscribers, the ones of us who paid the 20 bucks.

[00:23:32] However... For, through Bing, you'll be able to get it for free, and it's going to be part of Windows. Right, so, my mom, who probably doesn't use Windows, okay, her husband, my mom's husband uses Windows, he'd be able to use GPT 4 to run his Windows and also generate images. I think that's incredible, and, only Microsoft can give it out for free.

[00:23:52] I think that's mostly it in... The Microsoft update. However, it's breaking news. Literally, they released the tweet once we started the space So I'm sure more stuff will come out of there But I invite folks on stage to chime in with Windows Copilot news What do you think about this whether or not, you know This is going to change multiple people's usage of Windows or Or not

[00:24:16] Nisten Tahiraj: I mean the whole Using software thing is all up in the air now, right? Everyone's in creative mode. Yeah, it's pretty hard to predict what's going to be the, the better interface voice is getting really good. Open interpreter show that it can do a whole bunch of stuff. You can also delete all the Jason files on your computer accidentally, but I think those, those will be worked out those issues.

[00:24:43] Yeah, it is hard to, it's hard to call because again, being is still a free beta service, they haven't quite figured out how to fully monetize that, because that's not cheap to run especially considering that it is the multimodal image one, so. Yeah, don't have that much an opinion.

[00:25:05] I think it's still too early to call as to how interfaces will change.

[00:25:09] Alex Volkov: I agree. I just, I'm excited that AI that we've come to known for less than a year is now baked into an operating system for everyone, right? Even going to a website like chatGPT registering is not for everyone and they will. They will definitely , lower the bar for usage here. What's up, Yam

[00:25:28] Yam Peleg: hi I just want to say that we've seen, because everything is so early, we've seen really great infrastructure for RAG but we haven't seen a wide scale product using RAG on this scale. So, and, and it makes sense at the end.

[00:25:47] I mean, you have a lot of information scattered around all different software and different devices. It's, I think it's the perfect idea to just merge everything with the RAG and just allow you to chat with whatever information you have everywhere. And Microsoft is perfectly positioned to do that. And I'm looking forward.

[00:26:13] I think that I think it's a great idea. I don't know if the implementation. Will be great. It's, we need to see, I think it will, but we need to see, but I think that's it. As a concept is a great concept.

[00:26:26] Alex Volkov: Something that I saw from a person who's very close with the Microsoft team, for some reason, the guy behind being his name is Michael Perakin, and he has this like very non branded Twitter account that barely has an avatar image.

[00:26:43] And he's been doing, he's open. Yeah. He's been doing, he's been doing like customer support basically on Twitter. Like people will say, Oh, Bing has this, has that. And he's like been very, very responsive to some people. And so two things that he did say, first of all, Dally three is already part of Bing for some percentage of population.

[00:27:00] So if you use Bing, and we've talked about Bing before about image and vision. If you use Bing, go try and generate images with it. It used to be Dally too, but if you get. Good ones. You may get value three, which is incredible. You may already have this. And the second thing is I saw somebody commented that he is now head of windows, right?

[00:27:17] So the guy behind being the guy who pushed a I into being is now moving to be ahead of windows. And I think this together with this release shows us that. How just how much Microsoft is serious about a I everywhere and is determined to not miss this new wave like they missed the mobile wave. And everybody says that, Apple overtook Microsoft and Microsoft was like late to mobile.

[00:27:37] And And it just goes to show like how much they invest in this whole thing. And I find it like very, very good because for many people, even going to a website is a barrier of entry. And then when it's just like one click in their operating system of choice, I think it's going to be very it's going to shove AI into way more people's faces.

[00:27:54] I also want to say that Microsoft out of the big ones is fairly based in terms of. Safety and regulation, which we usually don't talk about we can talk about in maybe the next space, but like, we can have worse than Microsoft, which is surprising for me because I used to hate on the Internet Explorer most of my life.

[00:28:12] And so now Microsoft is very based. I think less comments on Windows Copilot here, folks, and then we can move on to the next stuff from OpenAI, actually.

[00:28:22] Nisten Tahiraj: So my last one is I've started using Edge Canary as my daily browser just because of the sidebar and the splitting. So if you have a widescreen monitor, it's actually very handy because you can have code interpreter on one side, and I'll show an image of it very quickly.

[00:28:39] And I have Bing, which has an excellent voice back and forth. And it has really good voice generation, which normally would be very expensive if you're paying for it, but it's in beta And then I have the actual work and on the sidebar you can have Anyway, this interface is a bit convoluted and edge browser is it's still a little bit clunky, but Overall, it's been working pretty well for me.

[00:29:06] So I I don't know. I sort of see the browser as being more and more important. That's your operating system. Some people disagree. They're trying like Sean is, is trying to do more of a OS native stuff with his tool that lets you run multiple ones. But Yeah, you can see the screenshot of how I started using it with voice, so.

[00:29:28] In general, I see it as you'll just talk to it back and forth. I think That's,

[00:29:32] Alex Volkov: at least that's what I want. Were you referring to Swix's Godmode app where you can run all the LLMs in like a window?

[00:29:39] Nisten Tahiraj: Yes, but that one, for example, on the Mac is right, there's an icon right beside the clock. And you just click that and it pops up, so it's unintrusively there.

[00:29:49] And it adds to your experience instead of getting in the way. And I, I do like that part because it is using real estate on the screen efficiently, but again, if you have a. If you use a wider monitor, so can Edge with all of its right sidebar shortcuts, because then you can add your discord, your outlook and stuff there too, right where the GPT like right where I use the code interpreter window and even have some completion and document writing stuff too now.

[00:30:19] So that's how I see it. I, it's again, it's up in the air, what people will find most helpful

[00:30:25] Alex Volkov: absolutely. And I've been using Bing somewhat as well. And yes. The sidebar can also read from the page, right? So the Bing chat in the sidebar has access to the page if you give it.

[00:30:37] And that for like summarization and different things, that's really, really excellent as well. Like it completes your browsing experience. So I'm assuming that they're doing some stuff with the co pilot.

[00:30:46] Open AI leaks, GPT-4 Vision, Gobic

[00:30:46] Alex Volkov: All right, folks, we're moving forward because we have much to cover. And, there's more news from OpenAI.

[00:30:52] They actually came before DALL-E, and we were supposed to talk about them first, and then DALL-E, but sorry, and then DALL-E came out. And now let's cover some news from OpenAI. So... It feels like the theme behind all of these news is OpenAI is trying to rush stuff to the door or to announce some stuff to the door because they know or they hear or they saw the information from Google breaking out about Gemini, the multi model wolf.

[00:31:19] Huge model from, from Google that is potentially GPT 4 like and can do images in the input and output is multimodal on the output as well. And so we don't know many sorry, we don't know much information about Gemini so far, but we do know that the information kind of the publication called the information released that Gemini is coming very soon.

[00:31:40] And we see the response from OpenAI in multiple places, right? So DALL-E 3 is one of them. OpenAI released so the information also leaked. about open the eye gearing up to give us vision for those of you who remember pretty much every space since march we're talking about gpt4 that is also multi model on the input and yeah we can probably go into the details whether or not it's fully multi model versus gobby and i would love for you to participate in this but basically gpt4 when they announced they showed the demo of it they gave it some screenshot they gave it like a sketch of a website that was able to code that and then we didn't get That feature, the Multimodality from GPT 4, we didn't get it.

[00:32:20] The only people who got it, and me and Nisten interviewed the CEO of this, is Be My Eyes, which is this app for blind folks, and they just like shove GPT 4 vision in there to help those with eyesight issues. And it seems that now Google has finally stepping into the arena, sorry for the pun, and that we may get GPT 4 vision very soon.

[00:32:42] I actually saw some screenshots how it looks inside the GPT 4 chat GPT interface. And the additional exciting thing is, they have a different model. With the code name Gobi, that as apparently it works in OpenAI. And that one is going to be multi modal and like fully. So, Yam, I would love to, if you can repeat what we talked about last night, about the differences and how GPT 4 is multi modal, but not fully.

[00:33:06] I would love for you to expand on this.

[00:33:09] Yam Peleg: Yeah. First it's important to understand that there is a huge difference in infrastructure between the two companies. And the infrastructure dictates what is possible or not possible, what is hard or not hard. From the rumors nothing is confirmed, but from the rumors the, the structure and the size of GPT 4 is.

[00:33:34] It was chosen to fit the hardware, the infrastructure to actually run the model. It doesn't matter if you have the best model in the world, if you cannot just serve it. So Google is using its own hardware, which is not sharing with anyone else. And it's important to understand this. So when we see that Google is doing according to the rumors.

[00:33:58] And, and insane training run or, or preparing to ship or, or serve an insane model that is multimodal on the input and on the output. It, the reason we didn't see, I think, I think the reason open AI I didn't release a GPT with the image head is simply because it's. It's probably expensive. It's not that easy to deploy something like this, especially not with the amount of people that use OpenAI services.

[00:34:31] And, and I think this is this is what we see. This is the reason for what we see at the moment. Now it's important to understand that according to rumors, again, nothing is confirmed, take with a grain of salt, according to the rumors, which makes sense, GPT 4 is first a language model. It was trained as a language model, just language model.

[00:34:53] And once it was trained. It, there was they added an image head to the frozen model. This basically, this reduced the risk of something going wrong with full multimodality end to end. And moreover it allows you to just use the model on its own. And if you want, you can plug the head so you can use them.

[00:35:14] You can, it's flexible. You can use them with or without a head. Now, the thing is that there is you do pay a price because again, with a grain of salt, there, there is there is, there are caveats to this, but we have already seen multiple times that multimodality, when done right, benefits both modalities.

[00:35:36] So GPT 4 allegedly did not benefit from the multimodality. And this is the difference between GPT 4 and the new rumored model that we have. According to the rumors, the rumored model was trained end to end images and text throughout the whole training. So, we should, if it's true, if everything is true we should expect a better model only if you just use it for text, we should expect a better model because the, the images just influence the text and text influence the images and So on and so forth.

[00:36:12] Alex Volkov: That's great. That's what I have. One follow up question. You spoke about benefits from training on text and vision. And I remember Ilya Asatkov also talked about this. I think with the Jensen CEO of NVIDIA. He talked about different other places. Could you speak to some of those potential benefits of how multi model trains on text and images is actually better?

[00:36:37] Yam Peleg: If I remember correctly Ilya said Ilya gave the perfect example for this. You can, if you really want, you can describe what the color red mean with text or what, what objects are red. All of this will be nothing like just seeing the color red. So there is a difference between actually training on images.

[00:37:04] Versus training on text that describe the images which is just, it's just a different sensation. So the whole you can say the world model inside the, the language model is influenced by, by the images. And I think color is, is just a great example. And if I remember correctly, that was Example he gave in this interview.

[00:37:27] Alex Volkov: Yeah, absolutely. And I think the other one he said is It's obviously better at stuff like math or physics where it's able to actually read different, the graphs and everything. It's like, it just arrives at the question faster, but also like Yam you correctly pointed out the world model of this model is way better because it's able to see basically.

[00:37:50] So We have potentially exciting news. One thing I will add is that Yam I think you're correct opening. I just didn't want to spend the kind of this GPU cycles on the vision model and the being able to attach a head with vision. I think it's exciting. I do want to highlight that Microsoft likely has.

[00:38:08] The bandwidth for that, because being has the ability to have vision. Now, I don't know if it's like the full one. I don't know if they did some work because the examples that I tested with being vision gave less quality like responses on images than I was expecting GPT four from the example.

[00:38:25] So maybe they know if they, maybe they did some stuff for optimization speed, but yeah, definitely it feels like infrastructure was gearing up for this and hopefully we'll see it soon. From OpenAI.

[00:38:36] 3.5 instruct model from OpenAI

[00:38:36] Alex Volkov: Another thing we saw from OpenAI, and I think this is this last one, we have a bunch of OpenAI updates, is the 3.

[00:38:42] 5 Instruct model. And unlike the ChagGPT model, 3. 5 Instruct is very similar to how OpenAI APIs We're actually working before the ChatGPT explosion, right? Before you were able to like do back and forth conversation. Before it was RLHF for conversation purposes. And I saw many, many folks get very excited about 3.

[00:39:05] 5 Instruct. Because it's very similar to what we had before ChatGPT. But it's much faster. Now we don't know if it's faster because way less people use this because it's new. Or it's faster because they actually did some TurboMagic on it. But, we'd love to invite folks on stage, maybe Roy, maybe Mr.

[00:39:21] Yang to talk about the instruct and the difference between kind of this end point in the API versus the regular chat end point. If you have anything to, to, to, to add here from what you read, please feel free to, to add.

[00:39:36] Nisten Tahiraj: I used it in the playground to just like write An agreement for the site, like a privacy agreement.

[00:39:41] It was pretty good for that. It just it's annoying that the context window is so small. It's only a 4k context window. And it's more like only three and a half K because some of it will be your prompt. I think it has some very other very good usability uses, which we haven't experimented with yet.

[00:40:02] Like the one person got it to play chess very well. And I think it's, yeah, it's really worth looking at for stuff like doing automation or you're continuing some work on your desktop, for example, with open interpreter, and it'll be able to continue generating in that regard. So there are quite a few things to explore there.

[00:40:26] I'm just glad it's cheap and it's good. So that's that's what we want at the end of the day

[00:40:34] Alex Volkov: Yeah it's it's cheap and I think for many folks they were surprised with like the chat interface They had to switch for chgpt to get like the benefits and the speed and now they're happy that they have the instruct model of old They also added log props.

[00:40:47] So I, I would love to ask folks on stage because I'm not entirely sure what like logprops is in the API response. And I saw Alex Gravely and some other folks are getting excited about logprops. And I want to say, just before, I want to say hello to Ronak, if I'm pronouncing this correctly. Ronak. And we're going to talk about RamboRole in a second, or in a few minutes, but if you have comments on the Instruct API and LogProps, feel free to share.

[00:41:18] Raunak Chowdhuri: Yeah, I do. LogProps is awesome. It basically gives you like, token level probability distributions on, in terms of the model. So normally when you are using GPT 4 or GPT 3, You just get words back when you're, when you're querying the model. But what LogProbs allows you to do is, is see the probability distribution that's outputted by the model that is normally sampled by, like, the temperature parameter.

[00:41:43] And you can use that to do a lot of, like, really interesting things. Like, for example, if you're, if you're asking GPT to solve a multiple choice question, for example it's really useful to actually understand. the model's confidence in whether it's A, B, C, or D. And you can actually get that directly from the model by examining that probability distribution from the log prop.

[00:42:05] So it actually provides... A lot more insight into what the model is thinking and I think that's a pretty useful technology. You can actually do a lot of clever things with it, like someone built something called like JSONformer which is basically like a tool that allows you to if you have a model that exposes log props, you can only sample the tokens.

[00:42:24] That basically are valid JSON tokens, and construct a response that is very much aligned with like a certain format that you want. So I think that's a pretty powerful tool.

[00:42:36] Alex Volkov: Thank you, Ronak. Thank you. And I remember JSONformer and did not know that they use log, log prox for that. So here you have it, folks.

[00:42:43] There's a new endpoint for you, your usages that now exposes the token probabilities. So you can use this to build better tools and different types of tools. And yeah, Ronak, would you care to introduce yourself briefly? I will ask again once we record kind of your section, but feel free to introduce yourself.

[00:43:03] Raunak intro

[00:43:03] Raunak Chowdhuri: Yeah, absolutely. So I'm a senior at MIT. I'm graduating in a couple months. My background's in machine learning, artificial intelligence. I've been doing research in this area for quite a few years now. Yeah, I'm working on some interesting projects that we'll dive into later, but basically building long term memory for Language models.

[00:43:25] Bard Extensions allow access to GMail, Youtube, Drive

[00:43:25] Alex Volkov: Awesome, awesome. Thank you. Thank you for coming up and thank you for explaining log, log props as well. All right. So the next thing I want to talk about briefly, really briefly, because it's not that great is bard from Google. Before we get to Gemini, before we hear from Googles like Explosive, GPT 4, Combating Model, etc. Right now we have Bard. For some reason we also have Google Assistant, which I'm not sure what's the involvement with LLMs there. But Bard is something that some folks on stage here use. And I was never like very, very excited about Bard for some reason.

[00:44:00] However, they just released a few updates to Bard and they say like this is the best Bard ever. And it feels like very googly, very like product manager y to me, at least. What they released is something called extensions, right? So if you use Bard before and you haven't touched it in a while, like I haven't, if you go to Bard right now, what you will have is the chance to...

[00:44:22] Updated with extensions. Those extensions could access your Gmail, all of it, your Google Drive, all of it, YouTube. I think some other ones that I'm trying to remember. And the cool thing about this, which I actually like, is the UI. You can do at sign, like like you mentioned somebody on Twitter.

[00:44:38] And then you have access to those extensions. It's a different take on the plugins with ChagPT, where like ChagPT plugins, you have to be in that mode, it decides for you, blah, blah, blah. So here you can actually say like, add Gmail, and then ask it questions. It will actually go and do a search in your Gmail account and give you back answers with, with natural text.

[00:44:56] So. Conceptually pretty cool, right? We all use Gmail, or like at least most of us use Gmail. And so to be able to like get summaries of the latest emails, blah, blah, blah. So conceptually very cool. Google Docs as well. You can tag Google Docs. You can do Google Drive. Oh, Google Maps is the, is the other one.

[00:45:10] So you can actually say like, hey, what are some of the stuff that, in San Francisco, Seattle, whatever it will give you. The thing that I was really surprised by is just how bad it is, just honestly not to... If there's folks in the audience who work on Bard, I apologize. And sometimes we say these things, but there's like so many, so many people working on this stuff.

[00:45:31] And like, it's, the, the nights and weekends, they don't see family. So like, I apologize. Just network from the comparison point in my experience, I was really disappointed in how... Google, who's this like huge company that like created Transformers for us, like they, they're not afraid to release something this bad.

[00:45:50] And what is bad, I mean, specifically, I literally used two of the extensions. One is Gmail. To ask it about my upcoming flight to San Francisco, which I told you guys about. I'm going to be at the AI engineer event as, as a media person. And it couldn't find any information from this flight and just gave me flights from the past.

[00:46:07] I literally asked, give me flights from the future or like, give me my upcoming flights. And it gave me flights from the past. It also gave me two trips to the Denver museum, which is, which are not flights. And so, yeah, we know LLM hallucinates, blah, blah, blah. But if you, if you put your brand behind this and you're Google and you put Gmail in there and you cannot like do a basic search, that's upsetting.

[00:46:30] And so I said, all right, I'll give it another try. I did YouTube. And I asked, Hey, what does MKBHD, Marques Brownlee, if you guys don't follow him, he's like this great tech reviewer. What does he think about the latest iPhone? And it went to YouTube and it searched and it gave me. Marquez's videos from last year about the iPhone 14, and I literally took the same string that I pasted into Barg, went to the YouTube interface, pasted it in the YouTube search, and got like the latest videos that he had about the iPhone 15.

[00:46:58] And so I was thinking there like why would I ever use this if like the first two searches did not work, where this is the whole promise of this. So again, not to be negative. I don't love being negative. It's just like from a comparison standpoint. It's really I really got to wonder how many folks in Google are trying to rush through the LLM craze.

[00:47:19] We remember Sundar Pichai saying, AI, AI, AI, AI, AI, AI, AI, AI, AI, on the stage like 48 times, right? And they're shoving AI into everywhere. It's just, for me, it wasn't that useful. So I would love to hear, Safrir, I see your hand up. I would love to hear from folks on stage about your experience with BARD and those specific kind of extension new things.

[00:47:41] Tzafrir Rehan: So I don't have much experience with it, actually, for the same reasons that you said. But I want to give the perspective that I think what we're seeing here is Google jumped early to stay in the game. Maybe they didn't expect ChatGPT to go viral that big so fast. Well, this was developed like a sci fi technology and suddenly it's a household item overnight.

[00:48:09] But, if you're talking about Google, and I worked at Google actually for three years, about a decade ago, it's a company that can make very big moves very slowly. That means, if Gmail data, Drive data, it's the holiest of holy of privacy. If you want as an engineer at Google, if you want to touch that data to read even a single bite, you need to go through quarters of legal meetings.

[00:48:41] So the fact that they are going in this direction indicates a course heading that they took the time to think of it through and decide, yes, we are doing this very risky move in terms of privacy and user expectations. Because they believe in the value. So let's see where they get to when they actually, when they are actually fully implemented.

[00:49:05] Because I think right now, what we are seeing is a rushed out version.

[00:49:09] Alex Volkov: I agree. I think that's how it definitely feels where the basic stuff, like. A keyword search works better than like this search and they're basically hitting the API, which they have behind it definitely feels rushed very polished UI wise, very safe, very like protective, like googly, but very, it's not super helpful.

[00:49:27] I think at this point Yeah, I think this is most of the news unless I'm missing some so let me look and see in my template that I already Drafted for myself. Let's see if we have any more things to cover before we before we move on to the interviews So yes, one last thing I wanted to find I'll just find this thing.

[00:49:48] It's called chain of density So, I saw this, I think it was a paper first, and then yeah, I'll share this in the chat. I'm sorry, not in the chat, in the, in the Jumbotron. I saw somebody release a paper on this, and then I think Harrison from LangChain reposted this and actually put it up on their website with the prompt sharing, where you can play with prompts, is this new method called chain of density, which is actually really, really good at getting summarizations from From ChatGPT and different other places like Cloud as well.

[00:50:21] And I think it's really cool because I just posted it on top. It it asks for four summarizations with more and more density, right? So it starts with like, hey, summarize this text or article. And then it says give me like a JSON file in response with like four summarizations. The second

[00:50:37] one, give me a summarization.

[00:50:40] Extract from the first one that you just gave me, extract the entities that were missing, and give me another summarization with those entities, and then do it again and again. And I think there's, like, some cool prompt magic in there that says something to the tune of, make sure that this is understood on its own, and the person doesn't have to read the article to understand the summarization.

[00:51:00] I personally have gotten really good summarizations based on this technique, so much so that I've added it to my snippets. Where, where I have different snippets for prompts. And if you are doing any type of summarization, definitely check it out. Nistan, I saw your hand briefly up if you want to comment on this thing.

[00:51:16] Nisten Tahiraj: Yeah. Like the first person that I knew who got a contract as a prompt engineer actually used this technique a lot last year. And the way he was explaining it was when you do, when you compress an idea and then you extrapolate, that's how creativity happens in general. Like you, you compress. You extrapolate out of it, you compress and then you extrapolate.

[00:51:36] So it's pretty interesting that someone did this in a much more systematic way. I'm, I'm going to check it out.

[00:51:43] Alex Volkov: Chain of density. And I wanted to ping back real quickly on the compressing part, because yeah, I saw your tweet and there was a paper about compression as well. And Ilya gave a talk about compression recently.

[00:51:55] And I wanted to see if you want to talk about that compression part and paper. Briefly and if not, that's also okay. We can move on but I just like I think this is also this week.

[00:52:07] Yam Peleg: Yeah, I got I Had some controversial opinions in the last couple of weeks And as it turns out there are papers that support them coming up after them.

[00:52:19] But yeah, I highly, highly suggest reading the compression paper. Basically, basically what it says is that it just it just conveys the idea that what, what we are actually doing is I want to say. Reversing the process that generates the data and by reversing the process that generates the data.

[00:52:39] If you think about it, the process that generates the data is us. So, I don't, I don't want to, I don't wanna say the, the the words that I shouldn't. I got a, I got some heat for them, but you can find in my tweets. It's it's a really good paper. It's really It's much more scientific, you can say versus other papers that talk about intelligence, about general intelligence, and poke on this idea, and I highly recommend reading this paper if you're interested in this part of what we're doing.

[00:53:13] It doesn't prove anything because, general intelligence is a, is a big thing, but it. It is it is interesting the ideas there are, are, are solid and great to see.

[00:53:24] Alex Volkov: Yeah, I, I, I heard this multiple times this comparison or metaphor that intelligence is compression and, compressing a lot of ideas into, First of all, it compares to natural language, the ability of us to understand something, to put it into words, that's compression.

[00:53:39] Obviously, Feynman's quote, where like, you really understand something if you can explain this to a five year old, is also like, compressing down and also being able to explain some stuff. And so I heard this multiple times, and it's great to see that, there's now papers to talk about this. And continuous compression, like Nissen said, actually...

[00:53:54] Actually brings out better results and it's also good to see on the topic of literal compression. I know this like it's confusing There was also another paper that's worth checking out from this week Where they actually used llms and different transformers for an actual compression to compare to like png or or jpeg, etc And I think they saw very interesting compression results as well I don't remember if I have a tweet for that But yeah, be on the lookout for the for multiple types of different compression, uh as we as we move forward Thank you I think With that, I think we are ready to move on to our guests here on stage and to talk about two exciting things.

[00:54:30] So, first of all, actually three exciting things. One of them is, Nisten you had a visit to Geoffrey Hinton's lab that I wanted to hear from you a brief story about. After that, we're going to talk with Arthur and Zenova about WebGPU and going to do like a brief interview about like... Running models locally in the browser.

[00:54:47] And then at the end, we're going to talk about remember all with Ronak and his exciting approach to extending complex windows. So with that, I'll just give a brief kind of summary of the spaces we had today and some logistics, and then we can get started with the second part of Thursday. So again, everybody in the audience, we're just drawing the joint in the middle or have joined us from week to week.

[00:55:08] Thursday is about. Staying up to date together and give giving updates every week so that folks don't have to follow everything because it's almost impossible. I'm very happy to be joined by multiple folks from different disciplines and folks who can answer stuff and complete and find new things to get excited about about AI.

[00:55:28] From different fields every week here on stage. We also have a podcast and newsletter. If you're here and you're new and you just like just joined us and you can join next week, you can sign up for the newsletter as well. We stay up to date. So you don't have to, this is the model. And the first part of this is usually updates from last week, breaking news.

[00:55:46] There's another breaking news with YouTube something, but I think we'll cover this next time unless folks here want to read up on this and then give us an update at the end. But the second part of it is usually A deep dive into different conversations and, and guests. And today we have Arthur and we have Ronak to, to talk about different, very exciting things.

[00:56:05] And we'll start with Nistan's brief foray into the lab, AKA yeah, Nistan, give us, give us like a few minutes on, on your, your excursions.

[00:56:16] Nisten Tahiraj: Well, I've been going as a guest to Vector Institute for. Over a year and a year and a half and this time I, I went in and I'd never met Pharrell in real life.

[00:56:28] I didn't even know what he looked like. It was just some dude. He was GitHub. And yeah, so I, I invited him in and we were going to work on making the bootable. Bootable OS that just boots straight into a GDML model and then hopefully gets Chromium running with WebGPU. And essentially I just, I made before a, a tiny 4.

[00:56:54] 7 gig ISO that includes an entire Llama 7b model and an entire Linux distro. I use Slackworks, that's the smallest, and I used that for like 20 years. And yeah, so we were in the lab and Eventually, he's like, let's just try and get the whole thing working. So let's just try and get the mixture of experts.

[00:57:14] Let's just do it all at once and see where, where we get stuck. And anyway, I had to call another friend who was an extremely good DevOps engineer. To help and, and yeah, anyway, long story short, I couldn't get it to run on the GPU because there were no instances and I only had an A10 24 gig and MixtureFX needs more than that because it's 32 experts.

[00:57:39] So I had to run it on the CPU and that's what we spent the entire day and evening on. And it was really slow, but then we realized, yeah, this is probably like the first time someone has effectively ran Mixture of experts model on on, on a CPU. And again, it's, you can check out the REPL.

[00:57:58] I made a CPU branch and it's the V1 branch if you really want to get it to work. But yeah, that was the story. I just met with a random person from Twitter for the first time who was in their discord and yeah, it was, it was fun. And we also, the funniest part was that. Happened to be there a call in Rafael, who has been teaching about mixture of experts and writing a lot of the papers, and then we look behind and he's just like literally like five dusks away.

[00:58:30] And I was just like, taking a back. It's like, Oh, holy cow. He's here. And he had no idea who we were or anything. So, yeah, that was that was fun.

[00:58:39] Alex Volkov: There, if you don't mind me completing this story from what you told me multiple times, because I think it's like way more colorful than you, than you let on. First of all, VectorLab is the lab of Geoffrey Hinton, the grandfather of AI, right?

[00:58:52] This is the lab. This is like, he's widely considered the person who like, have kickstarted this whole field, basically. Is that, is that, that lab? Was he there?

[00:59:02] Nisten Tahiraj: Yeah, yeah, yeah. Ilyas Iskever has been a student. He wasn't there. He's rarely there. He only has like one PhD, one student under his wing this year.

[00:59:12] So he comes in very rarely. But yeah, Ilya Suskever was not in, in the smaller lab before they moved here. Also Adrian Gomez, the, one of the writers of the Transformers paper still comes there every once in a while. He was there regularly up until Cohere got funded last year. And yeah, this is, this is the lab and it's it's pretty funny because everyone's very, very academic and we're just straight up hacking and whatever we can find.

[00:59:45] Alex Volkov: So the second thing that I wanted to cover here is that exactly what you built in the lab of Geoffrey Hinton, because He's now very public about the AI kind of doomerism and AI different kind of potential bad things that will happen with AI and how to not open source, how to regulate. He's very public.

[01:00:04] He's on every news. And here you are, you and Pharrell are working on an ISO, a bootable AI disc that you literally can run offline that has Lama and offline LLM. The, that basically will say, even if they regulate, you can just like take an old. CPU based machine and run this thing. So you basically get democratizing AI in the lab of the person who's now like very, very vocal about like stopping it.

[01:00:27] So that's, that's the second part that I personally like very enjoy.

[01:00:31] Nisten Tahiraj: It's not just that. Also, if you listen further than what the news media. Shows it's a lot more complex than that. He, he wants people to acknowledge that the risks are real and show that they are mitigating them. But at the same time, he's been doing research to do molecularly grown chips.

[01:00:51] And that architecture first didn't work. So. They're still going full speed ahead. They're just making, the reason that they went that way was just saying to a lot of the community, just don't act like idiots, just regulate yourselves. That, that was why they were challenging that.

[01:01:09] It's it was a lot more complex than people realize. And the professor there, Colin, he's been a big pusher for demarketizing and open sourcing. Model C in general, and so, yeah, it's a lot more, it's a lot more nuanced than what you see in the media, and when you think about it, the safest form of AI that you can have is one that you can just literally unplug, and you have full control over, so there is nothing safer than that.

[01:01:40] Otherwise, you're just trusting some incompetent politician with regulatory or whatever legal hacks to control it. So, it's yeah. It's a lot, I want to say, it's a lot more nuanced than people, than what you've just seen in media snippets and reactionary Twitter checks.

[01:01:58] Alex Volkov: Yeah, I hear you. And definitely we'll, we'll, we'll check out the nuances and Jeffrey Hinton on the topic very briefly before after our apologies, we'll get to in just a second, just like something that also happened this week. Yan Likun, the GOAT, aka, the chief meta AI chief scientist, well, went in front of the Senate. I think a couple of days ago, and he, I just pinned the tweet on the top that he actually retweeted, which was like, sent by notices, and he gave an incredible opening statement, talking about how open sourcing is very important, why the open source LLAMA, talking about the fact that, the open source LLAMA1 and the sky didn't fall, and all of these things, and he also outlined a bunch of the safety protocols that they have into account when they release LLAMA2, and I think it's a First of all, very important to have somebody like Jan in front of Senate and talking about legislators and regulators and about regulation, because we see more and more Jan.

[01:02:52] I think you brought up last week about there was another discussion and Elon Musk was there and Sundar Pichai was there. Everybody was there talking about AI and how to regulate. And I think it's very important to have voices like Jan LeCun talk about like, talk with different things with clarity and safety.

[01:03:07] And so definitely. Recommend everybody to check out his opening statement because you know the doomers it's very easy to scare Especially on like the engaged baiting networks like x and etc It's very easy to like take something that people don't understand use it to scare folks And I think it's very important to have very clear very Credentialed and very like, understanding people from this world to actually explain that there's benefits and explain how open source can benefit as well.

[01:03:36] And I think you also mentioned how excited the open source community was about the Lama to release. And I want to believe that we all had like a small, tiny part to play in, in this. And so, yeah, we're definitely on Yam's map sorry Yam Likun's map and definitely worth checking this out. I think with that listen, thank you for sharing your story.

[01:03:52] Tell us more escapades from vector lab. And if you get to meet Geoffrey Hinton, tell him about Thursday night and also Colin.

[01:03:59] ​

[01:03:59] Alex Volkov: All right folks this actually concludes the two hours that we've allotted for thursday night today I I know there's like many folks. I see dave in the audience. What's up, dave? I haven't I see. I see other folks just stepping in with, with all the sadness of, I want to keep talking with all of you.

[01:04:20] There's also now a need to, transcribe this and, and, and put this into a newsletter, a podcast form Thursday is here every week. We're here literally every week since GPT four came out, I think. Mr. Did I miss one week on vacation? Yeah, newsletter came out, but we didn't talk that week. I felt like, oh, I miss my guys.

[01:04:37] I miss my friends. We need to get up to date together. So we're here every Thursday. I there's so much always to talk about. I want to just like to highlight how much boring this would have been without friends like distance and Nova Arthur, now the new friend of the pod, it's a freer and some other folks who stepped away young and far and for real, like many, many other folks who joined this week to week.

[01:04:58] And Help us bring you, the audience, the best AI news roundup possible on, on X slash Twitter. Now almost six, seven months already into this. This has opened many, many opportunities for many folks on stage, including myself. I'm going to the AI Engineer Conference. As a media person, I'm going to do some spaces from there.

[01:05:19] If you're in the AI Engineer Conference in a couple of weeks, definitely reach out and, we'll talk over there. With that, I want to just say... Without the audience here, this also would be very, very boring. So thank you for joining from week to week. Thank you for listening. Tuning in. Thank you for subscribing.

[01:05:34] Thank you for sharing with your friends. And thank you for leaving comments as well. And with that, I wanna wish you a happy Thursday. I, I'm sure there's going to be many, many, many new things we're listening just today. But you know, we can only cover so much. With that, thank you folks. Have a nice rest of your Thursday.

[01:05:49] I, and we'll meet you here next week. And yeah. Cheers. Have a good one.

[00:00:00] Intro and welcome everyone

[00:00:52] GPT4 - Vision from OpenAI

[00:05:06] Safety concern with GPT4-V

[00:09:18] GPT4 can talk and listen as well

[00:12:15] Apple rumors, on device inference, and Siri

[00:17:01] OpenAI Voice Cloning Tech used in Spotify to translate podcasts

[00:19:44] On the risks of Voice Cloning tech being open sourced

[00:26:07] Alex statement on purpose of ThursdAI

[00:27:53] “AGI has been achieved internally”;

[00:32:10] OpenAI, Jonny Ive and Masa are rumored to be working on a hardware device

[00:33:51] Cloudflare AI - Serverless GPU on global scale

[00:37:13] Cloudflare AI partnership with HuggingFace to allow you to run many models in your own

[00:40:34] Cloudflare announced the Vectorize DB and embedings on edge

[00:46:52] Cloudflare AI gateway - proxy LLM calls, caching, monitoring, statistics and fallback

[00:51:15] Part 2 - intro an recap

[00:54:14] Meta AI announcements, bringing AI agents to 3 billion people next month

[00:56:22] Meta announces EMU image model to be integrated into AI agent on every platform

[00:59:38] Meta RayBan glasses upgraded to spatial computing, with AI and camera access

[01:00:39] On the topic os smart glasses, GoogleGlass, and the acceptance society wide to have

[01:05:37] Safety and societal implications of everyone having glasses and recording everything

[01:12:05] Part 3 - Open Source LLMs, Mistral, QWEN and CapyBara

[01:21:27] Mistral 7B - SOTA 7B general model from MIstralAI

[01:23:08] On the topic of releasing datasets publically and legal challenges with obtaining that

[01:24:42] Mistral GOAT team giving us a torrrent link to a model with Apache 2 license.

Truly, as I’ve been doing these coverages in one form or another for the past 9 months, and I don’t remember a week this full of updates, news, state of the art open source models and more.

So, here’s to acceleration (and me finally facing the fact that I need a niche, and decide what I’ll update on and what I won’t, and also be transparent with all of you about it)

On a separate note, this past two weeks, ThursdAI had exposure to Yann Lecun (RTs), joined on stage by VP of DevRel in Cloudflare and their counterpart in HuggingFace, CEO of Anaconda joined us on stage this episode and we’ve had the chief scientist of Mistral join in the audience 😮 ThursdAI really shapes to be the place where this community meets, and I couldn’t be more humbled and prouder of the show, the experts on stage that join from week to week, and the growing audience 🙇‍♂️ ok now let’s get to the actual news!

ThursdAI - Weeks like this one highlight how important it is to stay up to date on many AI news, subscribe, I’ve got some cool stuff coming! 🔥

All right so here’s everything we’ve covered on ThursdAI, September 28th:

(and if you’d like to watch the episode video with the full transcript, it’s here for free):

Show Notes + Links

* Vision

* 🔥 Open AI announces GPT4-Vision (Announcement, Model Card)

* Meta glasses will be multimodal + AI assistant (Announcement)

* Big Co + API updates

* Cloudflare AI on workers, serverless GPU, Vector DB and AI monitoring (Announcement, Documentation)

* Cloudflare announces partnerships with HuggingFace, Meta

* Claude announces $4 billion investment from Amazon (Announcement)

* Meta announces AI assistant across WhatsApp, Instagram

* Open Source LLM

* 🔥 Mistral AI releases - Mistral 7B - beating LLaMa2 13B (Announcement, Model)

* Alibaba releases Qwen 14B - beating LLaMa2 34B (Paper, Model, Vision Chat)

* AI Art & Diffusion

* Meta shows off EMU - new image model

* Still waiting for DALL-E3 😂

* Tools

* Spotify translation using Open AI voice cloning tech

Vision

GPT 4-Vision

I’ve been waiting for this release since March 14th (literally) and have been waiting and talking about this on literally every ThursdAI, and have been comparing every open source multimodality image model (IDEFICS, LlaVa, QWEN-VL, NeeVa and many others) to it, and none came close!

And here we are, a brief rumor about the upcoming Gemini release (potentially a multimodal big model form Google) and OpenAi decided to release GPT-4V and it’s as incredible as we’ve been waiting for!

From creating components from a picture of UI, to solving complex math problems with LaTex, to helping you get out of a parking ticket by looking at a picture of a complex set of parking rules, X folks report that GPT4-V is incredibly helpful and unlocks so many new possibilities!

Can’t wait to get access, and most of all, for OpenAI to land this in the API for developers to start building this into products!

On the pod, I’ve talked about how I personally don’t believe AGI can work without vision, and how personal AI assistants are going to need to see what I see to be really helpful in the real world, and we’re about to unlock this 👀 Super exciting.

I will add this one last thing, here’s Ilya Sutskever, OpenAI chief scientist, talking about AI + Vision, and this connects with our previous reporting that GPT-4 is not natively multimodal (while we’re waiting for rumored Gobi)

If you need more use-cases, check out this great breakdown by friend of the pod, SkalskiP (Pyotr) who is a vision engineer at RoboFlow which got really high Hacker News rankings.

https://blog.roboflow.com/gpt-4-vision/

Meta RayBan smartglasses will have multimodal AI 👀

To add to the above increased interest about AI (and to rumors about OpenAI working with Jonny Ive from Apple + Masayoshi San about a rumored hardware device) Meta has announced a new iteration of their RayBan glasses, that will include a camera that will help you go live, include an AI agent in the Glasses and most of all, will be multimodal, by which they mean, the AI agent in there (we don’t know if it’s LLaMa based or something else) will have access to the camera, and to what you see.

Given how well this works, it may be revolutionary on it’s own right!

I’ve been on a MultiModality kick since that incredible March 14th day, and I’m very excited that it’s here! 🙌

Big CO + API updates

Cloudflare AI - Serverless GPU inference, VectorDB and AI Gateway

I was blown away by this, so much so, that I’ve hopped on an emergency space on Wednesday, to talk all about this. Some of you know, I’ve created https://targum.video a year ago, and it’s been accepted to CloudFlare workers launchpad. The whole website and backend is on workers, but the GPU and inference, I had to build in python and put on a LambdaLabs GPU machine.

So starting today, folks could build something like Targum, end to end on Cloudflare with the announcement of GPU inference.

If you’d like all the details, I was really humbled to host Ricky Robinette (VP Developer Experience @ Cloudflare) and Phillip Schmidt from Hugging Face join the X space on launch day (to my complete surprise) and you can find that conversation here (it’s going to be on the pod soon after I find some time to edit this 😅)

Here’s my notes from that conversation:

* Inference on edge is here

* Serverless GPUs on cloudflare edge network

* Integrated with Workers platform

* What is the workers platform

* Give example of the many tools it has

* Targum example for what is done on workers and what is done on GPU

* Easy to get started and deploy

* Will have a free tier 🔆

* Models and Usecases

* LLMs - LLaMa 7B

* Embeddings - BGE-base

* Text Classification - DistillBert

* Translation - m2m100

* ASR - Whisper

* Preselected models right now

* Vectorize - an edge native vector DB

* Integrates with wrangler and ecosystem

* Supports existing vectors from OpenAI Ada (importable)

* Metadata can include R2 objects, KV storage and more!

* Build and deploy full RAG apps, including your own local models all inside 1 platform

* AI - gateway

* Proxy for OpenAI (and other providers calls)

* Shows a usage dashboard

* Global Coverage:

* Plan to be in 100 data centers by the end this year

* And nearly everywhere by the end of 2024

* WebGPU in workers

* Many HF models support ONNX

* WebGPU is now supporting FP-16

* This could open a new path to run smaller models within workers even without CFAI

* Partnership with HuggingFace

* 1 click deploy in a dropdown on half a million models

* Serverless inference - no more downloading and uploading

* Way faster as well

* Cloudflare will have a de-facto proxy/mirror of HF? 🤔

I’m very very excited by the HuggingFace partnership and you can hear it in the recording!

Meta announces AI assistant across chat apps, Instagram, WhatsApp, Messenger

I haven’t tested this yet, but this is going to be incredible to make AI experiences to over 3B people around the world!

In addition to just “chat with AI” , Meta has partnered with many celebs to “Embody” them into AI characters, which I found.. a bit unsettling? But I guess we’ll see how much this will affect the “personas” of the AI assistants.

Open Source LLM

Qwen 14B with chat and vision versions

QWEN model from Alibaba, which we’ve already talked about multiple times, then was taken down from the web, comes back, with a vengeance!

Qwen team comes back with a 14B model, that beats LlaMa2 34B on most evaluations, including a VL version (only 7B), which according to my tests, was the best performing open source vision model even at 7B

It was really cool to see the Qwen authors interact with Yam and I on Twitter, it’s like crossing the great firewall and hopefully we’ll have that team on ThursdAI recording at some point!

🔥 Mistral 7B (torrent tweet) - SOTA LLM

Mistral team have made news when they raised $113 million without a product, just 3 co-founders, back in Jun, and the “takes” on twitter were, “we’re in a bubble, bla bla bla” and yesterday, this Goated team just posted a tweet with a magnet torrent link, and no description. So of course everybody downloaded it and found the best SOTA 7B model, that outperforms a much larger LLaMa 2 13B and MUCH larger LLaMa 34B on several benchmarks!

It even comes very close to the Code LLaMa performance benchmarks on code, while being a general model, which is incredible.

Needless to say, the team delivered the promise, and to see them commit this fully to OpenSource, by dropping a modal with Apache 2 license, straight to bit-torrent, is a great sight to see!

Also, we caught glimpes of Guillaume Lample in the audience while we were gassing Mistral up, and potentially at some point we may get Mistral folks to join a ThursdAI live space? 🫡

AI Art + Diffusion

Meta introduced EMU - A diffusion model integrated into it’s AI offerings with a /imagine command, available for free, in all their products, and it looks really good!

I wonder if it will do the same “chat with image” thing as DALL-E3 was announced to do, but in any case, giving this, for free, in this quality, to so many people, is remarkable 🙇‍♂️ Kudos to the team at Meta for ALL the releases today! Can’t wait to play with them.

Tools

Spotify translates podcasts using stealth OpenAI tech

Spotify announced translations for podcast, using some secret OpenAI voice cloning tech, and we had a long discussion about the implication of voice cloning, deep fakes and everything in between with Peter Wang and other folks on the pod, definitely recommended listening!

I love this, absolutely, not just because you may want to listen to ThursdAI pod in your native language (and I could finally show my mom who doesn’t speak English what I’m doing!) but also because language barriers should NOT exist, and Targum.video and this and all the models that Meta is releasing are a great testament to how fast language barriers are coming down!

I’m very very happy with this development and will keep you guys posted on these developments.

With that, I should probably stop here, it’s been an absolutely insane week, and if this summary helped, like, share and consider a premium subscription?

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

P.S - If you scrolled all the way to here, send me 🧨 in a DM on any platform 😉, I may have something for you

Boy am I glad that not all AI weeks are like last week, where we had so much news and so many things happening that I was barely able to take a breath for the week!

I am very excited to bring you this newsletter from San Fancisco this week, the AI mecca, the arena, the place where there are so many AI events and hack-a-thons that I don’t actually know how people get any work done!

On that topic, I’m in SF to participate in the AI.engineer (by swyx and Benjamin Dunphy) next week, to host spaces and interviews with the top AI folks in here, and to discuss with the audience, what is an AI engineer, if you have any questions you’d like me to ask, please comment with them and I’ll make sure I’ll try to answer.

ThursdAI - subscribe eh? ↴

Here’s a table of contents of everything we chatted about:

[00:00:00] Intro and welcome

[00:04:53] Alex in San Francisco - AI Engineer

[00:07:32] Reka AI - Announcing a new multimodal Foundational model called Yasa-1

[00:12:42] Google adding Bard to Google Assistant

[00:18:56] Where is Gemini?

[00:23:06] Arc browser adding Arc Max with 5 new AI features

[00:24:56] 5 seconds link AI generated previews

[00:31:54] Ability to run LLMs on client side with WebGPU

[00:39:28] Mistral is getting love from Open Source,

[00:48:04] Mistral Open Orca 7B

[00:58:28] Acknowledging the experts of ThursdAI

[01:01:14] Voice based always on AI assistants

[01:09:00] Airchat adds voice cloning based translation tech

[01:14:23] Effects of AI voice cloning on society

[01:21:32] SDXL IKEA LORA

[01:23:17] Brief Recap

Show notes:

Big Co

* Google - adding Bard to Google Assistant (Announcement)Come on google, just give us Gemini already!

* Reka AI - Multimodal Yasa-1 from Yi Tay and team (Announcement)With Yi Tay from Flan/Bard fame as chief scientist! But I wasn’t able to test myself!

* Arc - first browser AI features (My thread, Brief video review, Arc Invite)I love Arc, I recommend it to everyone I meet, now with AI preview features it’s even more a non brainer, strongly recommend if you like productivity

Open Source LLMs

* Mistral vs LLaMa 2 boxing match (link)A fun little battle arena to select which responses you personally find better to see the difference between Mistral 7B and LLaMa 13B

* Mistral-7B-OpenOrca (announcement)The folks from Alignment labs do it again! Great finetune that comes very close (98%) to LLaMa 70B on benchmarks!

* SynthIA-7B-v1.3 - (Huggingface)An uncensored finetune on top of Mistral that Reddit claims is a great model, especially since a chain of thought is somehow built in apparently

VISION

* Radiologists thread about GPT-4 V taking over radiology (or maybe not?) (Thread)

Voice

* AirChat added voice clone + translation features (Room, Demo)I’ve been an avid AirChat user (It’s Naval’s social media platform that’s voice based) for a while, and am very excited they are destroying language barriers with this feature!

* Tab was revealed in a great demo by Avi Schiffman (Demo)Go Avi! Rooting for you brother, competition makes folk stronger!

* Rewind announced Rewind Pendant (Announcement)I ordered one, but Rewind didn’t announce a date of when this hits the market, going to be interesting to see how well they do!

Ai Art and Diffusion

- IKEA Lora generate IKEA style tutorials for everything with SDXL (Announcement, HuggingFace)

* DALL-E3 seems to be available to all Plus members now

This weeks pod was generated by talking to chatGPT, it’s so fun, you gotta try it!

No longer breakdown this week ,but we covered a bunch of it in the show, and I highly recommend listening to it!

Don’t forget to follow me on X to be aware of the spaces live from ai.engineer event in SF, the conference will be live-streamed as well on youtube!

See you next week 🫡

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

A week of horror, an AI conference of contrasts

Hi, this is Alex. In the podcast this week, you'll hear my conversation with Miguel, a new friend I made in AI.engineer event, and then a recap of the whole Ai.engineer event I had with Swyx after the end.

This newsletter is a difficult one for me to write, honestly, I wanted to skip this one entirely, struggling to fit the current events into my platform and the AI narrative, however, decided to write one anyway, as the events of the last week have merged into 1 for me in a flurry of contrasts.

Contrast 1 - Innovation vs Destruction

I was invited (among a few other Israelis or Israeli-Americans) to the ai.engineer summit in SF, to celebrate the rise of the AI engineer, and I was looking forward to that very much. Meeting many of you (Shoutout to everyone who listens to ThursdAI who I've met face to face!) and talking to new friends of the pod, interviewing speakers, meeting and making connections was a dream come true.

However a few days before the conference began, in a stark contrast to this dream, I had to call my mom, who was sheltering, 20km from the Gaza strip border, to ask if our friends and family are alive and accounted for, and to hear sirens as rockets flying above her head, as Hamas terrorists murder, pillage and kidnap, in what seems to be the 10x equivalent of 9/11 terror attack, relative to population size.

I grew up in Ashkelon, rocket attacks are nothing new to me, we've learned to live with them (thank you Iron Dome heroes) but this was something else entirely, a new world of terror.

So back to the conference, given that there's not a lot to be gained by doom scrolling, and watching (basically snuff) films coming out of the region, given that all my friends and family were accounted for, I decided to not give the terrorists what they want (which is to get people in state of terror) and instead to choose to have compassion, without empathy towards the situation and not bring sadness to every conversation I had there (over 200 I think)

So participating at an AI event, which hosts and celebrates folks who are literally at the pinnacle of innovation, building the future, using all the latest tools while also hurting and holding the dear ones in my thoughts was a very stark contrast between past and future, and huge credit goes to Dedy Kredo, CTO of Codium, who was in the same position, and gave a hell of a talk, with a kick-ass (no backup recording!) demo live, and then shared this image:

This is his co-founder, Itamar, who was called to reserve duty to protect his family and country, sitting with his rifle and his dashboard, seeing destruction + creation, past and future, negativity and positivity all at once. As Dedy masterfully said, we will prevail 🙏

Contrast 2 - Progress // Fear

At the event, Swyx and Benjamin gave me a media pass and a free reign, and I asked to be teamed with a camera-person to go around the event and do some (not live) interviews. I was teamed with the lovely Stacey, from Chico, CA. Stacey has nothing to do with AI, in fact she's a wedding photographer, however she definitely listened with interest to the interviews I was holding, and to speakers on stage.

While we were taking a break, I looked out the window, and saw a driverless car (waymo) zip by, and since they only started operating after I left SF 3 years ago, I didn't yet have a chance to ride in one.

So I asked Stacey and some other folks, if they'd like to go for a ride, and to my complete bewilderement, Stacey said "no 😳" and when I asked why not, she didn't want to admin but then said that it's scary.

This struck me and since that moment, I've had as many conversations with Stacey as I had with other folks who came to be AI.engineers, since this was such a stark contrast between progress and fear. I basically was walking, almost hand in hand, with a person who doesn't use or understand AI, and fears it, amongst the folks who are building the future, exist at the pinnacle of innovation and discuss how to connect more AI to more AI, and how to build complete autonomous agents to augment human productivity and bring about the world of abundance.

This contrast was supported by several new friends of mine, who came to the AI.engineer and SF for the first time, from countries where English is not the first language, and where Waymo's are not zipping about on the streets freely, and it highlighted for me, how much of this shift is global, and how concentrated the decision making, the building, the innovation is, within the arena, SF, California and US. It's almost expected that AI is going to speak english, and to use/build it, we have to speak it as well, while most of the world doesn't use English as their first language.

Contrast 3 - Technological // Spiritual

This contrast was intimate and personal to me. You see, this ai.engineer event was the first such sized event, professional, with folks talking "my language" since I had burned out this summer. If you've followed for a while, you may remember we talked about Lk-99 and superconductor, and I overclocked myself back then so much (scaling a whole another podcast, hosting 7 spaces in 2 weeks, creating a community of 1,500 and following all the news 24/7) that I had didn't want to go on speaking, doing spaces, recording podcasts... I was just done.

Luckily my friend Junaid sent me a meditation practice recording with the saying "fill your own cup, before you give out to others"

That recording led me to discover effortless meditation (via AirChat, DM me for details), and going on a 5 day free virtual retreat, which changed my life in several very significant ways and just keeps going.

So back to the conference, coming from that place, a place of center, a place where I balance my own inner piece + interact with 150+ people (I think, I haven't really counted) and equalizing energy among them, connecting many folks to many other folks, introducing and getting introduced, giving interviews, remembering names and faces, this was an incredible experience to me, and many of my new friends asked me, what's going on, and why and how I meditate for an hour every day.

So the contrast between my own spiritual journey, and the technological journey was not a very strong one, beyond 1 very... interesting and weird conversation I had, everyone else seemed to be into self improvement, spiritual growth, I've met many folks who went on vipasana retreats, had meditation practices themselves or used to have one and wanted to come back to it.

Maybe it was self selection, but I really enjoyed seeing and hearing all the stories from folks, as I believe that given contrast number 2, the way we talk about AI, the way we build AI, the way we then show AI to the world, needs to take into account compassion towards those who don't understand it, who fear it, and want nothing to do with it.

Not to mention, that I doubt that any of the doomers are walking on their own spiritual path.

I wanted to just send this newsletter but as I always do, I pasted it into Claude.ai, and asked for a review. And Claude suggested I end it on a positive note, because the middle was heavy, and even gave me some suggestion as to how to end it on a positive note. So instead of me doing it, here's an AI, doing a human job on ending something on a positive note:

[Claude] Here are some examples of how you could end your newsletter on a more hopeful, forward-looking note:

* Remind readers of the optimism and excitement you felt at the AI conference, and how seeing the potential of technology firsthand gives you hope for the future.

* Share an insight or realization you had about how to stay grounded and centered even amidst chaos - something your spiritual practices have taught you.

* Affirm that while destruction and violence will always be part of the human experience, you believe innovative thinkers will lead us to a brighter tomorrow.

* Note how connecting deeply with other humans, even strangers, fills you with hope. Finding common ground and shared humanity counterbalances the forces that divide.

* Point to a specific AI-related project or breakthrough that gives you hope for progress. e.g. medical advancements, climate change solutions, etc.

* Share an inspirational quote or verse that reflects resilience and perseverance in difficult times.

* End by expressing gratitude - for your community of readers, your loved ones, the chance to have profound human experiences, or your inner stillness amidst outer turmoil.

The key is ending on a note of hope, possibility, love, resilience - something that uplifts readers after processing weighty material. Let me know if these spark any ideas for how to conclude your newsletter on a high note!

P.S - I know this newsletter is not what you expected, and maybe not what you signed up for, and I deliberated if I even should write it and what if anything should I post on the podcast. However, this week was an incredibly full of contrast, of sadness and excitement, of sorrow and bewilderment, so I had to share my take on all this.

P.P.S - as always, if you read all the way to the end, dm me the ☮️ emoji

Hey friends, welcome to ThursdAI Oct - 19. Here’s everything we covered + a little deep dive after the TL;DR for those who like extra credit.

ThursdAI - If you like staying up to date, join our community

Also, here’s the reason why the newsletter is a bit delayed today, I played with Riffusion to try and get a cool song for ThursdAI 😂

ThursdAI October 19th

TL;DR of all topics covered:

* Open Source MLLMs

* Adept open sources Fuyu 8B - multi modal trained on understanding charts and UI (Announcement, Hugging face, Demo)

* Teknium releases Open Hermes 2 on Mistral 7B (Announcement, Model)

* NEFTune - a "one simple trick" to get higher quality finetunes by adding noise (Thread, Github)

* Mistral is on fire, most fine-tunes are on top of Mistral now

* Big CO LLMs + APIs

* Inflection Pi got internet access & New therapy mode (Announcement)

* Mojo 🔥 is working on Apple silicon Macs and has LLaMa.cpp level performance (Announcement, Performance thread)

* Anthropic Claude.ai is rolled out to additional 95 countries (Announcement)

* Baidu AI announcements - ERNIE 4, multimodal foundational model, integrated with many applications (Announcement, Thread)

* Vision

* Meta is decoding brain activity in near real time using non intrusive MEG (Announcement, Blog, Paper)

* Baidu YunYiduo drive - Can use text prompts to extract precise frames from video, and summarize videos, transcribe and add subtitles. (Announcement)

* Voice & Audio

* Near real time voice generation with play.ht - under 300ms (Announcement)

* I'm having a lot of fun with Airpods + chatGPT voice (X)

* Riffusion - generate short songs with sound and singing (Riffusion, X)

* AI Art & Diffusion

* Adobe releases Firefly 2 - lifelike and realistic images, generative match, prompt remix and prompt suggestions (X, Firefly)DALL-E 3 is now available to all chatGPT Plus uses (Announcement, Research paper!)

* Tools

* LMStudio - a great and easy way to download models and run on M1 straight on your mac (Download)

* Other

* ThursdAI is adhering to the techno-optimist manifesto by Pmarca (Link)

Open source mLLMs

Welcome to multimodal future with Fuyu 8B from Adept

We've seen and covered many multi-modal models before, and in fact, most of them will start being multimodal, so get ready to say "MLLMs" or... we come up with something better.

Most of them so far have been pretty heavy, IDEFICS was 80B parameters etc'

This week we received a new, 8B multi modal with great OCR abilities from Adept, the same guys who gave us Persimmon 8B a few weeks ago, in fact, Fuyu is a type of persimmon tree (we see you Adept!)

In the podcast I talked about having 2 separate benchmarks for myself, one for chatGPT or any MultiModal coming from huge companies, and another for open source/tiny models. Given that Fuyu is a tiny model, it's quite impressive! It's OCR capabilities are impressive, and the QA is really on point (as well as captioning)

An interesting thing about FuYu architecture is, because it doesn't use the traditional vision encoders, it can scale to arbitrary image sizes and resolutions, and is really fast (large image responses under 100ms)

Additionally, during the release of Fuyu, Arushi from Adept authored a thread about visualQA evaluation datasets are, which... they really are bad, and I hope we get better ones!

NEFTune - 1 weird trick of adding noise to embeddings makes models better (announcement thread)

If you guys remember, a "this one weird trick" was discovered by KaiokenDev back in June, to extend the context window of LLaMa models, which then turned into RoPE scaling and YaRN scaling (which we covered in a special episode with the authors)

Well, now we have a similar "1 weird trick" that by just adding some noise to embeddings at training time, the model performance can grow by up to 25%!

The results very per dataset of course, however, considering how easy it is to try, literally:

It's as simple as doing this in your forward pass if training: return orig_embed(x) + noise else: return orig_embed(x)

We should be happy that the "free lunch" tricks like this exist.

Notably, we had a great guest, Wing Lian the maintainer of Axolotl, a very popular tool to streamline fine-tuning, chime in and say that in his tests, and among the discord folks, they couldn't reproduce some of these claims (as they are adding everything that's super cool and beneficial for finetuners to their library) so it remains to be seen how far this "trick" scales, and what else needed to be done here.

Similarly, back when the context extend trick was discovered, there was a lot of debates about it's effectiveness from Ofir Press (author of ALiBi, another context scaling methond) and futher iterations of the trick made into a paper and a robust method, so this develompment is indeed exciting!

Mojo 🔥 now supports Apple silicon Macs and has LLaMa.cpp level performance!

I've been waiting for this day! We've covered Mojo from Modular a couple of times and it seems that the promise behind it starts to materialize. Modular promises an incredible unbelieavable 68,000X boost over vanilla python, and it's been great to see that develop.

Today (October 19) they have released their support of Mojo Lang on Apple silicon which most developers use, and it's a native one and you can use it right now via CLI.

A friend of the pod Aydyn Tairov, hopped on the live recording and talked to use about his LLama.🔥 project (Github) that he ported to the Apple silicon, and showed an incredible, LLaMa.cpp like performance, without crazy optimizations!

Aydyn collected many LLaMa implementations, including Llama.cpp, LLama.c by Karpathy and many others, and included his LLama.mojo (or Llama.🔥) and saw that the mojo one is coming very very close to LLama.cpp and significantly beats Rust and Go and Julia examples (on specific baby llama models)

The Mojo future is bright, and we'll keep updating with more, but for now, go play with it!

Meta is doing near-real time brain → image research! 🤯

We've talked about fMRI signals (and EEG) signals being translated to diffusion imagery before, and this week, Meta has shown that while fMRI signals to brain imagery is pretty crazy on it's own, using something called MEG (non invasive Magnetoencephalography) they can generate and keep generating images based on the brain signals, in near real time!

[TK video here]

I don't have a LOT to say about this topic, besides the fact that as an Aphant (I have Aphantasia) I can't wait to try this on myself and see what my brain actually "sees"

Baidu announces ERNIE and a bunch AI native products including maps, drive, autonomous ride hailing and more.

Baidu has just wrapped up their biggest conference of the year, BaiduWorld, where they announced a new version of their foundational model called ERNIE4, which is a multimodal (of unknown size) and is now integrated into quite a few of their products, many of which are re-imagined with AI.

A few examples beyond a basic LLM chat like interface are, a new revamped map experience with an AI assistant (with voice) built in to help you navigate and find locations, a new office management app that handles appointments and time slots called InfoFlow, and it apparently even does travel booking, to an AI "google drive" like product called YunYidou, that is able to find video content, based on what was said and when, and even pinpoint specific frames, summarize and do a bunch fo other incredible AI stuff, here's a translated video of someone interacting with YunYinou and asking for a bunch of stuff one after another.

Disclosure: I don't know if the video is edited or in real time.

Voice & Audio

Real time voice for agents is almost here, chatGPT voice mode is powerful

I've spent maybe 2 hours this week, with chatGPT in my ear, using the new voice mode + AirPods. It's almost like... being on a call with chatGPT. I started talking to it in the store, asking for different produce to buy for a recipe, then drove home and ask it to "prepare" me for the task (I don't usually cook this specific thing) and then during my cooking, I kept talking to it, asking for next steps. With the new IOS the voice mode shows up as a live activity and you can pause it and resume without opening the app:

It was literally present in my world, without me having to watch the screen or type.

It's a completely new paradigm of interactions when you don't have to type anymore, or pick up a screen and read, and it's wonderful!

Play.ht shows off an impressive <300ms voice generation for agents

After spending almost 2 hours talking to chatGPT, I was thinking, why aren't all AI assistants like this, and the answer was, well... generating voice takes time, which takes you out of your "conversation flow"

And then today, play.ht showed off a new update to their API that generates voice in <300ms, and that can be a clone of your voice, with your accent and all. We truly live in unprecedented times.

I can't wait for agents to start talking and seeing what I see (and remember everything I heard, via Tab or Pendant or Pin)

Riffusion is addictive, generate song snippets with life-like lyrics!

We've talked about music gen before, however, Riffusion is a new addition and is now generating short song segments with VOICE! Here are a few samples, and honestly, I've procrastinated writing this newsletter because it's so fun to generate these, and I wish they went for longer!

AI Art & Diffusion

Adobe releases Firefly2 which is significantly better at skin textures, realism, and hands. Additionally they have added a style transfer which is wonderful, upload a picture of something with a style you'd like, and your prompt will be generated in that style, it works really really well. The extra details on the skin is just something else, though I did cherry pick this example, the other hands were a dead give-away, still, the hands are getting better across the board!

Plus they have a bunch of prompt features, like prompt suggestion, ability to remix other creations and more, it's really quite developed at this point.

Also: DALL-E is now available to 100% of plus users and enterprise, have you tried it yet? What do you think? Let me know in replies!

That’s it for October 19. If you're into AI engineering, make sure you listen to the previous weeks podcast where Swyx and I recapped everything that happened on stage and off it in the seminal AI Engineer summit. And make sure to share this newsletter with your friends who like AI!

For those who are 'in the know', emoji of the week is 📣, please DM or reply with it if you got all the way here 🫡 and we'll see you next week (where I will have some exciting news to share!)

ThursdAI October 26th

Timestamps and full transcript for your convinience

## [00:00:00] Intro and brief updates

## [00:02:00] Interview with Bo Weng, author of Jina Embeddings V2

## [00:33:40] Hugging Face open sourcing a fast Text Embeddings

## [00:36:52] Data Provenance Initiative at dataprovenance.org

## [00:39:27] LocalLLama effort to compare 39 open source LLMs +

## [00:53:13] Gradio Interview with Abubakar, Xenova, Yuichiro

## [00:56:13] Gradio effects on the open source LLM ecosystem

## [01:02:23] Gradio local URL via Gradio Proxy

## [01:07:10] Local inference on device with Gradio - Lite

## [01:14:02] Transformers.js integration with Gradio-lite

## [01:28:00] Recap and bye bye

Hey everyone, welcome to ThursdAI, this is Alex Volkov, I'm very happy to bring you another weekly installment of 📅 ThursdAI.

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

TL;DR of all topics covered:

* Open Source LLMs

* JINA - jina-embeddings-v2 - First OSS embeddings models with 8K context (Announcement, HuggingFace)

* Simon Willison guide to Embeddings (Blogpost)

* Hugging Face - Text embeddings inference (X, Github)

* Data Provenance Initiative - public audit of 1800+ datasets (Announcement)

* Huge open source LLM comparison from r/LocalLLama (Thread)

* Big CO LLMs + APIs

* NVIDIA research new spin on Robot Learning (Announcement, Project)

* Microsoft / Github - Copilot crossed 100 million paying users (X)

* RememberAll open source (X)

* Voice

* Gladia announces multilingual near real time whisper transcriptions (X, Announcement)

* AI Art & Diffusion

* Segmind releases SSD-1B - 50% smaller and 60% faster version of SDXL (Blog, Hugging Face, Demo)

* Prompt techniques

* How to use seeds in DALL-E to add/remove objects from generations (by - Thread)

This week was a mild one in terms of updates, believe it or not, we didn't get a new State of the art open source large language model this week, however, we did get a new state of the art Embeddings model from JinaAI (supporting 8K sequence length).

We also had quite the quiet week from the big dogs, OpenAI is probably sitting on updates until Dev Day (which I'm going to cover for all of you, thanks to Logan for the invite), Google had some leaks about Gemini (we're waiting!) and another AI app builder thing, Apple is teasing new hardware (but nothing AI related) coming soon, and Microsoft / Github announced that CoPilot has 100 million paying users! (I tweeted this and Idan Gazit, Sr. Director GithubNext where Copilot was born, tweeted that "we're literally just getting started" and mentioned November 8th as... a date to watch, so mark your calendars for some craziness next two weeks)

Additionally, we covered the Data provenance initiative that helps sort and validate licenses for over 1800 public datasets, a massive effort led by Shayne Redford with assistance from many folks including friend of the pod Enrico Shippole, we also covered another massive evaluation effort by a user named WolframRavenwolf on the LocalLLama subreddit, that evaluated and compared 39 open source models and GPT4. Not surprisingly the best model right now is the one we covered last week, OpenHermes 7B from Teknium.

Two additional updates were covered, one of them is Gladia AI, released their version of whisper over web-sockets, and I covered it on X with a reaction video, it allows developers to stream speech to text, with very low latency and it's multi-lingual as well, so if you're building an agent that folks can talk to, definitely give this a try, and finally, we covered SegMind SSD-1B, a distilled version of SDXL, making it 50% smaller in size and 60% faster in generation speed (you can play with it here)

This week I was lucky to host 2 deep dive conversations, one with Bo Wang, from Jina AI, and we covered embeddings, vector latent spaces, dimensionality, and how they retrained BERT to allow for longer sequence length, it was a fascinating conversation, even if you don't understand what embeddings are, it's well worth a listen.

And in the second part, I had the pleasure to have Abubakar Abid, head of Gradio at Hugging Face, to talk about Gradio, it's effect on the open source community, and then joined by Yuichiro and Xenova to talk about the next iteration of Gradio, called Gradio-lite that runs completely within the browser, no server required.

A fascinating conversation, if you're a machine learning engineer, AI engineer, or just someone who is interested in this field, we covered a LOT of ground, including Emscripten, python in the browser, Gradio as a tool for ML, webGPU and much more.

I hope you enjoy this deep dive episode with 2 authors of the updates this week, and hope to see you in the next one.

P.S - if you've been participating in the emoji of the week, and have read all the way up to here, your emoji of the week is 🦾, please reply or DM me with it 👀

Timestamps and full transcript for your convinience

## [00:00:00] Intro and brief updates

## [00:02:00] Interview with Bo Weng, author of Jina Embeddings V2

## [00:33:40] Hugging Face open sourcing a fast Text Embeddings

## [00:36:52] Data Provenance Initiative at dataprovenance.org

## [00:39:27] LocalLLama effort to compare 39 open source LLMs +

## [00:53:13] Gradio Interview with Abubakar, Xenova, Yuichiro

## [00:56:13] Gradio effects on the open source LLM ecosystem

## [01:02:23] Gradio local URL via Gradio Proxy

## [01:07:10] Local inference on device with Gradio - Lite

## [01:14:02] Transformers.js integration with Gradio-lite

## [01:28:00] Recap and bye bye

Full Transcription:

[00:00:00] Alex Volkov: Hey, everyone. Welcome to Thursday. My name is Alex Volkov, and I'm very happy to bring you another weekly installment of Thursday. I. This week was actually a mild one in terms of updates, believe it or not. Or we didn't get the new state of the art opensource, large language model this week. However, we did get a new state of the art embeddings model. And we're going to talk about that. we got very lucky that one of the authors of this, a medics model, gold Gina embeddings V2, Bo Wang joined us on stage and gave us a masterclass in embeddings and share some very interesting things about this, including some stuff they haven't charged yet. So definitely worth a listen. Additionally recovered the data provenance initiative that helps sort and validate licenses for over 1800 public data sets. A massive effort led by Shane Redford with assistance from many folks, including a friend of the pod. Enrico Shippole.

[00:01:07] we also covered the massive effort by another user named Wolf from Ravenwolfe on the local Lama subreddit. Uh, that effort evaluated and compared to 39 open source models ranging from 7 billion parameters to 70 billion parameters and threw in the GPT4 comparison as well. Not surprisingly, the best model right now is the one we covered last week from friends of the politic new called open Hermes seven B.

[00:01:34] Do additional updates we've covered. One of them is Gladia AI, a company that offers transcription and translation APIs release their version of whisper over WebSockets. So live transcription, and I covered it on X with a reaction video. And I'll add that link in the show notes. It allows developers like you to stream speech, to text and. Very low latency and high quality and it's multi-lingual as well. So if you're building an agent that your users can talk to. Um, definitely give this a try. And finally Segmind segued mind accompany that just decided to open source a distilled version of. SDXL, making it 50% smaller in size and the in addition to that 60% faster in generation speed. The links to all these will be in the show notes.

[00:02:23] But this week I was lucky to host two deep dives, one with Bo Weng which I mentioned. Uh, we've covered the embeddings vector led in spaces that dimensionality and how they retrained Bert model to allow for a longer sequence length. It was a fascinating conversation. Even if you don't understand what embeddings are, it's well worth the listen. And, , I learned a lot. Now I hope you will, as well. And the second part, I had the pleasure to have a Brubaker a bit. The head of grandio at hugging face to talk about gradient. What is it? Uh, its effect on the open source community. And then joined by utero. And Sunnova to talk about the next iteration of Grigio called Grigio light that runs completely within the browser. No Serra required. We also covered a bit of what's coming to Gradio in the next release. on October 31st.

[00:03:15] A fascinating conversation. If you're a machine learning engineer, AI engineer, or just somebody who's interested in this skilled. You've probably used radio, even if you haven't written any Gradio apps, every model and hugging face usually gets a great deal demo.

[00:03:30] And we've covered a lot of ground, including M scripting. Then by filling the browser. As a tool for machine learning, web GPU, and so much more.

[00:03:38] Again, fascinating conversation. I hope you enjoy this deep dive episode. Um, humbled by the fact that sometimes the people. Who produced the updates we cover actually come to Thursday and talk to me about the things they released. And I hope this trend continues, and I hope you enjoyed this deep dive over an episode. And, um, I'll see you in the next one. And now I give you thursday october 26. oh, awesome. It looks like Bo, you joined us. Let's see if you're connecting to the audience, and can you unmute yourself, can you see if we can hear you?

[00:04:22] Bo Wang: Hi, can you hear me? Oh, we can hear you fine, awesome. this, this, this feature of, of Twitter.

[00:04:30] Alex Volkov: That's awesome. This, this usually happens, folks join and it's their first face and then they can't leave us. And so let me just do a little, maybe... Maybe, actually, maybe you can do it, right? Let me just present yourself.

[00:04:42] I think I followed you a while ago, because I've been mentioning embeddings and the MTB dashboard and Hug and Face for a while. And, obviously, embeddings are not a new concept, right? We started with Word2Vec ten years ago, but now, with the rise of LLMs, And now with the rise of AI tools and many people wanting to understand the similarity between the user query and an actual thing they, they, they stored in some database, embeddings have seen a huge boon.

[00:05:10] And also we've saw like all the vector databases pop up like mushrooms after the rain. I think Spotify just released a new one. And my tweet was like, Hey, do we really need another vector database? But Boaz, I think I started following you because you mentioned that you were working on something that's.

[00:05:25] It's coming very soon, and finally this week this was released. So actually, thank you for joining us, Beau, and thank you for doing the first ever Twitter space for yourself. How about can we start with your introduction of who you are and how are you involved with this effort, and then we can talk about Jina.

[00:05:41] Bo Wang: Yes, sure. Basically I have a very different background. I guess I was oriJinally from China, but my bachelor was more related to text retrieval. I have a retrieval experience rather than pure machine learning background, I would say. Then I came to the Europe. I came to the Netherlands like seven or eight years ago as a, as an international student.

[00:06:04] And I was really, really lucky and met my supervisor there. She basically guided me into the, in the world of the multimedia information retrieval, multimodal information retrieval, this kind of thing. And that was around 2015 or 2016. So I also picked up machine learning there because when I was doing my bachelor, it's not really hot at that moment.

[00:06:27] It's like 2013, 2014. Then machine learning becomes really good. And then I was really motivated, okay, how can I apply machine learning to, to search? That is, that is my biggest motivation. So when I was doing my master, I, I collaborated with my friends in, in, in the US, in China, in Europe. We started with a project called Match Zoo.

[00:06:51] And at that time, the embedding on search is just a nothing. We basically built a open source. Software and became at that time the standard of neural retrieval or neural search, this kind of thing. Then when the bird got released, then our project basically got queue because. Everyone's focus basically shifted to BERT, but it's quite interesting.

[00:07:16] Then I graduated and started to work as a machine learning engineer for three years in Amsterdam. Then I moved to Berlin and joined Jina AI three years ago as a machine learning engineer. Then basically always doing neural search, vector search, how to use machine learning to improve search. That is my biggest motivation.

[00:07:37] That's it.

[00:07:38] Alex Volkov: Awesome. Thank you. And thank you for sharing with us and, and coming up and Gene. ai is the company that you're now working and the embeddings thing that we're going to talk about is from Gene. ai. I will just mention the one thing that I missed in my introduction is the reason why embeddings are so hot right now.

[00:07:53] The reason why vectorDB is so hot right now is that pretty much everybody does RAG, Retrieval Augmented Generation. And obviously, For that, you have to store some information in embeddings, you have to do some retrieval, you have to figure out how to do chunking of your text, you have to figure out how to do the retrieval, like all these things.

[00:08:10] Many people understand that whether or not in context learning is this incredible thing for LLMs, and you can do a lot with it, you may not want to spend as much tokens on your allowance, right? Or you maybe not have enough in the context window in some in some other LLMs. So embeddings... Are a way for us to do one of the main ways to interact with these models right now, which is RAC.

[00:08:33] And I think we've covered open source embeddings compared to OpenAI's ADA002 embedding model a while ago, on ThursDAI. And I think It's been clear that models like GTE and BGE, I think those are the top ones, at least before you guys released, on the Hugging Face big embedding model kind of leaderboard, and thank you Hugging Face for doing this leaderboard.

[00:09:02] They are great for open source, but I think recently it was talked about they're lacking some context. And Bo, if you don't mind, please present what you guys open sourced this week, or released this week, I guess it's open source as well. Please talk through Jina Embeddings v2 and how it differs from everything else we've talked about.

[00:09:21] Bo Wang: Okay, good. Basically, it's not like embeddings for, how can I say, maybe two... point five years. But previously we are doing at a much smaller scale. Basically we built all the algorithm, all the platform, even like cloud fine tuning platform to helping people build better embeddings. So there is a not really open source, but a closed source project called fine tuner, which we built to helping user build better embeddings.

[00:09:53] But we didn't, we found it okay. Maybe we are maybe too early. because people are not even using embeddings. How could they find embeddings? So we decided to make a move. Basically, we basically scaled up our how can I say ambition. We decided to train, train our own embeddings. So six months ago, we started to train from scratch, but not really from scratch because in binding training, normally you have to train in two stages.

[00:10:23] The first stage, you need to pre train on massive scale of like text pairs. Your objective is to bring these text pairs as closer as possible, as possible, because these text pairs should be semantically related to each other. In the next stage, you need to fine tune with Carefully selected triplets, all this kind of thing.

[00:10:43] So we basically started from scratch, but by collecting data, I think it was like six months ago, we working with three to four engineers together, basically scouting every possible pairs from the internet. Then we basically created like one billion, 1. 2 billion sentence pairs from there. And we started to train our model based on the T5.

[00:11:07] Basically it's a very popular encoder decoder model. This is on the market. But if you look at the MTB leaderboard or all the models on the market, the reason why they only support 512 sequence lengths is constrained actually by the backbone itself. Okay, we figure out another reason after we release the V1 model.

[00:11:31] Basically, if you look at. And the leaderboard or massive text embedding leaderboard, that is the one Alex just mentioned. Sorry, it's really bad because everyone is trying to overfitting the leaderboard. That naturally happens because if you look at BGE, GTE, the scores will never that high if you don't add the training data into the, into the, That's really bad.

[00:12:00] And we decided to take a different approach. Okay. The biggest problem we want to solve first, improving the quality of the embeddings. The second thing we want to solve is. Enable user to making longer context lens. If we want to making user make user have longer context lens, so we have to rework the BERT model, because every basically the embedding model, the backbone was from BERT or T5.

[00:12:27] So we basically started from scratch. Why not we just borrow the latest research from large language model? Every large language model wants large context. Why not we just borrow the research ideas? into the musk language modeling modelings. So we basically borrowed some ideas, such as rotary position embeddings or alibi, maybe you did, and reworked BERT.

[00:12:49] We call it JinaBERT. So basically now the JinaBERT can handle much longer sequence. So we trained BERT from scratch. Now BERT has been a byproduct of our embeddings. Then we use this JinaBERT to contrastively train the models on the semantic pairs and triplets that finally allow us to encode 8K content.

[00:13:15] Alex Volkov: Wow, that's impressive. Just, just to react to what you're saying, because BERT is pretty much every, everyone uses BERT or at least use BERT, right? At least in the MTB leaderboard. I've also noticed many other examples that use BERT or distilled BERT and stuff like this. You're saying, what you're saying, if I'm understanding correctly, is this was the limitation for sequence length?

[00:13:36] for other embedding models in the open source, right? And the OpenAI one that's not open source, that does have 8, 000 sequence length. Basically, sequence length, if I'm explaining correctly, is just how much text you can embed without chunking.

[00:13:51] Yes. And you're basically saying that you, you guys saw this limitation and then retrained BERT to use rotary embeddings. We've talked about rotary embeddings multiple times here. We had folks behind the yarn paper for extending context windows. Alibi is we follow Ophir Press.

[00:14:08] I don't think Ophir ever joined ThursdAI, but Ophir, if you hear this, you're welcome to join as well. So Alibi is another way to extend context windows and I think Mosaic folks used Alibha and some other folks as well. Bo, could you speak more about like borrowing the context from there and retraining BERT to JinaBERT and whether or not JinaBERT is also open source?

[00:14:28] Bo Wang: Oh, we actually want to make JinaBERT open source, but I need to align with my colleagues. That's, that's, that's really, that's a decision to be made. And the, the idea is quite naive. If you didn't know, I don't want to dive into too much about technical details, but basically the idea of Alibi basically removed the position embeddings from the large language model pre training.

[00:14:55] And the Alibi technique allow us to train on the shorter sequences. But inference at every very long sequence. So in the end, I think if I, my remember is correct, the author of alibi paper, basically trained model on 512 sequence lens and 1,024 sequence lens, but he's able to inference on 16 K. 16 K, like sequence lens.

[00:15:23] If you further expand it, you are not capable because that's the limitation of hardware, that's the limitation of GPE. So he, he actually tested 16 K like a sequence lens. So what we did is just. Borrowed this idea from the autoregressive models into the mask language models. And integrate Alibi, remove the position embeddings from the bird, and add this Alibi slope and all the Alibi stuff back into the bird.

[00:15:49] And just borrowed the things how we train bird or something Roberta, something from Roberta, and retrained the bird. I never imagined bird could be a by product of our embedding model, but this... This happened. We could open source it. Maybe I have to discuss with my colleague.

[00:16:09] Alex Volkov: Okay. So when you talk to your colleagues, tell them that first of all, you already said that you may do this on ThursdAI Stage.

[00:16:15] So your colleagues are welcome also to join. And when you open source this, you guys are welcome to come here and tell us about this. We love the open source. The more you guys do, the better. And the more it happens on ThursdAI Stage, the better, of course, as well. Bo, you guys released the Jina Embedding Version 2, correct?

[00:16:33] Gene Embedding Version 2 has a sequence length of 8k tokens. So that actually allows to, if, just for folks in the audience, 8, 000 tokens is, I want to say, maybe like 6, 000 words in English around, right? And different languages as well. Could you talk about multilinguality as well? Is it multilingual, is it only English?

[00:16:53] How that how that appears within the embedding model?

[00:16:57] Bo Wang: Okay, actually, our Jina Embedding V2 is only English, so it's a monolingual embedding model. If you look at the MTV benchmark or all the public multilingual models, they are multilingual. But to be frankly, I don't think this is a fair solution for that.

[00:17:18] I think at least every major language.

[00:17:24] We decided to choose another hard way. We will not train a multilingual model, but we will train a bilingual model. Our first target will be German and Spanish. What we are doing at Jina AI is we basically Fix our English embedding model as it is just keep it at is, but we are continuously adding the German data, adding the Spanish data into the embedding model.

[00:17:51] And our embedding model cares two things. We make it bilingual. So it's either German, English or German English, Spanish, Spanish, English, German, English, or Japanese, English, whatever. And what we are doing is we want to build this embedding model to make it monolingual. So imagine you are, you have a German English embedding model.

[00:18:12] So if you search for German, you'll get German results. If you use English, you'll get English results. But we also care about the cross linguality of this bilingual model. So imagine you, you, you encode two, two sentences. One is in German, one is in English, which they are With the same meaning, we also want these vectors to be mapped into the similar semantic space.

[00:18:36] Because I, I'm a foreigner myself, sometimes, imagine I, I, I buy some stuff in the supermarket. Sometimes I have to translate, use Google Translate, for example, milk into Milch in German, then, then, then put it into the search box. I really want this bilingual model happen. And I believe every, at least, major language deserves such an embedding model.

[00:19:03] Alex Volkov: Absolutely. And thanks for clarifying this because one of the things that I often talk about here on Thursday Night is as a founder of Targum, which is inside videos, is just how much language barriers are preventing folks from conversing to each other. And definitely embeddings are... The way people extend memories parallel lines, right?

[00:19:21] So like a huge, a huge thing that you guys are working on and especially helpful. The sequence length is, and I think we have a question from the audience is what is the sequence lengths actually allow people to do? I guess Jina and I worked with some, some other folks in the embedding space. Could you talk about what is the longer sequence lengths now unlocking for people who want to use open source embeddings?

[00:19:41] Obviously. My answer here is, well, OpenAI's embeddings is the one that's most widely used, but that one you have to do online, and you have to send it to OpenAI, you have to have a credit card with them, blah, blah, blah, you have to be from supported countries. Could you talk about a little bit of what sequence length allows unlocks once you guys release something like this?

[00:20:02] Bo Wang: Okay, actually, we didn't think too much about applications. Most of the vector embeddings applications, you can imagine search and classification. You build another layer of, I don't know, classifier to classify items based on the representation. You can build some clustering. You can do some anomaly detection on the NLP text.

[00:20:22] This is something I can imagine. But the most important thing I I have to be frankly to you because we are, we are like writing a technical report as well. Something like a paper maybe we'll submit to academic conference. Longer embeddings doesn't really always work. That is because sometimes if the important message is in in the front of the document you want to embed, then it makes most of the sense just to encode let's say 256 tokens.

[00:20:53] or 512. But sometimes if you you have a document which the answer is at the middle or the end of the document, then you will never find it if the message is truncated. Another situation we find very interesting is for clustering tasks. Imagine you want to visualize your embeddings. Longer longer sequence length almost always helps and for clustering tasks.

[00:21:21] And to be frankly, I don't care too much about the application. I think people, we, what we're offering is the, how can I say, offering is, is like a key. We, we unlock this 512 sequence length. To educate and people can explore it. People, let's say I, I only need two K then, then people just set tokenize max lens to two k.

[00:21:44] Then, then embed. Based on their needs, I just don't want to be, people to be limited by the backbone, by the 500 to 12 sequence lengths. I think that's the most important thing.

[00:21:55] Alex Volkov: That's awesome. Thank you. Thank you for that. Thank you for your honesty as well. I love it. I appreciate it. The fact that, there's research and there's application and you not necessarily have to be limited with the application set in mind.

[00:22:07] We do research because you're just opening up doors. And I love, I love hearing that. Bo maybe last thing that I would love to talk to you about as the expert here on the topic of dimensions. Right. So dimensionality with embeddings I think is very important. Open the eye, I think is one of the highest ones.

[00:22:21] The kind of the, the thing that they give us is like 1200 mentioned as well. You guys, I

[00:22:26] think

[00:22:26] Jina is around 500. Or so is that correct? Could you talk a bit about that concept in broad strokes for people who may be not familiar? And then also talk about the why the state of the art OpenAI is so far ahead?

[00:22:39] And what will it take to get the open source embeddings also to catch up in dimensionality?

[00:22:46] Bo Wang: You mean the dimensionality of the vectors? Okay, basically we follow a very standard BERT size. The only thing we modified is actually the the alibi part and some training part.

[00:22:58] And our small model dimensionality is 512, and the base model is 768 and we have also a large model, haven't been released because of the training is too slow. We have so much data to change. Even the model size is small, but we have so much data and so large model dimensionality size is 1,024. And if my memory is correct, so are I embedding 0 0 2?

[00:23:23] Have but dimensionality of. 1, 5, 3, 6, something like that, which is a very strange dimensionality, I have to say, but I would say the dimensionality is, is, is the longer might be more Better or more expressive, but shorter, which means when you are doing the vector search, it's gonna be much more faster.

[00:23:48] So it's something you have to balance. So if you think the speed query speed, or the retrieval speed or whatever is more important to you. And if I, if I know correct, some of the Vector database, they make money by the dimensionality, let's say. They, they charge you by the dimensionality, so it's actually quite expensive if your dimensionality is too high.

[00:24:13] So it's a balance between expressionist and the, the, the, the speed and the, the, the, the cost you want to invest. So it's. It's very hard to determine, but I think 512, 768, and 1024 is very common as BERT.

[00:24:34] Alex Volkov: So great to hear that a bigger model is also coming, but it hasn't been released yet. So there's like the base model and the small model for embeddings, and we're waiting for the next one as well.

[00:24:46] I wanted to maybe ask you to maybe simplify for the audience, the concept of dimensionality. What does it mean between, what is the difference between embeddings that were stored with 512 and like 1235 or whatever OpenAI does? What does it mean for quality? So you mentioned the speed, right? It's easier to look up nearest neighbors, maybe within the 512 dimension space, what does it actually mean for quality of look up of different other ways that strings can compare? Could you maybe simplify the whole concept, if possible, for people who don't speak embeddings?

[00:25:19] Bo Wang: Okay maybe let me quickly start with the most basic version.

[00:25:24] If you imagine, if you type something in the search box right now, when doing, doing the matching, and it's actually also embedding, but it's something like if I make a simple version, it's a binary embedding. Imagine there 3, 000 words in English. Maybe there are much more, definitely. Imagine it's 3, 000 words in English, then the vector is 3, 000 dimensionality.

[00:25:48] Then what current solution of searching or matching do is just making... If the query has a token, if your document has a token, if your document has this token, then your occurrence will be one. If you query has the token, and this one will match your document token. But it's also about the, the frequency it appears, it's how, how rare it is.

[00:26:12] But the current solution is basically matching by the. By the English word, but with neural network, basically if you know about this, for example, ResNet know about a lot of different, for example, classification models, basically the output class of item, but if you chop up the classification layer, it will give you some a vector.

[00:26:36] Basically this vector is It's the representation of the information you want to encode. Basically it's a compressed version of the information in a certain dimensionality such as 512, 768, something like this. So it's a compressed list of non numerical numbers, which we normally call it dense vectors.

[00:26:57] because it's much more how can I say in English dense, right? Compared to the traditional way we store vectors, it's much more sparse. There is a lot of zero, there is a lot of one, because zero means not exist, one means exist. When one exists, then there is a match, then you've got the search result.

[00:27:16] So these dense vectors capture more about semantics, but if you match by the occurrence, then you might lose the semantics. But only matching by the occurrence of a token or a word.

[00:27:31] Alex Volkov: Thank you. More dimensions, basically, if I'm not saying it correctly, more dimensions just have more similarity vector. So like more things two strings or tokens can be similar on. And this basically means higher match rate. For more similarity things. And I think the basic stuff I think is covered in the Simon Wilson, the first pin tweet here, Simon Wilson did a basic, basic intro into what do dimensions embeddings mean and why they matter.

[00:28:00] And I specifically love the fact that there's arithmetic that can be done. I think somebody reads the paper even before this whole LLM thing, where if you take embeddings for Spain and embeddings for Germany, and then you take you, you can subtract like the embedding for Paris and then you get something closer to, to like Berlin, for example, right?

[00:28:19] So there's like concepts in, inside these things that are they're even arithmetic works and if you take like King and you subscribe male, then you get something closer to Queen and stuff like this. It's really, really interesting. And also Bo, you mentioned visualization as well. It's really impossible to visualize.

[00:28:36] 10, 24, et cetera, dimensions, right? Like we humans, we have perceived maybe three, maybe three and a half, four with time, whatever. And usually what happens is those multiple dimensions get down scaled to 3D in order to visualize in neighborhoods. And I think we've talked with folks from ARISE. They have a software called Phoenix that allows you to visualize embeddings for clustering and for semantics.

[00:29:02] Atlas does this as well, right? Nomic AI's Atlas does this as well. You can provide dimensions as well. And so you can provide embeddings and see clustering for concepts. And it's really pretty cool. If you haven't played with this, if you only did VectorDBs and you stored your stuff after you've done chunking, but you've never visualized how this looks, I strongly recommend you to do and I think well, thank you so much for joining us and explaining to us, the internals and sharing with us some exciting things about what's to come. Jina Burt is hopefully hopefully is coming, a, a retrained version of Burt, the, the, the, the, the... The grease of all how should I say, I can't, it's hard for me to define a verb, but I see it everywhere it's, it's the big base bone of a lot of NLP tasks, and it's great to see that you guys are about to first of all, retrain it for longer sequences, using tricks like Alibi and and I think you said Positional Embeddings, and hoping to see some open source action from this, but also that Jina Embedding's large model is coming as well with more dimensions waiting for that. Hopefully you guys didn't stop training that. And I just want to tell folks why I'm excited for this. And this kind of will take us to the next.

[00:30:08] Point as well is because, while I love OpenAI, I honestly do, I'm going to join their Dev Day, I'm going to report from their Dev Day and tell you all the interesting things that OpenAI does. We've been talking about we've been talking and we'll be talking today about local inference, about running models on edge, about running models of your own.

[00:30:28] Mistin is here, he even works on some bootable stuff that you can like completely off the grid run. And, so far, we've been focused on open source LLMs, for example, right? So we've had I see Pharrell in the audience from Skunks Works, and many other fine tuners, like Tignium, Alignment Labs, all these folks are working on local LLMs, and they never get to GPT 4 level yet.

[00:30:51] We're waiting for that, and they will. But the whole point of them is, you run them locally, they're uncensored, you can do whatever you want, you can fine tune them on whatever you want. However, the kind of the embeddings part Is the glue to connect it to an application and the reason is because there's only so much context window also context window is expensive and even if theoretically the yarn paper that we've talked with the authors of allows you to extend the context window to 128, 000 tokens The hardware requirements for that are incredible, right?

[00:31:22] Everybody in the world of AI engineers, they switch up to, to, to retrieval of data generation. Basically, instead of shoving everything in the context, they switched Hey, let's use a vector database. Let's say a Chroma. Or Pinecone, or Waviate, like all of those, vectorized from Cloudflare, and the other one from Spotify there, I forget its name or even Superbase now has one.

[00:31:43] Everybody has a vector database it seems these days, and the reason for that is because all the AI engineers now understand that you need to put some text into some embeddings, store them in some database. And many pieces of that were still requiring internet, requiring OpenAI API calls, requiring credit cards, like all these things.

[00:32:03] And I think it's great that we've finally got to a point where, first of all there are embeddings that are matching whatever OpenAI has given us. And now you can run them locally as well. You don't have to go to OpenAI. If you don't want to host, you can probably run them. I think though GeneEmbedding's base is very tiny.

[00:32:20] Like it's half like the small model is 770 megabytes, I think. Maybe a little bit more, if

[00:32:27] Bo Wang: I'm looking at this correctly. Sorry, it's half precision. So you need to double it to make it FP32.

[00:32:33] Alex Volkov: Oh yeah, it's half precision. So it's already quantized, you mean?

[00:32:37] Bo Wang: Oh no, it's just to store it as FV16,

[00:32:39] Alex Volkov: if you store it as FV16.

[00:32:43] Oh, if you store it as FV16. But the whole point is the next segment in ThursdAI today is going to be less about updates and more about the very specific things. We've been talking about local inference as well, and these models are tiny, you can run them on your own hardware, on Edge via Cloudflare, let's say, or on your computer.

[00:32:58] And you now can do almost end to end application wise. From the point of your user inputting a query embedding this query, running a match, a vector search, KNNN and whatever you want nearest neighbor search for that query for the user. Retrieve that all from like local open source. You basically you, you can basically go offline.

[00:33:20] And this is what we want in, in the era of upcoming regulation towards what AI can be and cannot be. And the era of like open source models getting better and better. We've talked last week where Zephyr and I think Mistral News from Technium is also matching some GPT 3. 5. All of those models you can download and nobody can tell you not to run inference on them.

[00:33:40] Hugging Face open sourcing a fast Text Embeddings Inference Server with Rust / Candle

[00:33:40] Alex Volkov: But the actual applications, they still require the web or they used to. And now I'm, I'm loving this like new move towards. Even the application layer, even the RAG systems, which are augmented generation, even the vector databases, and even the embeddings are now coming to, to open source, coming to your local computer.

[00:33:57] And this will just mean like more applications either on your phone or your computer. And absolutely love that. Bo, thank you for that. And thank you for coming to the stage here and talking about the things that you guys open sourced and hopefully we'll see more open source from Jina and everybody should follow you and, and Jina as well.

[00:34:13] Thank you. It looks like. Thank you for joining. I think the next thing that I wanna talk about is actually in this vein as well. Let me go find this o Of course, we love hug and face and the thing that I think that's already on top if you look, yeah if you look at the last thing, last tweet that's pinned it's a tweet from Jeri Lou from Lama Index, obviously.

[00:34:33] Well, well, well, we're following Jerry and whatever they're building and doing over at Lama Index because they implement everything like super fast. I think they also added support for Jina like extremely fast. He talks about this thing where HugInFace opensource for us something in Rust and Candlestick?

[00:34:51] Candlelight? Something like that? I forgot that they're like iteration on top of Rust. Basically, the open source is a server that's called TextEmbeddingsInferenceServer that you can run on your hardware, on your Linux. boxes and basically get the same thing that you get from OpenAI Embeddings.

[00:35:07] Because Embeddings is just one thing, but it's a model. And I think you could use this model. You could use this model with transformers but it wasn't as fast. And as Bo previously mentioned, there's considerations of latency for user experience, right? If you're building an application, you want it to be as responsive as possible.

[00:35:24] You need to look at all the places in your stack and say, Hey. What slows me down? For many of us, the actual inference, let's say use GPT 4, waiting on OpenAI to respond and stream that response is what slows many applications down. And but many people who do embeddings, let's say you have a interface of a chat or a search, you need to embed every query the user sends to you.

[00:35:48] And one such slowness there is how do you actually How do you actually embed this? And so it's great to see that Hackenface is working on that and improving that. So you previously could do this with transformers, and now they released this specific server for embeddings called TextEmbeddings Inference Server.

[00:36:04] And I think it's four, four times faster. than the previous way to run this, and I absolutely love it. So I wanted to highlight this in case you are interested. You don't have to, you can use OpenAI Embeddings. Like we said, we love OpenAI, it's very cheap. But if you are interested in doing the local embedding way, if you want to go end to end, complete, like offline, you want to build like an offline application, using their internet server I think is a good idea.

[00:36:29] And also it shows what HuggingFace is doing with Rust and I really need to remember what language there is but definitely a great attempt from Hug and Face, and yeah, just wanted to highlight that. Let's see. Before we are joined from the Grad. io folks, and I think there's some folks in the audience who are ready from Grad.

[00:36:48] io to come up here and talk about local inference which 15 minutes left,

[00:36:52] Data Provenance Initiative at dataprovenance.org

[00:36:52] Alex Volkov: I wanted to also mention the Data Provenance Initiative. Let me actually find this announcement, and then quickly... Quickly paste this here, and I was hoping that Enrico can be here. . There's a guy named Shane Longfree,

[00:37:05] and he released this massive, massive effort, included with many people. And basically what this effort is, it's called the Data Provenance Initiative. Data Provenance Initiative is now existing in dataprovenance. org. And hopefully can somebody maybe send me the, the direct link to the suite to add this.

[00:37:23] It... It is a massive effort to take 1, 800, so 1, 800 Instruct and Align datasets that are public, and to go through them to identify multiple things. You can filter them, exclude them, you can look at creators, and the most important thing, you can look at licenses. Why would you do this? Well, I don't know if somebody who builds an application needs this necessarily, but everybody who wants to fine tune models, the data is the most important key for this, and building data sets and running them through your fine tuning efforts is basically the number one thing that many people do in the fine tune community, right?

[00:38:04] Data wranglers, and now, thank you, Nishtan, thank you so much, and a friend of the pod, Enrico. is now pinned to the top of the tweet. Thank you for to the top of the space, the nest, whatever it's called. A friend of Enrico Cipolla, who we've talked previously in the context of extending I think Lama to first 16k and then 128k.

[00:38:24] I think Enrico is part of the team on yarn paper as well. I joined this effort, and I was hoping Enrique could join us to talk about this. But basically, if you're doing anything with data, this seems like a massive, massive effort. Many datasets from Lion, and we've talked about Lion, and Alpaca, GPT 4L Gorilla, all these datasets.

[00:38:46] It's very important when you release your model as open source that you have the license to actually release this. You don't want to get exposure, you don't want to get sued, whatever. And if you're in finding data sets and creating different mixes to fine tune different models, this is a very important thing.

[00:39:03] And we want to shout out, Shane Longpre, Enrico, and everybody who worked on this because I think... Just, I love these efforts for the open source, for the community, and it just makes, it's easier to fine tune, to train models. It makes it easier for us to advance and get better and smaller models, and it's worth celebrating and ThursdAI is the place to celebrate this, right?

[00:39:27] LocalLLama effort to compare 39 open source LLMs + GPT4

[00:39:27] Alex Volkov: On the topic of extreme, how should I say efforts that are happening by the community on the same topic, I want to add another one, and this one I think I have a way to pull it up, so give me just a second give me just a second, yes. A Twitter user named Wolfram Ravenwolf who is a participant of the local Lama community on Reddit and now is pinned to the nest at the top of the tweet did this massive effort of comparing open source LLMs and tested 39 different models ranging from 7 billion parameters to 70 billion, and also compared them to chat GPT, GPT 4.

[00:40:06] And I just want to circle back to something we've said. In the previous space as well, and I welcome like folks on stage also to jam in here. I've seen also the same kind of concepts from Hug and Face folks. I think Glenn said the same thing. It's really unfair to, to take a open source model like Mistral7b and then start comparing this to GPT 4.

[00:40:26] It's unfair for several reasons. But also I think it, it, it can obfuscate to some people when they do this comparison of how, just how advanced we've come for the past year in open source models. OpenAI has the infrastructure, they're backed by Microsoft, they have the pipelines to serve these models way faster.

[00:40:47] And also, those models don't run on like local hardware, they don't run on like one GPU. It's like a whole, a whole... Amazing MLOps effort to bring you this speed. When you're running local source model open source models locally when they're open source, they're, they're, they're small, there's drawbacks and there's like takeaways that you have to bake in into your evaluation.

[00:41:09] So comparing to the GPT 4, which is super general in many, many things, that will just lead to your disappointment. However, and we've been talking about this like with other open source models If you have a different benchmark in your head of if you're comparing open source to open source, then it's a whole completely different ballgame.

[00:41:26] And then you start seeing things like, Hey, we're noticing that the 7 billion parameter model is, beating 70 billion. We're noticing that size is not necessarily the king because if you guys remember, Three months ago, ni, I wanna say we've talked about Falcon 180 B. 180 B was like, three times the size of like the, the, the next largest model.

[00:41:47] And it was incredible the Falcon Open source this, and then it was like, like a wo like, no, nobody really was able to run 180 B because it's huge. But also once, once we did run it, we saw that like the difference between that and LAMA are not great at all. Maybe a few percentage points on, on the valuations.

[00:42:04] However, the benefits that we see are from local, like for tinier and tinier models from like 7D Mistral, for example, which is the, the one that the fine tuners of the world are now preferring to everything else. And so the kind of, when you're about to evaluate whatever next model that's coming up that we're going to talk about please remember that Comparing to large, open, big companies backed by billions of dollars that run on multiple split hardware, it's just going to lead to disappointment.

[00:42:34] However, when you do some comparisons, like the guy did, that is now pinned to the tweet this is the way to actually do this. However, on specific tasks, like for, say, coding go ahead,

[00:42:46] Nisten Tahiraj: Nisam. I was going to say, we're still a bit early to judge for example, Falcon could have used a lot more training.

[00:42:53] There's also other. parts where larger models play a big effect stuff like if you want to do very long context summarization then you want to use the 70b and as far as i'm getting it and this is probably inaccurate right now but the more tokens you have the more meat you have in there the Then the larger the thoughts can can be.

[00:43:23] So that's the principle which are going by. Well, Mistral will do extremely well in small analytical tasks and in benchmarks, and it's amazing as a tool. It doesn't necessarily mean that it'll be good at thinking big. You still need The meat there, the amount of tokens to do that. Now you could chop it up and, and do it one, one at a time, but anyway, just something to keep in mind because lately we also saw the announcement from Lama70Blong, which started getting really good at at summarization.

[00:44:04] So again, there's one particular part. Which is summarization where you, it looks like you need longer you need bigger models for that. And I've tested it myself with Falcon and stuff, and it's pretty good at summarization, I just want to also give them the benefit of the doubt that there is still something that could be done there.

[00:44:28] I wouldn't just outright dismiss.

[00:44:31] Alex Volkov: Yeah, absolutely, absolutely, and I want to join this non dismissal. Falcon open sourced fully commercially like Falcon 70B before, and this was the biggest open source model at the time. And then they gave us 180B, they didn't have to, and we appreciate like open sourcing.

[00:44:46] We're not going to say no. Bigger models have more information, more, more, maybe world model in them, and there's definitely place for that, for sure. The, the next thing you mentioned also, and I think I, I strongly connect to that, Nissen, and thank you, is GPT 4, for example, is very generalized. It does many, many, many things well.

[00:45:08] It's like kind of impressive and whatever Gemini is going to be from Google soon, hopefully, we're always waiting on ThursdAI, that the breaking news will come on Thursday. We're gonna be talking about something else and then Google suddenly drops Gemini on us. There's also other rumors for Google's other stuff.

[00:45:22] Whatever OpenAI's Arrakis was, and then they stopped training, and whatever next they're coming from OpenAI, will probably blow everything we expect in terms of generality out of the water. And this, the open source models, as, as they currently are, they're really great at... Focused tasks, right? So like the coder model, for example that recently Glaive Coder was released by Anton Bakaj, I think is doing very well on the evaluations for code.

[00:45:51] However, on general stuff, it's probably less, less good. And I think for open models expecting generality on the same level as GPT 4, I think, is, is going to lead to disappointment. But for tasks, I see, I think we're coming close to different things that a year ago seemed state of the art. If you guys remember, it's not even a year since JetGPT was released, right?

[00:46:14] I think JetGPT was released in November? No, not as an API even, just the UI, like middle of November. So we're coming up on one year, I think the Dev Day will actually be one year. That was 3. 5. 3. 5 now, many people use 3. 5 for applications, but, you want to go for 4. If you're paying for Chattopadhyay Plus and you have a task to solve, you're not going to go 3.

[00:46:35] 5 just because you feel like it. You know that 4 is better. But now we're having open source models way smaller. They're actually getting to some levels of 3. 5, and the above effort is actually an effort to try to figure out which ones. And so I strongly recommend, first of all, to get familiar with local Llama subreddit.

[00:46:54] If you don't use Reddit, I feel you, and I've been a Reddit user for a long time, and I stopped. Some parts of Reddit are really annoying. This is actually a very good one, where I get a bunch of my information outside of Twitter. And I think Andrej Karpathy also recommended this recently, which... Then became an item on that subreddit.

[00:47:12] It was really funny. And this massive effort was done by this user and he, he did like a full comparison of just 39 different models. And he outlined the testing methodology as well. We've talked about testing and evaluation methodology. Between ourselves, it's not easy to evaluate these things. A lot of them are like gut feeling.

[00:47:31] A lot of the, the evaluation, and Nathan and I have like our own prompts that we try on every new model, right? It's, it's like a lot, a lot of this for many people is like gut feel. And many people also talk about the problem with evals and I think Bo mentioned the same thing with the embedding leaderboards that then, you know.

[00:47:48] It then becomes like a sport for people to like fine tune and, and release models just to put their name on the board to overfit on whatever whatever metrics and evaluations happen there. And then there's a whole discussion on Twitter whether or not this new model that beats that model on, on some, some score actually, was trained on the actual evaluation data.

[00:48:09] But... Definitely the gut feeling variation is important and definitely having different things to test for is important. And you guys know, I think, those of you who come to ThursdAI, my specific gut feels are about like translation and multilingual abilities, for example, and direction following some other people like Jeremy, Jeremy Howard from ThursdAI have his own like approach.

[00:48:29] Everybody has their own approach. I think what's interesting there is... Kind of the community provides, right? We're like this decentralized brain of evaluating every new model. And for now, the community definitely landed on Mistral as being like the top. At least a model in the 7b range, and Falcon, even though it's huge and can do some tasks like Nissan said is less, less and Lama was there before. So if you start measuring the community responses to open source models, you start noticing better what does what. And this effort from this guy, he actually outlined the methodology, and I want to shout out... Friends of the pod, Tignium being the go to many, many things, specifically because Open Hermes, which, Hermes, which we've talked about before which was fine tuned from Mistral7b is probably like getting the, the, the top leaderboard from there, but also based on my experiences, right?

[00:49:22] So we've talked last week about Open Hermes being able, you're able to run Open Hermes on your... Basically, M1, M2, Max with LM Studio, which also shout out to LM Studio, they're great, and I've tested this, and this seems to be, like, a very, very well rounded model, especially for one that you can run yourself and comparing to GPT 4 and other stuff, this model for specific things is really great.

[00:49:45] It's good for coding. It's not the best for coding. I think there's a coding equivalent. And I just really encourage you, if you're interested, like figuring out what to use. And we've talked about this before. What to use. Is an interesting concept, because if you come to these spaces every week and you're like, Oh, this model is now state of the art, that model is state of the art.

[00:50:05] You may end up not building anything, because you just won't have the, you always keep chasing the latest and greatest. The differences are not vast from week to week, we're just seeing like better scores. But it's well worth checking out this effort for the methodology, for the confirmation that you have.

[00:50:21] Let's say you, you felt that Mistral is better and now you can actually understand. And also for friends of the pod I think John Durbin is also, Error Boris model is really great and it's also up there. And what Nistan highlighted is that bigger models sometimes excel at different things some summarization or just more knowledge.

[00:50:38] It's also outlined there as well. And You can also see models that are not that great, that maybe look good on the leaderboards, but don't necessarily perform as well, and you can see them as well in that effort.

[00:50:49] So maybe actually let me reset the space. Everybody who joined in the middle of me speaking is like, why is this guy speaking?

[00:50:56] And what's going on here? You are welcome. You, you're in the space of ThursdayAI. ThursdayAI we are meeting every week to talk about everything that happens in the world of AI. If you're listening to this and you're enjoying, you're the target audience, but generally we talk about everything from open source LLMs and now embeddings.

[00:51:13] We, we talk about big company APIs. There's not a lot of updates from OpenAI this week. I think they're quiet and they're going to release everything in a week and a half in their dev day. And, and Tropic obviously, and, and... Cloud and Microsoft and Google, like all these things we cover as much as possible.

[00:51:29] We also cover voice and audio. And in that vein, I want to shout out to friends from Gladia and I'll pin there actually, let me just pin this right now. Gladia just released a streaming of Whisper and I've been waiting for something like this to happen. Sometimes for AI engineers, you don't want to host everything yourself. And you want to trust that, the, the WebSocket infrastructure is going to be there when you don't want to build it out. And I'm not getting paid for this.

[00:51:53] This is like my, my personal, if I had to implement like something like the voice interface with ChatGPT, I would not build it myself. I would not trust my own MLOps skills for that. And so for that Gladia is, I've been following them since I wanted to implement some of their stuff and they just implemented like a WebSocket.

[00:52:11] Whisper transcription streaming, and it's multilingual, and it's quite fast, and I definitely recommend folks to check it out. Or check out my review of it, and try out the demo, and if you want it, use it. Because we've talked last week about the interface for ChatGPT that's voice based, and you can actually have a FaceTime call with ChatGPT, and that's incredible.

[00:52:30] And I think more and more removing the screen out of this talking to your AI agents, I think, with the latest releases also in text to speech, like 11 Labs and XTTS that we've covered as well. With advances there, with speed, you can actually start getting interfaces where you can talk, and the AI listens and answers back to you very fast.

[00:52:52] Worth checking out, and definitely an update. Thank you.

[00:52:57] Nisten Tahiraj: Okay. So this is a complete product. I was,

[00:53:00] Alex Volkov: yeah, this is a full, pay a little bit, get a WebSocket and then you use this WebSocket to just like stream and you can embed this into your applications like very fast. Setting that up, I think Koki, you can do this with Koki, which we also covered.

[00:53:13] Gradio Interview with Abubakar, Xenova, Yuichiro

[00:53:13] Alex Volkov: Alright, I think it's time to again, reset the space. ThursdAI I wanna thank Bo who is still on stage. Bo, you're welcome to keep, stay with us a little bit and now we're moving on to the second part of this.

[00:53:30] Welcome, Abubakar. Welcome, Zinova, Joshua. Welcome some folks in the audience from Hugging Face. It's great to see you here on ThursdAI, well, Zinova is always here, or hopefully, but Abubakar, I think this is your first time.

[00:53:41] I'll do a brief intro, and then we can, we can go and talk about Gradio as well.

[00:53:45] I my first inference that I ran on a machine model was a year and something ago, and this was via Gradio, because I, I got this weights file, and I was like, okay, I can, I can probably run something with CLI, but how do I actually visualize this? And back then, Gradio was... was the way and I think since then you already guys you were already part of Hug and Face and Everybody who visited a model page and tried a demo or something probably experienced Gradua even without knowing that this is what is behind all the demos So welcome, please feel free to present yourself.

[00:54:17] Give us Maybe two line, three line of how you explain Gradio to folks, and then we can talk about some exciting stuff that you guys have released this week.

[00:54:25] Abubakar Abid: Awesome. Yeah, first of all, thank you again for, for having me and for having several folks from the Gradio team here. I've known you, Alex, for a long time.

[00:54:32] I think you were one of the early users of Gradio or at least one of the early users of Gradio blocks and, and some of these viral demos. So I've seen, this podcast develop over time and it's It's a real honor to be to be able to come here and to be able to talk about Gradio.

[00:54:45] Yeah. Hi everyone. I'm Abu Bakr. I'm, I lead the Gradio team at Hugging Face. So Gradio is basically the way we describe it is it's the fastest way to, to build a GUI or an app from a machine learning model. So traditionally have, taking a machine learning model to production or at least letting...

[00:55:01] Users try it out has meant that you need to know a lot of front end. You need to know how to, setting up a server, web hosting. You have to figure all of these things out so that other people can play around with your machine learning model. But Gradio lets you do all of that with just a few lines of Python as I think Joshua was mentioning earlier.

[00:55:18] And Gradio has been used by a lot of people. We're very lucky that, we kind of coincide. We started Gradio a few years ago late 2019. It grew out of A project at Stanford, and then spun out to be a startup, and then we got acquired by Hugging Face, and we've been growing Gradio within that kind of ecosystem.

[00:55:32] But we're very lucky because during this time has coincided with a lot of real developments in machine learning. I come from an academic background, so before 2019 I was doing my PhD at Stanford. And, everyone's been doing, machine learning for a while now, but...

[00:55:45] The types of machine learning models that people wanted to build, you built it, you published a paper and that was it, but, since then, recently people are building machine learning models that other people actually want to use, other people want to play around with, things have gone very, exciting, and so that's led to a lot of people building radio demos I think, I was looking at the the stats recently we have something around more than three, four million demos, Gradio demos that have been built since we started the, library.

[00:56:09] And yeah, so recently we released something called Gradio Lite, which lets you run...

[00:56:13] Gradio effects on the open source LLM ecosystem

[00:56:13] Alex Volkov: Wait, before, before, Abubakar, if you don't mind, before Gradio Lite, let's not I just want to highlight how important this is to the ecosystem, right? I'm oriJinally a front end engineer I do component libraries for breakfast, and basically, I don't want to do them it's really nice to have a component library maybe Tailwind UI, or ShadCN, like, all these things, so even front end engineers, they don't like building things from scratch.

[00:56:35] Switching to machine learning folks who like build the model, let's say, and want to run some inference, that's not their cup of tea at all. And, just thinking about like installing some JavaScript packages, like running NPM, like all these things, it's not like where they live at all. And so what Gradio allows us to do this in Python.

[00:56:51] And I think this is, let's start there. That's on its own is incredible and lead, led to so many demos just look to happen in Gradio. And you guys built out pretty much everything else for them, like everything that you would need. And I think recently you've added stuff, before we get to Gradual Light, like components like chat, because you notice that, many people talk to LLMs, they need the chat interface, right?

[00:57:10] There's a bunch of multi modal stuff for video and stuff. Could you talk about, the component approach of how you think about providing tools for people that don't have to be designers?

[00:57:20] Abubakar Abid: Yeah, absolutely. So yeah, that's exactly right. Most of the time when you're, machine learning, developer you don't want to be thinking about writing front end, components that coupled with some, an interesting insight that we had with machine learning models.

[00:57:31] It's much more like the components from machine learning models are tend to be much more usable than in other kinds of applications, right? So one thing I want to be clear is that Gradio is actually not meant to be like a, build web apps in general in Python. That's not our goal. Our goal, we're heavily optimized toward building machine learning apps.

[00:57:50] And what that means is, the types of inputs and outputs that people tend to work with are a little bit more, contained. So we have a library right now of about 30 30, Different types of like inputs and outputs. So what does that mean? So things like images, image editing video inputs and outputs, chatbots as outputs JSON, data frames, various types of inputs and outputs that components that come prepackaged with Gradio.

[00:58:15] And then when you build a Gradio application, you basically say, Hey, this is my function. These are my inputs, and these are my outputs. And then Gradio takes care of everything else, stringing everything together sending them, message back and forth, and pre processing, post processing everything in the right way.

[00:58:29] So yeah, you just have to define your function in the backend, and your inputs and your outputs, and then Gradio spins up a UI for you.

[00:58:36] Alex Volkov: And so I really find it funny and I sent the laughing emoji and said that Gradio was not meant to build web apps, like full scale web apps, because I think the first time that we've talked, you reached out, because I joined whatever open source that was running for stable diffusion, this was before automatic, I think, and you told me hey, Alex, You did some stuff that we didn't mean for you to do, so I injected like a bunch of JavaScript, I injected a bunch of CSS, like I had to I had to go with my full on like front end developer, I was limited with this thing, and so I, even despite the limitation, I think we did like a bunch of stuff with just like raw JavaScript injection, and since then I think it's very interesting, you're mentioning like Gradio demos, Gradio demos, Automatic 1.

[00:59:16] 1. 1, which is maybe for most people, is the only way they know like how to run stable diffusion, is now getting investments from like NVIDIA and getting right, I saw like a bunch of stuff that Automatic does, so it's very interesting like how you started and how the community picked it up. So can you talk about like the bigger parts of this, like Automatic and some other that are like taking Gradio and pushing it to the absolute limit?

[00:59:37] Abubakar Abid: Yeah, absolutely. So that's yeah we're, we're, I'm, I'm, like, perpetually shocked by Automatic 111, every time I see a plug in, or, kind of the, the I think, like you said NVIDIA, now IBM, or something, released a plug in for Automatic 111? It's crazy. But yeah, so basically it's ever since we started Gradio, we've been noticing that, okay, okay, Gradio seems to work for 90 percent of the use cases, but then the last 10 percent people are pushing the limits of, of what's possible with Gradio.

[01:00:06] And so we've progressively increased what's possible. So in the early days of Gradio, there was actually just one class called Interface. And what that did was it allowed you to Specify some inputs and some outputs and a single function. And we quickly realized, okay, people are trying to do a lot more.

[01:00:20] So then about a year and a half ago, we released grad your Blocks, which allow you to like have arbitrary layouts. You can have multiple functions, string together connect inputs and in different ways. And that is what kind of allowed these very, very complex apps like automatic 1 1 1 SSD Next, and the equivalence in other domains as well.

[01:00:36] Of course, like the, the text, the text web, the, the UBA Booga. Text UI as well and then there's also similar kind of, very complex demos in the audio space as well. And music generation as well. So like these super complex, multiple tabs, all of that, that's possible with this new kind of architecture that we laid out called GradioBlox.

[01:00:55] And GradioBlox is this whole system for specifying layouts and and, and functions. And it's defined in a way that's intuitive to. Python developers, the, we like a lot of these like web frameworks in Python have, have popped up. And one of the things that I've noticed as someone who knows Python, but really not much JavaScript is that they're very much coming in from the perspective of a JavaScript engineer, and so like this kind of React inspired kind of frameworks and, and stuff like that.

[01:01:21] And, and what, that's not very intuitive to a Python developer, in my opinion. And so we've defined this whole thing. Where you can, have these, build these arbitrary web, kind of web apps, but still in this Pythonic way. And we're actually about to take this a step farther, and maybe I can talk about this at some point but next week we're going to release Gradio 4.

[01:01:38] 0, which takes this idea of being able to control what's happening on the page. To the next level. You can have arbitrary control over the ui, ux of any of our components. You can build your own components and use them within a Grady app app and get all of the features that you want in a grad app.

[01:01:52] Like the, the, API usage, pre-processing, post-processing. Everything just works out of out of out of the box. But now with your own kind of level of control, yeah. Awesome.

[01:02:01] Alex Volkov: And it's been honestly great to see just how much enablement. Something like as simple as Gradio for folks who don't necessarily want to install npm and css packages.

[01:02:11] There's not much enablement this gave the open source community because People release, like you said, different significant things. Many of them, maybe you are not even aware of, right? They're running in some Discord, they're running in some Reddit. It's not like you guys follow everything that happens.

[01:02:23] Gradio local URL via Gradio Proxy

[01:02:23] Alex Volkov: Additional thing that I want to just mention that's very important that. When you run Gradio locally, you guys actually expose it via like your server, basically my local machine. And that's been like a blast that that's been like a very, very important feature that people may be sitting behind the proxy or everything.

[01:02:39] You can share your like local instance with some folks, unfortunately only for 72 hours. But actually

[01:02:44] Abubakar Abid: that's about to change. So in 4. 0, one of the things that we're trying to get, so actually, we've been very lucky because Gradio has been developed along with the community. Like you said, like often times we don't know what people are using Gradio for until, they come to us and tell us that this doesn't work, and then they'll link to their repo and it's this super complex Gradio app and we're like, what?

[01:03:01] Okay, why are you even trying that? That's way too complicated. But, but, then we'll realize like to the extent to what people are building. And so this you mentioned the share, these share links as well, which I want to just briefly touch upon. So one, one of the things that we released in like the early days of, of, of Gradio is we realize People don't want to worry about hosting their machine learning apps.

[01:03:19] Oftentimes you want to share your machine learning app with your colleague. Let's say you're like the engineer and you have a colleague who's a PM or something who wants to try it out. Or it might be if you're in academia, you want to share it with fellow researchers or your professors, whatever it may be.

[01:03:33] And like, why do all of this hosting stuff if you just are, are, like, building an MVP, right? So we built this idea of a share link. So you just, when you launch your Gradio app, you just say share equals true. And what that does is it creates a it uses something called Fast Reverse Proxy to actually expose your local port to a to this FRP server which is running in a public...

[01:03:53] Machine, and what that does is it forwards any request from a public URL to your local, port. And, what the, in a, the long story short, what that does is it makes your Gradio app available on the web for anyone to try. It runs for 72 hours by default, but now what we're doing as part of 4.

[01:04:08] 0, we'll, announce this, is you can actually build your own share servers. So we have instructions for how to do that very easily and you can point your Gradio instance to that share server. So if you have an EC2 instance running somewhere, just point to it and then you can have that share link running for as long as you want and you can, share your share servers with other people at your company or your organization or whatever it may be and they can use that share link and, again, they can run for however they want.

[01:04:30] Wait,

[01:04:31] Nisten Tahiraj: wait, wait, is this out? Which branch is this? Is

[01:04:34] Abubakar Abid: this going to be out? This is going to be out on Tuesday for Gradio 4. 0 we're, we're going to launch on Tuesday.

[01:04:41] Nisten Tahiraj: It's like the most useful feature I'd say of, of Gradio, especially when you make a Google collab that you want people to just run in one click.

[01:04:49] And like, how are they going to even use this model? And you just throw the entire Gradio interface in there and you share equals true. And then they know, they can just give it, give the link to their friends and stuff. It's really, it makes it really easy, especially with Google Colab. But now that you can host your own, this is huge.

[01:05:09] This is going to... to another level. I have more questions for

[01:05:14] Alex Volkov: Google. I think I just Nissen, thank you. I just want to touch upon the Google collab thing. I think at some point Google started restricting how long you can run like a collab for, and I think you guys are the reason. This exact thing that Nissen said.

[01:05:30] People just kept running the Gradio thing with the URL within the Google collab and exposing like stable diffusion. They didn't build collab for that, and I think they quickly had to figure out how to go around this.

[01:05:41] Abubakar Abid: Yeah. And their approach is like literally blacklisting the name of the the of specific, GitHub repos, which, I, I completely understand where, where Colab is coming from, right?

[01:05:50] They're giving these GPUs for free. They have to have to prioritize certain use cases, but we're working with the Colab team and we're seeing if, there's ways, like right now it's like a blacklist on, on automatic one on one, and some other repos. So we're hoping we can find another way that's not That's not so restrictive.

[01:06:05] Nisten Tahiraj: No, but it still works. You can just fork the repo. It works for everything else. It works for LLMs. So if anybody else really needs it. Gradio works on Colab. Well, as far as language stuff goes, I haven't done that much.

[01:06:18] Abubakar Abid: Yeah, so Gradio works on Colab for sure. And, and that's, and that's early on, like one of the decisions we had to make actually was...

[01:06:25] Should we use like, the default python runtime or should we like change, like the interpreter and stuff like that? Because building GUIs is not necessarily python's like strength, and like oftentimes you wanna render re-render everything, and you, you wanna do certain things that may not be like what Python is suited for.

[01:06:42] But early on we decided, yeah, we wanna stick with the default python runtime because. One of the reasons was things like Colab, because we wanted people to be able to run Gradio wherever they normally run Python without having to change their workflows. And Colab, Gradio works in Colab.

[01:06:56] We had to do a lot of... Trickery to make it work. But yeah, it works. It's just like these certain very, very specific apps that have become too popular and apparently consume too many resources. They're blacklisted by Colab right now.

[01:07:10] Local inference on device with Gradio - Lite

[01:07:10] Alex Volkov: Alright thank you for this intro for Gradio. To continue this, we have on stage Zinova who introduced himself, authors of TransformerJS, we've been talking with Bo in the audience, also somebody who's like just recently open sourced, with Jina, the embeddings model, and everything that we love to cover in ThursdAI, a lot of it is talking about As open source, as local as possible, for different reasons, for, not getting restricted reasons.

[01:07:36] And you guys just recently launched Gradio Lite, and actually we have Yuichiro here on stage as well. So I would love to have you, Abubakar introduce and maybe have Yuichiro then follow up with some of the stuff about what is Gradio Lite? How does it relate to running models on, on device and open source?

[01:07:52] And yeah, please, please introduce it.

[01:07:54] Abubakar Abid: Yeah, absolutely. Yeah. Like you mentioned, I think one of the things that we think about a lot about at Gradia is like, it's the open source ecosystem and, right now, for example, where can open source LMs, for example, really shine and things like that.

[01:08:06] And one of the places is on device, right? On device or in browser is, open source has a huge, edge over proprietary models. And so we were thinking about how can Gradio be useful in this setting. And we were thinking about the in browser application in particular. And we were very, very lucky to have Yuchi actually reach out to us.

[01:08:25] And Yuchi has this, fantastic tracker, but if you don't already don't know Yuchi, he built Streamlit Lite, which is a way to run Streamlit apps in the browser. And then he reached out to us and basically had this idea of doing something similar with Gradio as well. And basically, I, almost like single handedly refactored much of the Gradio library so that it could run.

[01:08:43] In, with Pyodide, in WebAssembly, and basically just run in the browser. I'll let Yuchi talk more about that, but basically, if you know how to use Gradio, then you know how to use Gradiolite. You just write the same Python code, but wrap it inside Gradiolite tags, and then it runs within the browser, like in the front end.

[01:08:59] You can execute arbitrary Python, and it just works. Yuchi, if you want to share a little bit more. About that, yeah, or introduce yourself.

[01:09:08] Yuichiro: All right, Hey, can you hear me? Well, thank you very much, thank you very much for the quick short interaction about Gradual Light and Streaming Light 2.

[01:09:15] Well, as Abhakal explained about it,

[01:09:18] there was

[01:09:18] Sorry. OriJinally there was kind of a, a tech technological movement about edge computing or Python. It was started by uh ide that was c Python runtime compiled for web assembly that can completely run on web browsers. It started, it triggers the big band of edge computational Python, random starting with project that was.

[01:09:43] That was already deported to Web Assembly runtime as D Light, and it inspired many other Python frameworks, including Streamlet and any other existing Python frameworks. I dunno uh pet script or. HoloVis panel or Shiny for Python, something like that. So there was a huge movement about to, to make Python frameworks to compatible with WebAssembly or web browser environment.

[01:10:13] And I thought that was a great, opportunity to make machine learning or data science stuff completely run on web browser, including transformer things or many more stuff existing in the stream machine learning ecosystem. And I first created a Streamlit Lite that was forked version of Streamlit to WebAssembly.

[01:10:36] And yeah, the, remaining story were the same as what Abhakaal introduced. So technically it was not, my oriJinal stuff, but there was a huge movement about such kind of stuff. And I, simply followed that. flow and my the transfer such kind of analogies to gradial repository.

[01:10:58] Alex Volkov: Yeah, that's it. Awesome. Thank you so much. And okay. So can we talk about what actually do we do with now the ability to run Gradio all in the browser? Could maybe both of you give some examples and then I would like also to to add Zenova to the conversation because much of the stuff is using Transformers.

[01:11:18] js, correct? Can we maybe go and talk about what is now actually possible compared to like when I run Gradio on my machine with a GPU and I can run like Stable Diffusion? I just

[01:11:27] Nisten Tahiraj: want to say that this is crazy that this can happen at all for the audience to

[01:11:32] Abubakar Abid: prepare. Yeah, I was honestly blown away the first time Yuchi showed me a demo as well.

[01:11:36] Imagine you have a, any sort of machine learning model. Practically, not almost anything, but a super, really good speech recognition model running completely in your browser. Meaning that, for example, now you can take that demo, you can put it inside GitHub Pages.

[01:11:51] You can host it inside. We've seen people embed Gradio demos now with Gradio Lite inside Notion. So you have a Notion whatever page, you can take that demo, you can embed it inside Notion. One of the things that we launched when we launched Gradio Lite at the same time is we also launched something called the Gradio Playground.

[01:12:07] Now the Gradio Playground, you can actually just Google this, you can find this. But basically what it allows you to do is it allows you to write code in the browser. And as you're editing the code, you can see live previews of your Gradio application. And, and basically what's happening is, is taking that Gradio code, it's wrapping it inside Gradio Lite tags, and it's just running it.

[01:12:27] It's just straightforward application of Gradio Lite. And, we're excited by this personally just because if one, it opens up, it allows us to write interactive documentation. You can write, you can try stuff, you can, you can immediately, see the results. We're also excited because we've seen interest from other libraries including, for example, SacketLearn, who want to embed Gradio demos within their documentation.

[01:12:49] Within their docs, right? But they're they were hesitant before because they didn't want to have a separate server running these radio applications and have to worry about maintaining those servers, making sure they were up all the time, making sure they could handle the load. Now they can write it in their docs and they're like, their, their demos and everything, they'll just run in the user's browser.

[01:13:07] They won't have to worry about maintaining everything since it's, in the same code base and everything. So I think that's another kind of cool application that we're excited by is just... These potential for interactive documentations that, maybe potentially other, other maintainers or other libraries might want to include.

[01:13:22] So yeah, so stuff like, security, privacy, serverless type stuff, hosting, and all of that. And then also like these interactive documentations.

[01:13:30] Alex Volkov: I think the demo that you mentioned with the transition within Notion from VB from Hive Interface, I think that was great. I'm trying to find the actual link, but basically, because Notion allows to embed like basically iframes, right?

[01:13:42] So he embedded this whole Gradio Lite interface to translate. I think using... Burt or something like very similar that all runs within like the notion page. I think that's awesome. Joshua, you want to chime in here and say how transformers is built into this and now this allows for way more people to use transformers in like a UI way.

[01:14:02] Transformers.js integration with Gradio-lite

[01:14:02] Xenova: Yeah, sure. So first of all, literally nine, like almost, I would say the whole. Everything that we are talking about now has been, like,

[01:14:12] Abubakar Abid: led by the Gradio team.

[01:14:14] Xenova: And I am here piggybacking and be like, Whoa, look at this, Transformers JS is now working. That's really not what we're talking about today.

[01:14:23] It's the amazing work that the team has been able to do. To achieve the past, this has been going on for, for quite a while. It's been like codenamed like Gradioasm and now finally being released as as GradioLite and now. Sort of like the Transformers J side of it. Just oh, by the way there's this library called Transformers J you can sort of use it and, and with the, the Transformers.

[01:14:48] Oh, was that ? Sorry. You've been way too humble.

[01:14:51] Abubakar Abid: No, no, absolutely

[01:14:52] Xenova: not. I think it's, it's, it's so much has been done by, by you and your, and, and the amazing radio team that's it, it just so happens to be that these things are like coinciding. And now you can end up using Transformers. js with, with Gradio and Gradio Lite.

[01:15:07] And obviously this is also made possible by, okay, everyone, everyone stick with me. It's going to be a little, get a little complicated when I try to explain this. But Transformers. js Pi, which is, are you ready? A JavaScript port of a Python library turned into a JavaScript library so that I can run in a Python environment.

[01:15:29] Okay. We all caught up? That's, that's Transformers. js. py, which is which, which Yushi wrote in the audience obviously with his experience with streamlets bringing streamlets to the browser. It's sort of his, his invention, which is quite funny, but that's sort of how Transformers. js is able to be run.

[01:15:49] Inside Gradiolites there are other ways, but from what you'll see in the documentation, that's sort of like the, the go to way. And it's

[01:15:57] Alex Volkov: Yeah, I wanna ask about this, because I saw from Transformers. js, import import on Discord Transformers. js.

[01:16:04] So maybe you should, could you talk about this part that, that Zenova tried to explain? It was, like, a little A little complex Transformer. js is you can install it through npm and then run this, right? And then it runs in the, in the node environment and browser environment. Gradualite is basically Python within JavaScript.

[01:16:19] So then you have to turn Transformers into Python in order to get it into Gradualite so that it runs within the JavaScript context again? Is that, is that correct? Am I getting this right?

[01:16:30] Nisten Tahiraj: If I could say something for the audience, what has, what's happening here is that there's a layer called Pyodide and that uses kind of like what WebAssembly uses to run Python at native speeds.

[01:16:44] So it runs in the browser. And it goes down that stack, there's like a virtual machine and compiler, all that stuff in there. And then that's how Python is able to run at native speed. And this means that with PyScript, you can have inside the same index. html, just your regular index. html, you can have your JavaScript code and your objects and stuff.

[01:17:06] And you can have just straight Python code in there. Like you just add the tag. You just dump the Python as is nothing else. And the crazy part is that it can access JavaScript objects now. So you can do the math in Python in the browser, 'cause JavaScript can do math well again, but, and then you can access those objects.

[01:17:30] So this is a whole crazy stack here with PIO IDE and EM scripting. And again, that's only WebAssembly. So that's CPU. Only for now, because there's still a mountain of work to get, and to finish it off, Emscripten is like your POSIX layer, like your Unix layer. It's like there's an operating system being built inside the browser here that's going on.

[01:17:54] So that's why things are getting complicated, but yeah, just to keep that in mind, that's the base.

[01:17:59] Yuichiro: Yeah, but, what Nisten talked about was everything, because we can access the JS object from Python world inside the browser if you import transformer.

[01:18:10] js. py on Gradle Lite, under the hood, transformer. js is now still being imported in the browser environment. And what... When you write a Python code as a Gradle like application on the browser, what you do is, simply using the oriJinal JavaScript, JavaScript version of Transformer. js, just proxy from the Python code through the, proxying mechanism provided by Pyodite.

[01:18:42] What Transformer. js. py does is just a, thin, Proxying layer or some glue code between bridging these two, two words, Python and JavaScript. That's it.

[01:18:56] Abubakar Abid: Yeah, I just zooming out a little bit. So basically what, what Transformers. js underscore pi does, it lets you run everything that Transformers.

[01:19:03] js does. And what Transformers. js does, it lets you run a lot of the models, a lot of the tasks. There's a lot of the models there, you can now run in your browser, right? We're talking about all of the NLP related tasks, like things like translation, LLMs, but also, a lot of the vision tasks, a lot of the audio stuff.

[01:19:22] We're talking about speech recognition that's powered by Transformers, what Josh has been doing with Transformers. js. And I think Transformers. js just released, even for example, speech generation. Text to speech. And so now you can do that within Transformer. js, which means you can do it within Transformer.

[01:19:34] js Pi, which means now you can do it within Gradial Light as well.

[01:19:40] Alex Volkov: That's incredible. And I think the biggest part for me is that... Now that you guys ported Gradio, which is ubiquitous in machine learning, and everybody who releases the model uses either this or Streamlit, but I think it's, it's a clear winner between the two, as this is, as I'm concerned and as I see, then now you basically ported the same thing towards the browser, and the more we see, Models getting smaller and we've been always talking about this models getting smaller, models being uploaded to the browser.

[01:20:08] Browsers

[01:20:09] Abubakar Abid: getting more powerful and,

[01:20:10] Alex Volkov: and WebP is more browser getting more powerful. Yeah. And yeah, I'm getting, I'm getting to web GPU because we have backend here, Arthur, on stage. And I would love to introduce you guys if, unless you're already familiar. The more we see this move, the more like the need for something like a component library that's built in is, is very interesting.

[01:20:25] Even though this world already has a bunch of libraries. But you're basically, with this, you're also porting the people with the experience of Gradio, right? With the existing with the existing frameworks, with the existing Gradio interfaces, to this world. I find it very exciting, so thank you.

[01:20:38] And I want to introduce Arthur. Arthur, feel free to unmute yourself and maybe introduce yourself, briefly, and then yeah, feel free to chime in to this conversation.

[01:20:46] Arthur Islamov: Okay, so I did quite a lot of things with ONNX to create the Diffuser. js library and to load stable diffusion in the browser, and now I'm working on the SDXL version.

[01:20:58] So I was going to ask, do you know if there are some plans on adding WebGPU backends for PyTorch? Because when it happens... It'll be so much easier as web GPU backend can be launched on any platform, not even in the browser, but also locally without the browser, just using the metal backend the direct ticks or Vulcan or Linux.

[01:21:28] So I guess when that happens, we'll go to a whole new era as you'll be able to run those PyTorch models in the browser with GPU acceleration.

[01:21:40] Xenova: I can tag on to this. The TLDR of it is it's not at the point...

[01:21:46] Where, sort of, I'm sort of comfortable with upgrading the ONNX Runtime web Runtime, basically, to support the WebGPU backend right now, just because there's quite a few issues still left, like left, so we'll see. To solve before we get to the point where you can start running these models completely on WebGPU.

[01:22:07] The main, I think, the current issue at the moment is with, like, when you're generating text a lot of the... The buffers aren't reused properly during when you, when you start decoding. That's sort of leading to quite a massive performance bottleneck just because you're transferring memory between CPU and GPU every single time you're you're decoding.

[01:22:31] So that's, that's not quite there yet, however, with things like image image classification and I guess models with encode only, encoder only models, those are getting quite good, like births pretty fast we've, segment anything when you're just doing the encoding step, we the Onyx Runtime team has got to the point where it used to take around 40 seconds and now it takes around 4 seconds.

[01:22:55] And that's currently being worked on in like a dev branch, basically, of Transformers. js, just like making sure the integration's working. But it's, it's almost there. I keep, I keep saying it's almost there, but the amazing Microsoft team has been, has been really working hard on this. And if you just look at the commit history of on GitHub Microsoft slash Onyx Runtime and you go to the web version.

[01:23:18] There's just so many amazing people working on it and it's slowly getting to a point where and this will sort of be released with Transformers. js version 3. When we upgrade the Onyx runtime version to probably 1. 17, which will be, which will be the next one. It's currently 1. 16. 1. And then they'll, it's, and, and literally from the user's perspective, it's as simple as adding a line of code just saying, Basically use web GPU instead of web assembly.

[01:23:46] And, and also in the case where it's not supported, it'll fall back to the web assembly implementation. And, and that's, this will completely be transferable to how grid your light works, just because as was mentioned, it's sort of use as transformers js under the hood. So you any benefits that you'll see in Transformers j you'll see in transformers Jss pie, which you'll see in radio lights, which is, which is great.

[01:24:11] TLDR coming soon, it's an, it's an annoying answer to give, but it's, it's so close. And I guess this is also good because it sort of aligns with the time that more browsers will support WebGPU, sort of like without flags. I know Chrome is sort of leading the charge and other Chromium based browsers.

[01:24:30] But if you look at things like Safari and Firefox, it's quite far behind to the point that you it's. It's not, it's not ready for like mass adoption yet, but once it is, and once the ONNX Runtime backend has the WebGPU support has improved, you'll definitely be seeing that in Transformers Jest.

[01:24:48] So hopefully that answers the question. I

[01:24:52] Nisten Tahiraj: think stuff's about to get crazy on the front end because of this because the thing about you have all your WebGL stuff, you have all your maps, all your 3D, all your games. Now you can have an LLM even generate code for them, manipulate those objects, move stuff around on screen in 3D, and like the, the AI.

[01:25:14] Does that at all, all within your machine, but I do want to say that for, for Pyodide itself, it might take a long time for a WebGPU support because it depends on EM scripting. And if you want to do anything with Python, like open a file, write a file, output a file, you only can do what EM scripting gives you and EM scripting is like the base layer.

[01:25:39] Of the operating system, like it pretends, it fools your apps into thinking that there's an operating system there when, when there isn't. And as far as I've seen, like two, three months ago, WebGPU support was like really, really early on and might take a while for Emscripten to support that. So you're going to have to do that other ways by going straight to using WebGPU versus using it with that layer.

[01:26:06] So it might get a bit complex

[01:26:09] Alex Volkov: there. I agree about the stuff is about to get crazy. Go ahead, Arthur, and then we'll follow up on Gradio 4 and then we'll conclude.

[01:26:18] Nisten Tahiraj: Yeah, I

[01:26:18] Arthur Islamov: just wanted to tell that as yesterday or a few days ago I have seen that This distilled stable diffusion model I saw that they have previously released not Excel version, but the ordinary 2.1 or something like that, the distilled one.

[01:26:35] So I'm thinking to try to make my edema work with that distilled model without 64 beats. So for just ordinary 32 bit that will work in almost any browser without any additional flags or launching with some special parameters.

[01:26:54] Alex Volkov: Yeah. Arthur, you can't just mention an item on my updates list and not talk about this, right?

[01:26:59] Folks, let me just briefly cover what Arthur just said. Just on the fly SegMind, a company called SegMind, introduced like a distilled version of SDXL. And it's okay, Diffusion Excel, something they released a while ago. We've covered this multiple times. Understand like way better. Quality, obviously, generations and diffusion, but also way better text understanding, right?

[01:27:20] And it has two parts there's like a refiner part in addition. And so this company basically distilled that distillation we've talked about multiple times before. It's when you train your own model, but then you steal data from GPT 4 and you create the data so that GPT 4, you basically distill, it's like smartness to your own models.

[01:27:37] So they basically did this for SDXL they call it SegMind Stable Diffusion 1B, and it's a. 50 percent smaller and 60 percent faster than SDXL. Again, just to put in some time frames what Abubakar and I talked about, where I first experienced Gladio, this was Stable Diffusion 1. 4, a year ago, a year and a couple of months ago.

[01:27:57] Since then, we got Stable Diffusion multiple iterations of Stable Diffusion, then there is SDXL, which is like the Excel version. It it generates 124 by 124 images. And, and then now a few months after they released that, now we have a version that's 50 percent smaller and 60 percent faster.

[01:28:16] And so what Arthur is like now talking about diffusers JS is the ability to like load stable diffusion in the browser. Now there's a model that's half the size, and 60 percent has passed, which is good for the browser context. So I pinned it to the top of the tweet check out SegMind, it's definitely super cool.

[01:28:34] And the advancements that we see from week to week, and this is obviously super cool as well. And Arthur, sorry to interrupt with this, but you had one of my tasks that I had to finish before we finish and talk about this. So are you, have you already introduced it to diffusers? Have you tried it?

[01:28:52] Nisten Tahiraj: I have

[01:28:52] Arthur Islamov: tried it to convert it to omx, but it didn't work or maybe some of my code didn't work. So I guess I will try again on the weekend and yeah, most likely I will make it running.

[01:29:06] Alex Volkov: I think we had some folks on segment react and we will, let's try to connect there and, and hopefully get it running on, on as well so that we all benefit and I guess, maybe as the last part of this conversation Abukar and thank you for joining Uchi ABA Ali and the folks from Hugging Face. It is great to see all of you. I think you mentioned some folks that joined before like Uhb and some other folks on the Hugging Face. We're big fans here on ThursdAI, and we're like, always welcome you guys.

[01:29:33] Could you talk about what's coming in version four, because I think you, you, you gave us like one tidbit. But give us an update on that. I would love to hear.

[01:29:40] Abubakar Abid: Yeah, yeah, definitely. Yeah, so we're launching Gradio 4. 0 on Tuesday October 31st. And basically, the, the team has been working very, you mentioned earlier that, people are, are building these very, very complex apps with, with Gradio and, and really, honestly, stuff that we did not anticipate when we were designing Gradio.

[01:29:57] And more and more, what we want to do is almost take ourselves out of this feed, feedback loop. And let people build what they want to build, but let the community build stuff that, whatever you imagine, kind of just be, just be able to put that in a Gradio app. Let me be a little bit more concrete.

[01:30:11] So what is Gradio 4. 0 going to introduce? For example it's going to introduce the idea of custom components. So if you know a little bit of Python, a little bit of JavaScript, you can build your own component. You can use that within a Gradio app, just like you do normally, just like you use our built in, 30 or so built in components.

[01:30:27] Speaking of the built in components, we're redesigning some of the components from scratch, particularly the media components. So things like image audio video, they're going to be much, much nicer and they're going to be fully accessible. So one of the things that we're realizing is that, we're not, at Gradio, we're not just building a product for a specific audience, but we're building tools that let people build, apps for many different audiences.

[01:30:50] And so we want to make sure that all of the core components are accessible. That way it's easy to do the right thing and build accessible web applications. So we're redesigning that we're switching over from WebSockets to as server side events. There's several reasons for this and, and we'll talk about more about this on, on Tuesday.

[01:31:07] We're, we're having a little long live stream as well, but there's several reasons why server side events is the way to go for Gradio. And so there's, that's more of an internal refactor. You probably won't notice things. You might notice some speed ups in certain situations. It'll unlock a lot of things later on.

[01:31:22] We're open sourcing the the the sharing links process, the share servers at Gradio. So everyone will be able to set up their own, custom share links. So instead of, whatever dot Gradio dot live, you can have, you can have, some, some code, dot turjom dot video if you want, you can have whatever URL, custom URL you want for your share links.

[01:31:42] And then a lot of other changes as well we'll, we'll, we'll talk more about that on Tuesday. The team has been working super hard, so I'm, I'm excited to, to get it out for you guys to try out.

[01:31:51] Alex Volkov: That's so awesome, and, and can't, can't wait to see this. I I think the share links is like such a powerful virality thing, that once people start adding this to their domains, and, and start running different Gradio interfaces within Colab, outside of Colab, with their own domains.

[01:32:08] I think it's going to be super cool, especially if they don't expire. I absolutely received many of these links over DMs from multiple people. I think even people in the audience so far. And I think adding them to the custom domains. Thank you for open sourcing that. That's great.

[01:32:21] Abubakar Abid: I think part of it is also we want to reduce the load on our shared servers.

[01:32:25] We're getting too many of these links being created

[01:32:27] and stuff.

[01:32:27] Alex Volkov: Yes, absolutely. And I think the accessibility features are great. Folks, definitely check out follow Abu Bakr, follow Yuchi, and folks on stage, do Ali as well. To stay tuned to what's coming up to Gradio and then make sure to update your Gradio interfaces to the new accessible ones because you're building it's no longer demos.

[01:32:46] Everybody's using every new model is, is getting a Gradio interface and accessibility is very important for the level of web applications. I think... With that, I want to thank you guys for coming up and sharing with us Radio Light, which is very, very in accordance to what we love to talk about here, open source, open source LLM, on device inference, and taking control of your own, LLMs.

[01:33:07] I think, Nistan, you briefly, briefly talked about how crazy it's going to be. Where there is an LLM built into your website or web application that runs on the GPU of your device and is able to do stuff. And you can interact with it without basically offline. That's great. I think Nishtan, something that I want to talk about, but maybe I'll let you talk about this.

[01:33:27] I will say this thing. Now that we've concluded the interview with Gladio folks, one of the things that we love. Most of all, on Thursday, I have breaking news, and we actually have some breaking news. Nissen, go ahead, please, present the breaking news that you just sent.

[01:33:39] Nisten Tahiraj: I pasted a Gradius space above.

[01:33:43] If you click on it, that's what it is. And it's it's Kokui's, new release, new voice model. This is huge because they're allowing fine tuning on their voice model. And one criticism of the open source voice models has been that the dataset for training them has been of poor quality, like the, the microphone and, and stuff and the, the dataset that people use to, to train the models has been bad.

[01:34:12] So this is. Pretty important in this regard, because it's one of the very few, there's the one that Zenova released, and the Kokui one, that are open source and usable when it comes to text to speech, and that are like somewhat, somewhat pleasant, and that run relatively fast. Otherwise, it's pretty hard to have text to speech.

[01:34:37] Yeah, the, the part that you can fine tune, they, they open source the fine tuning code. Yeah, go there and

[01:34:43] Alex Volkov: get that, yeah. Thank you, Nissan. The, the folks from Cochlea, when they released XTTS, which is the open source text to speech that kind of, we know 11 Labs, we know Play. ht, we know OpenAI has one that Spotify uses to translation, and OpenAI haven't released any.

[01:34:59] We'll see next week if they're gonna give us an API for that. All of those require... A lot of money, just a lot of money 11 labs is basically rolling in cash because everybody wants to get their AIs to talk, right? And so we previously here, we talked about the listen part, we've talked about the streaming from Gladiator, but now you can have Whisper basically streaming.

[01:35:18] The other part of that was, hey, well, once your LLM listens and thinks, which is the inference part, you also want it. to talk to you. And TTS Texas speech is the way to do that. And Kokui, we had a chat with Joshua when they released TTS, which was very exciting. And now live on stage, live on Thursday.

[01:35:34] I, because this is why Thursday exists. Many people release stuff on Thursday. There is their own fine tuning with minutes of data. So you can create a voice, let's say maybe this is going to be disappointing for folks here on stage, but everybody here who spoke on stage more than a minute now is basically public for everybody else to take your voice and clone it with XTDS.

[01:35:56] It was possible before, somebody just had to pay money for it, but now... And Ali's laughing because Ali didn't talk yet, but everybody's basically now is, is going to get voice cloned. It's very easy. We're going towards this future. If this future scares you, there's no escape from that. Even I think VAL E from Microsoft, when it was released, they talked about like maybe 30 seconds of voice is enough to clone.

[01:36:18] But XTTS now gives us basically a framework and, and even the new language they said to, to add to Kokui, to the XTTS, and then you can use this. Within Transformers. Zinova, can we use Cochlear within Transformers. js or not yet? I think, I think we can. Not yet. Not yet. Okay, so soon, soon you'll be able to even do this all completely all within the browser.

[01:36:41] Hopefully once integration with WebGPU lands. So here we have it folks. We had an incredible Thursday I today. We started with talking with Bo and the guys from the embeddings team that released like the, the most kind of up how should I say, most comparable to OpenAI embeddings in open source.

[01:36:59] And that was great. Bo actually gave us like a masterclass in how embeddings work and Jina embedding models are available within, talked with Bakar and Yuchi and ANOVA and Arthur and all on stage the team behind Grado that if you haven't used Grado, you probably have used Grado. You just didn't know that it's Grado And, actually, this interface or slash library that started for demos only scaled all the way up to something like automatic, where like multiple people compute, contribute thousands of contributions including like NVIDIA and and I think IBM contribute now. It's like a full businesses run on this quote unquote component library.

[01:37:37] And I I just want to invite you to join Thursday and next week as well because... Some of this was planned, but definitely not all of this. But also, this is the way to stay up to date. And next week we're going to see some more incredible things. I think some very interesting things are coming up.

[01:37:52] I will have a personal announcement to make that's going to be very surprising to some folks here on stage. But definitely we'll keep ThursdAI going significantly, significantly more. And with that, I just want to thank you for joining us. It's been a pleasure to have these. It's been a pleasure to like have a space where, the graduate folks and the Jina folks can come and talk about what they released.

[01:38:12] And we can actually ask them questions. I want to thank everybody who joined on stage. Nistan, thank you always for joining, Zinova, Arthur. We, we were joined by new folks that we'll introduce next. Thank you so much for joining us this week because we just don't have the time and obviously thank you for folks in the audience who join every week and I see Enrico in there and Junaid and Tony and some other folks that like I love to see from week to week.

[01:38:33] If you missed any part of this, any part at all, or if the internet connection for you got stuck ThursdAI is about a live recording, but then it's getting released as a podcast episode, so you will get, if you're subscribed, and you should be already subscribed to ThursdAI on Apple or Spotify, you'll get this episode hopefully very quickly edited, if I'm not getting lost in some other interesting stuff, like Kokui.

[01:38:57] Thank you. And we will also release a newsletter with all the links and the conversations with Guardia team and, and Bo and all the updates as well in the form of links. And with that, I thank you for joining. It's been two hours. It's been a lovely time and now I need to go and actually edit the podcast.

[01:39:12] See you here next week. Thank you and yeah, please share with your friends as much as possible. The more crowd there is, the better these will be. And yeah, help and participate. Thank you all and have a good rest of your week. Bye bye.

ThursdAI November 2nd

Hey everyone, welcome to yet another exciting ThursdAI. This week we have a special announcement, the co-host of and I will be hosting a shared X space live from Open AI Dev Day! Monday next week (and then will likely follow up with interviews, analysis and potentially a shared episode!)

Make sure you set a reminder on X (https://thursdai.news/next) , we’re going to open the live stream early, 8:30am on Monday, and we’ll live stream all throughout the keynote! It’ll be super fun!

Back to our regular schedule, we covered a LOT of stuff today, and again, were lucky enough to have BREAKING NEWS and the authors of said breaking news (VB from HuggingFace and Emozilla from Yarn-Mistral-128K) to join us and talk a little bit in depth about their updates!

[00:00:34] Recap of Previous Week's Topics

[00:00:50] Discussion on AI Embeddings

[00:01:49] Gradio Interface and its Applications

[00:02:56] Gradio UI Hosting and its Advantages

[00:04:50] Introduction of Baklava Model

[00:05:11] Zenova's Input on Distilled Whisper

[00:10:32] AI Regulation Week Discussion

[00:24:14] ChatGPT new All Tools mode (aka MMIO)

[00:35:45] Discussion on Multimodal Input and Output Models

[00:36:55] BREAKING NEWS: Mistral YaRN 7B - 128K context window

[00:37:02] Announcement of Mistral Yarn Release

[00:46:47] Exploring the Limitations of Current AI Models

[00:47:25] The Potential of Vicuna 16k and Memory Usage

[00:49:43] The Impact of Apple's New Silicon on AI Models

[00:51:23] Introduction to New Models from Nius Research

[00:51:39] The Future of Long Context Inference

[00:53:42] Exploring the Capabilities of Obsidian

[00:54:29] The Future of Multimodality in AI

[00:58:48] The Exciting Developments in CodeFusion

[01:06:49] The Release of the Red Pajama V2 Dataset

[01:12:07] The Introduction of Luma's Genie

[01:16:37] Discussion on 3D Models and Stable Diffusion

[01:17:08] Excitement about AI Art and Diffusion Models

[01:17:48] Regulation of AI and OpenAI Developments

[01:18:24] Guest Introduction: VB from Hug& Face

[01:18:53] VB's Presentation on Distilled Whisper

[01:21:54] Discussion on Distillation Concept

[01:27:35] Insanely Fast Whisper Framework

[01:32:32] Conclusion and Recap

Show notes and links:

* AI Regulation

* Biden Executive Order on AI was signed (Full EO, Deep dive)

* UK AI regulation forum (King AI speech, no really, Arthur from Mistral)

* Mozilla - Joint statement on AI and openness (Sign the letter)

* Open Source LLMs

* Together AI releases RedPajama 2, 25x larger dataset (30T tokens) (Blog, X, HF)

* Alignment Lab - OpenChat-3.5 a chatGPT beating open source model (HF)

* Emozilla + Nous Research - Yarn-Mistral-7b-128k (and 64K) longest context window (Announcement, HF)

* LDJ + Nous Research release Capybara 3B & 7B (Announcement, HF)

* LDJ - Obsidian 3B - the smallest open source multi modal model (HF, Quantized)

* Big CO LLMs + APIs

* ChatGPT "all tools" MMIO mode - Combines vision, browsing, ADA and DALL-E into 1 model (Thread, Examples, System prompt)

* Microsoft CodeFusion paper - a tiny (75M parameters) model beats a 20B GPT-3.5-turbo (Thread, ArXiv)

* Voice

* Hugging Face - Distill whisper - 2x smaller english only version of Whisper (X, paper, code)

* AI Art & Diffusion & 3D

* Luma - text-to-3D Genie bot (Announcement, Try it)

* Stable 3D & Sky changer

AI Regulation IS HERE

Look, to be very frank, I want to focus ThursdAI on all the news that we're getting from week to week, and to bring a positive outlook, so politics, doomerism, and regulation weren't on the roadmap, however, with weeks like these, it's really hard to ignore, so let's talk about this.

President Biden signed an Executive Order, citing the old, wartime era Defence Production act (looks like the US gov. also has "one weird trick" to make the gov move faster) and it wasn't as bombastic as people thought. X being X, there has been so many takes pre this executive order even releasing about regulatory capture being done by the big AI labs, about how open source is no longer going to be possible, and if you visit Mark Andressen feed you'll see he's only reposting AI generated memes to the tune of "don't tread on me" about GPU and compute rights.

However, at least on the face of it, this executive order was mild, and discussed many AI risks and focused on regulating models from huge compute runs (~28M H100 hours // $50M dollars worth). Here's the relevant section.

Many in the open source community reacted to the flops limitation with a response that it's very much a lobbyist based decision, and that the application should be regulated, not only the compute.

There's much more to say about the EO, if you want to dig deeper, I strongly recommend this piece from AI Snake oil :

and check out Yan Lecun's whole feed.

UK AI safety summit in Bletchley Park

Look, did I ever expect to add the King of England into an AI weekly recap newsletter? Surely, if he was AI Art generated or something, not the real king, addressing the topic of AI safety!

This video was played for the attendees of a few day AI safety summit in Blecheley park, where AI luminaries (Yan Lecun, Elon Musk, Arthur Mensch Mistral CEO, Naveen Rao) attended and talked about the risks and benefits of AI and regulation. I think Naveen Rao had a great recap here, but additionally, there were announcements about Safety Institute in the UK, and they outlined what actions the government can take.

In other regulation related news, Mozilla has a joint statement on AI safety and openness (link) that many signed, which makes the case for openness and open source as the way to AI safety. Kudos on mozilla, we stand by the letter 🤝

Big CO LLMs + APIs

OpenAI - ChatGPT "all tools" aka MMIO mode (that's now dubbed "confidential")

Just a week before the first Dev Day from OpenAI, we were hanging out in X spaces talking about what the regulation might bring, when a few folks noticed that their ChatGPT interface looks different, and saw a very specific popup message saying that ChatGPT can now talk with documents, and "use tools without switching", see and interact with DALL-E and Advanced Data Analysis (FKA Code Interpreter) all in one prompt.

While many X takes focused solely on just how many "chat with your PDF" startups OpenAI just "killed", and indeed, the "work with PDFs" functionality seemed new, chatGPT could now get uploads of files, had the ability to search, to go to a specific page, even do a basic summary on PDF files, I was interested in the second part!

Specifically because given GPT-4V is now basically enabled for everyone, this "combined" mode makes ChatGPT the first MMIO model that we have, which is a multi modal on input (Text, Voice, Images) and output (Text, Images). You see, most MultiModal Models so far have been only multimodal on the input, ie, take in text or images or a combination, and while playing around with the above, we noticed some incredible use-cases that are now available!

ChatGPT (for some lucky folks) can now do all these things in one prompt with shared context:

* Read and interact with PDFs

* See and understand images + text

* Browse & Search up to date info with Bing

* Write and execute code with ADA

* Generate images with DALL-E

All in the same prompt, one after another, and often for several steps and iterations.

One such example was, I asked for "get the current weather in Denver and generate an image based on the conditions" and we got this incredible, almost on the fly "weather" UI, showing the conditions (it was the first snow in CO this year), weather, humidity and everything. Now, DALL-E is ok with text but not great, but it's incredible with scenery, so having this "on the fly UI" that has real time info was super great to show off the capabilities of a general model.

We also saw prompts from folks who uploaded a picture of an obscure object, and asked DALL-E to "add" this object to an already generated image, so DALL-E now has eyes, and can understand and "draw" some of the objects and add them to other images, which was an amazing thing to see, and I can't wait to play around with this functionality.

We noticed a few more things, specifically that DALL-E images are now stored on the same disk that you get access to with ADA, so you can then ask ChatGPT to upscale, crop and do things with those images for example, and generate code with those images as a background!

There are so many new potential use-cases that have opened up, that we spent a long evening / night on X spaces trying to throw the kitchen sink onto this mode, in the fear that it was a fluke by OpenAI and they weren't meant to release this, and we were right! Today on ThursdAI live recording, some users reported that they no longer have access to it (and they miss it!) and some reported that it's now called something like "Confidential"

Someone also leaked the full prompt for this "all tools" mode and it's a doosy! The "All Tools" omni-prompt takes a whopping 2,756 tokens, but it's also using the GPT-4 32k model, with a 32,767 token context window. (link)

I guess we're going to see the announcement on Dev Day (did you set a reminder?)

This new mode that we saw and played with, added to the many many leaks and semi-confirmed modes that are coming out of Reddit make it seem like ChatGPT is going to have an all out Birthday party next week and is about to blow some people's minds! We're here for it! 👏

Open Source LLMs

CodeFusion - 75M parameters model based on Diffusion Model for Code Generation

Code-fusion was taken down from ArXiv, claimed 3.5 is 20B params (and then taken down saying that this was unsubstantiated) X link

The paper itself discusses the ability to use diffusion to generate code, and has much less data to get to a very good coding level, with a model small enough to fit on a chip's cache (not even memory) and be very very fast. Of course, this is only theoretical and we're going to wait for a while until we see if this replicates, especially since the PDF was taken down due to someone attributing the 20B parameters note to a forbes article.

The size of the model, and the performance score on some coding tasks make me very very excited about tiny models on edge/local future!

I find the parameter obsession folks have about OpenAI models incredible, because parameter size really doesn’t matter, it's a bad estimation anyway, OAI can train their models for years and keep them in the same parameter size and they would be vastly different models at the start and finish!

Together releases a massive 30T tokens dataset - RedPajama-Data-v2 (Announcement, HF)

This massive massive dataset is 25x the previous RedPajama, and is completely open, deduplicated and has enormous wealth of data to train models from. For folks who were talking the "there's no more tokens" book, this came as a surprise for sure! It's also multi-lingual, with tokens in English, French, Italian, German and Spanish in there. Kudos to Together compute for this massive massive open source effort 👏

Open source Finetunes Roundup

This week was another crazy one for open source fine-tuners, releasing SOTA after SOTA, many of them on ThursdAI itself 😅 Barely possible to keep up (and that's quite literally my job!)

Mistral 7B - 128K (and 64K) (HF)

The same folks who brought you the YaRN paper, Emozilla, Bowen Peng and Enrico Shippole (frequent friends of the pod, we had quite a few conversations with them in the past) have released the longest context Mistral fine-tune, able to take 64K and a whopping 128K tokens in it's context length, making one of the best open source model now compatible with book length prompts and very very long memory!

Capybara + Obsidian (HF, Quantized)

Friend of the pod (and weekly cohost) LDJ releases 2 Nous research models, Capybara (trained on StableLM 3B and Mistral 7B) and Obsidian, the first vision enabled multi modal 3B model that can run on an iPhone!

Capybara is a great dataset that he compiled and the Obsidian model uses the LLaVa architecture for input multimodality and even shows some understanding of humor in images!

Alignment Lab - OpenChat-3.5 a chatGPT beating open source model (Announcement, HF)

According to friends of the pod Alignment Lab (of OpenOrca fame) we get a Mistral finetune that beats! chatGPT on many code based evaluations (from march, we all think chatGPT became much better since then)

OpenChat is by nature a conversationally focused model optimized to provide a very high quality user experience in addition to performing extremely powerfully on reasoning benchmarks.

Open source is truly unmatched, and in the face of a government regulation week, open sources is coming out in full!

Voice

HuggingFace Distill Whisper - 6x performance of whisper with 1% WER rate (Announcement, HF)

Hugging face folks release a distillation of Whisper, a process (and a paper) with which they use a "teacher" model like the original Open AI whisper, to "teach" a smaller model, and in the process of distillation, transfer capabilities from one to another, while also making the models smaller!

This makes a significantly smaller model (2x smaller) with comparative (and even better) performance on some use-cases, while being 6x faster!

This distill-whisper is now included with latest transformers (and transformers.js) releases and you can start using this faster whisper today! 👏

That's it for today folks, it's been a busy busy week, and many more things were announced, make sure to join our space and if you have read all the way until here, DM me the 🧯 emoji as a reply or in a DM, it’s how I know who are the most engaged users are!

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Hey everyone, this is Alex Volkov 👋

This week was an incredibly packed with news, started strong on Sunday with x.ai GrŌk announcement, Monday with all the releases during OpenAI Dev Day, then topped of with Github Universe Copilot announcements, and to top it all of, we postponed the live recording to see what hu.ma.ne has in store for us as AI devices go (Finally announced Pin with all the features)

In between we had a new AI Unicorn from HongKong called Yi from 01.ai which dropped a new SOTA 34B model with a whopping 200K context window and a commercial license by ex-Google China lead Kai Fu Lee.

Above all, this week was a monumental for me personally, ThursdAI has been a passion project for the longest time (240 days), and it led me to incredible places, like being invited to ai.engineer summit to do media, then getting invited to OpenAI Dev Day (to also do podcasting from there), interview and befriend folks from HuggingFace, Github, Adobe, Google, OpenAI and of course open source friends like Nous Research, Alignment Labs, and interview authors of papers, hackers of projects, and fine-tuners and of course all of you, who tune in from week to week 🙏 Thank you!

It's all been so humbling and fun, which makes me ever more excited to share the next chapter. Starting Monday I'm joining Weights & Biases as an AI Evangelist! 🎊

I couldn't be more excited to continue ThursdAI mission, of spreading knowledge about AI, connecting between the AI engineers and the fine-tuners, the Data Scientists and the GEN AI folks, the super advanced cutting edge stuff, and the folks who fear AI with the backing of such an incredible and important company in the AI space.

ThursdAI will continue as a X space, newsletter and podcast, as we'll gradually find a common voice, and continue bringing folks awareness of WandB incredible brand to newer developers, products and communities. Expect more on this very soon!

Ok now to the actual AI news 😅

TL;DR of all topics covered:

* OpenAI Dev Day

* GPT-4 Turbo with 128K context, 3x cheaper than GPT-4

* Assistant API - OpenAI's new Agent API, with retrieval memory, code interpreter, function calling, JSON mode

* GPTs - Shareable, configurable GPT agents with memory, code interpreter, DALL-E, Browsing, custom instructions and actions

* Privacy Shield - Open AI lawyers will protect you from copyright lawsuits

* Dev Day emergency pod with Latent Space with Swyx, Allesio, Simon and Me! (Listen)

* OpenSource LLMs

* 01 launches YI-34B, a 200K context window model commercially licensed and it tops all HuggingFace leaderboards across all sizes (Announcement)

* Vision

* GPT-4 Vision API finally announced, rejoice, it's as incredible as we've imagined it to be

* Voice

* Open AI TTS models with 6 very-realistic, multilingual voices, no cloning tho

* AI Art & Diffusion

* <2.5 seconds full SDXL inference with FAL (Announcement)

OpenAI Dev Day

So much to cover from OpenAI that this has it's own section today in the newsletter.

I was lucky enough to get invited, and attend the first ever OpenAI developer conference (AKA Dev Day) and it was an absolute blast to attend. It was also incredible to attend it together with all 8.5 thousand of you who tuned into our live stream on X, as we were walking to the event, and then watched the keynote together (Thanks Ray for the restream) and talked with OpenAI folks about the updates. Huge shoutout to LDJ, Nisten, Ray, Phlo, Swyx and many other folks who held the space, while we were otherwise engaged with deep dives and meeting folks and doing interviews!

So now for some actual reporting! What did we get from OpenAI? omg we got so much, as developers, as users (and as attendees, I will add more on this later)

GPT4-Turbo with 128K context length

The major thing that was announced is a new model, GPT-4-turbo, which is supposedly faster than GPT-4, while being 3x cheaper (2x on output) and having a whopping 128K context length while also being more accurate (with significantly better recall and attention throughout this context length)

With JSON mode and significantly improved function calling capabilities, updated cut-off time (April 2023), and higher rate limits, this new model is already being implemented across all the products and is a significant significant upgrade to many folks

GPTs - A massive shift in agent landscapes by OpenAI

Another (semi-separate) thing that Sam talked about was the GPTs, their version of agents

not to be confused with the Assistants API, which is also Agents, but for developers, and they are not the same and it's confusing

GPTs I think is a genius marketing move by OpenAI and replaces Plugins (that didn't even meet product market fit) in many regards.

GPTs are instances of well... GPT4-turbo, that you can create by simply chatting with BuilderGPT, and they can have their own custom instruction set, and capabilities that you can turn on and off, like browse the web with Bing, Create images with DALL-E and write and execute code with Code Interpreter (bye bye Advanced Data Analysis, we don't miss ya).

GPTs also have memory, you can upload a bunch of documents (and your users as well) and GPTs will do vectorization and extract the relevant information out of those documents, so think, your personal Tax assistant that has all 3 years of your tax returns

And they have eyes, GPT4-V is built in so you can drop in screenshots, images and all kinds of combinations of things.

Additionally, you can define actions for assistants (which is similar to how Plugins were developed previously, via an OpenAPI schema) and the GPT will be able to use those actions to do tasks outside of the GPT context, like send emails, check stuff in your documentation and much more, pretty much anything that's possible via API is now possible via the actions.

One big thing that's missing for me is, GPTs are reactive, so they won't reach out to you or your user when there's a new thing, like a new email to summarize or a new task completed, but I'm sure OpenAI will close that gap at some point.

GPTs are not Assistants, they are similar but not the same and it's quite confusing. GPTs are created online, and then are share-able with links.

Which btw, I created a GPT that uses several of the available tools, browsing for real time weather info, and date/time and generates an on the fly, never seen before weather art for everyone. It's really fun to play with, let me know what you think (HERE) the image above is generated by the Visual Weather GPT

Unified "All tools" mode for everyone (who pays)

One tiny thing that Sam mentioned on stage, is in fact huge IMO, is the removal of the selector in chatGPT, and all premium users now have access to 1 interface that is multi modal on input and output (I call it MMIO) - This mode now understands images (vision) + text on input, and can browse the web and generate images, text, graphs (as it runs code) on the output.

This is a significant capabilities upgrade to many folks who will use these tools, but previously didn't want to choose between DALL-E mode and browse or Code Interpreter mode. The model now intelligently selects what tool to use for a given task, and this means more and more "generality" for the models, as they are learning and getting new capabilities in the form of tools.

This in addition to a MASSIVE 128K context window, means that chatGPT has now been significantly upgraded, and you still pay $20/mo 👏 Gotta love that!

Assistant API (OpenAI Agents)

This is the big announcement for developers, we all got access to a new and significantly improved Assistants API, which improves on several our experience on several categories:

* Creating Assistants - Assistants are OpenAI's first foray into the world of AGENTS, and it's quite exciting! You can create an assistant via API (not quite the same as GPTs, we'll cover the differences later), you can create each assistant with it's own set of instructions (that you don't have to pass each time with the prompt), tools like code interpreter and retrieval, and functions. Also you can select models, so you don't have to use the new GPT-4-turbo (but you should!)

* Code Interpreter - Assistants are able to write and execute code now, which is a whole world of excitement! Having code abilities (that executes in a safe environment on OAI side) is a significant boost in many regards and many tasks require bits of code "on the fly", for example time-zone tasks. You will no longer have to write that code yourself, you can ask your assistant

* Retrieval - OpenAI (and apparently QDrant!) have given all the developers a built in RAG (retrieval augmented generation) capabilities + document uploading and understanding. You can upload files like documentation via the API or let your users upload files, and parse and extract information out of! This is an additional huge huge thing, basically memory is built in for you now

* Stateful API - this API introduces the concept of threads, where OpenAI will manage the state of your conversation, and you can assign 1 user per thread and then just send the responses back to the user, and send the user queries to the same thread. No longer do you have to send the whole history back and forth! It's quite incredible, however it raises the question of pricing, and calculating tokens. Per OpenAI (I asked), if you would like to calculate costs on the fly, you'd have to use the get thread endpoint, and then count the amount of tokens that's already in the thread (and it can be a LOT since there's now 128K tokens in the context length)

* JSON and Better functions calling - You can now set the API to respond in JSON mode! Which is an incredible improvement for devs, and which we only were able to do via Functions before, and even functions got an upgrade, with an ability to call multiple functions. Functions are added as "actions" in the assistant creation, so you can give your assistant abilities that it will execute by returing to you functions with the right parameters. Thing "set the mood" will return a function to call the smart lights, and "play" will return a function that will call Spotify API

* Multiple Assistants can join a thread - you can create specific assistants that can all join the same thread with the user, each with a set of custom instructions and capabilities and tools

* Parallel Functions - this is also new, the assistant API can now return several functions for you to execute, which could lead to the creation of scenes, for example in a smart home, you want to "set the mood" and several functions would be returned from the API, one that will turn of the lights, one that will start the music, and one that will turn on mood lighting.

Vision

GPT-4 Vision

Finally, it's here, multimodality for developers to implement, the moment I personally have been waiting for since GPT-4 was launched (and ThursdAI started) back on March 14 (240 days ago, but who's counting)

GPT-4 vision is able to take images, and text, and respond with many vision related tasks, like analysis, understanding, summarization of captions. Many folks are splitting videos frame by frame and analyzing whole videos already (in addition to whispering the video to get what is said)

Hackers and developers like friend of the pod Robert, created quick hacks like a browser extension that lets you select any screenshot on the page and ask GPT4 vision things about it, another friend of the pod SkalskiP created a hot dog classifier Gradio space 😂 and is maintaining an awesome list of experiments with vision on Github

Voice

Text to speech models

OpenAI decided to help us all build agents properly, and agents need not only ears (for which they gave us whisper, and released V3 as well) but also voice, and we finally got the TTS from OpenAI, 6 very beautiful, emotional voices, that you can run very easily, and cheaply. You can't generate more or clone yet (that's only for friends of OpenAI like Spotify and others) but you can use the 6 we got (plus a secret pirate one apparently they trained but never released!)

They sound ultra-realistic, and are multi-linugal as well, you can just pass different languages and voila. Friend of the pod Simon Willison created a quick CLI tool called ospeak to pipe text into and it'll use your OAI key to read that text out with those super nice voices!

Whisper v3 was released!

https://github.com/openai/whisper/discussions/1762

The large-v3 model shows improved performance over a wide variety of languages, and the plot below includes all languages where Whisper large-v3 performs lower than 60% error rate on Common Voice 15 and Fleurs, showing 10% to 20% reduction of errors compared to large-v2:

HUMANE

Humane AI pin is ready for pre-order at 699

HUMANE pin was finally announced, and here is the break-down, they have a clever way to achieve "all day battery life" by having a hot swap system, with a magnetic booster that you can swap when you get low on battery (pretty genius TBH)

It's passive so it's not "always listening" but there is a wake word apparently, and you can activate by touch. Runs on the T-mobile Network ( which sucks for folks like me where T-mobile just doesn't have reception in their neighborhood 😂 )

No apps, just AI experiences powered by OpenAI, with a laser powered projector UI on your hand, and voice controls

AI voice input will allow interactions like asking for information (which has browsing) and is SIGNIFICANTLY better than "Siri" or "Ok Google" from the demo, being able to rewrite your messages for you, catch you up on multiple messages and even search through them! You can ask for retrieval from previous messages

Pin is multimodal, voice input and vision

Holding the microphone on Tab while someone's speaking to you in a different language will automatically translate that language for you and then translate you back to that language with your own intonation! Bye bye language barriers!

And with vision, you can do things like tracking calories from showing it what you ate, or buy things you're seeing in the store, but online, take pictures and videos and then store all of them transcribed in your personal AI memory

Starting at $699, with a $24/mo payment that comes with unlimited AI queries, storage and service (again, just T-mobile), Tidal music subscription and more.

I think it's lovely that someone tries to take on Google/Apple duopoly with a completely re-imagined AI device, and can't wait to pre-order mine and test it out. It will be an interesting experience of balance with 2 phone numbers, but also a monthly payment that basically makes the device use-less if you stop paying.

Phew, this was a big update, not to mention there's a whole 2 hour podcast I want you to listen to on top of this, thank you for reading, for subscribing, for participating in the community and I can't wait to finally relax after this long week (still Jet-lagged) and prepare for my new Monday!

I want to send a heartfelt shoutout to my friend swyx who not only let me on to Latent Space from time to time (including the last recap emergency pod), but also is my blood-line to SF, where everything happens! Thanks man, I really appreciate all you did for me and ThursdAI 🫡

Can't wait to see you all on the next ThursdAI, and as always, replies, comments, congratulations, are welcome as replies, DMs and send me the 🎉 for this one, I'd really appreciate it!

— Alex

Hey yall, welcome to this special edition of ThursdAI. This is the first one that I'm sending in my new capacity as the AI Evangelist Weights & Biases (on the growth team)

I made the announcement last week, but this week is my first official week at W&B, and oh boy... how humbled and excited I was to receive all the inspiring and supporting feedback from the community, friends, colleagues and family 🙇‍♂️

I promise to continue my mission of delivering AI news, positivity and excitement, and to be that one place where we stay up to date so you don't have to.

This week we also had one of our biggest live recordings yet, with 900 folks tuned in so far 😮 and it was my pleasure to again to chat with folks who "made the news" so we had a brief interview with Steve Ruiz and Lou from TLDraw, about their incredible GPT-4 Vision enabled "make real" functionality and finally got to catch up with my good friend Idan Gazit who's heading the Github@Next team (the birthplace of Github Copilot) about how they see the future. So definitely definitely check out the full conversation!

TL;DR of all topics covered:

* Open Source LLMs

* Nous Capybara 34B on top of Yi-34B (with 200K context length!) (Eval, HF)

* Microsoft - Phi 2 will be open sourced (barely) (Announcement, Model)

* HF adds finetune chain genealogy (Announcement)

* Big CO LLMs + APIs

* Microsoft - Everything is CoPilot (Summary, copilot.microsoft.com)

* CoPilot for work and 365 (Blogpost)

* CoPilot studio - low code "tools" builder for CoPilot + GPTs access (Thread)

* OpenAI Assistants API cookbook (Link)

* Vision

* 🔥 TLdraw make real button - turn sketches into code in seconds with vision (Video, makereal.tldraw.com)

* Humane Pin - Orders are out, shipping early 2024, multimodal AI agent on your lapel (

* )

* Voice & Audio

* 🔥 DeepMind (Youtube) - Lyria high quality music generations you can HUM into (Announcement)

* EmotiVoice - 2000 different voices with emotional synthesis (Github)

* Whisper V3 is top of the charts again (Announcement, Leaderboard, Github)

* AI Art & Diffusion

* 🔥 Real-time LCM (latent consistency model) AI art is blowing up (Krea, Fal Demo)

* 🔥 Meta announces EMU-video and EMU-edit (Thread, Blog)

* Runway motion brush (Announcement)

* Agents

* Alex's Visual Weather GPT (Announcement, Demo)

* AutoGen, Microsoft agents framework is now supporting assistants API (Announcement)

* Tools

* Gobble Bot - scrape everything into 1 long file for GPT consumption (Announcement, Link)

* ReTool state of AI 2023 - https://retool.com/reports/state-of-ai-2023

* Notion Q&A AI - search through a company Notion and QA things (announcement)

* GPTs shortlinks + analytics from Steven Tey (

https://chatg.pt

* )

This Week's Buzz from WandB (aka what I learned this week)

Introducing a new section in the newsletter called "The Week's Buzz from WandB" (AKA What I Learned This Week).

As someone who joined Weights and Biases without prior knowledge of the product, I'll be learning a lot. I'll also share my knowledge here, so you can learn alongside me. Here's what I learned this week:

The most important things I learned this week is just how prevelant and how much of a leader Weights&Biases is. W&B main product is used by most of the foundation LLM trainers including OpenAI.

In fact GPT-4 was completely trained on W&B!

It's used by pretty much everyone besides Google. In addition to that it's not only about LLMs, W&B products are used to train models in many many different areas of the industry.

Some incredible examples are a pesticide dispenser that's part of the John Deere tractors that only spreads pesticides onto weeds and not actual produce. And Big Pharma who's using W&B to help create better drugs that are now in trial. And it's just incredible how much machine learning that's outside of just LLMs is there. But also I'm absolutely floored by just the amount of ubiquity that W&B has in the LLM World.

W&B has two main products, Models & Prompts, Prompts is a newer one, and we're going to dig into both of these more next week!

Additionally, it's striking how many AI Engineers, API users such as myself and many of my friends, have no idea of who W&B even is, of if they do, they never used it!

Well, that's what I'm here to change, so stay tuned!

Open source & LLMs

In the open source corner, we have the first Nous fine-tune of Yi-34B, which is a great model that we've covered in the last episode and now is fine-tuned with the Capybara dataset by ThursdAI cohost, LDJ! Not only is that a great model, it now tops the charts for the resident reviewer we WolframRavenwolf on /r/LocalLLama (and X)

Additionally, Open-Hermes 2.5 7B from Teknium is now second place on HuggingFace leaderboards, it was released recently but we haven't covered until now, I still think that Hermes is one of the more capable local models you can get!

Also in open source this week, guess who loves it? Satya (and Microsoft)

They love it so much that they not only created this awesome slide (altho, what's SLMs? Small Language Models? I don't like it), they also announced that LLaMa and Mistral are coming to Azure services as inference!

And they gave us a little treat, Phi2 is coming. They said OpenSource (but folks looking at the license saw that it's only for research capabilities) but supposedly it's a significantly more capable model while only being 2.7B weights (super super tiny)

Big Companies & APIs

Speaking of Microsoft, they announced so much during their Ignite event on Wednesday (15th) that it's impossible to cover all of it in this newsletter, but basically here are the main things that got me excited!

CoPilot everywhere, everything is CoPilot

Microsoft rebranded Bing Chat to Copilot and it now lives on copilot.microsoft.com

and it's basically a free GPT4, with vision and DALL-e capabilities. If you're not signed up for OpenAI's plus membership, this is as good as it gets for free!

They also announced CoPilot for 365, which means that everything from office (word, excel!) to your mail, and your teams conversations will have a CoPilot that will be able to draw from your organizations knowledge and help you do incredible things. Things like help book appointments, pull in relevant people for the meeting based on previous documents, summarize that meeting, schedule follow ups, and like a TON more stuff. Dall-e integration will help you create awesome powerpoint slides.

(p.s. all of this will be allegedly data protected and won't be shared with MS or be trained on)

They literally went and did "AI everywhere" with CoPilot and it's kinda incredible to see how big they are betting the farm on AI with Microsoft while Google... where's Google™?

CoPilot Studio

One of the more exciting things for me was, the CoPilot Studio announcement, a low-code tool to extend your company's CoPilot by your IT, for your organization. Think, getting HR data from your HR system, or your sales data from your SalesForce!

They will launch with 1100 connectors for many services but allow you to easily build your own.

One notable thing is, Custom GPTs will also be a connector! You will be literally able to connect yyour CoPilot with your (or someone's) GPTs! Are you getting this? AI Employees are coming faster than you think!

Vision

I've been waiting for cool vision demos since GPT-4V API was launched and oh boy did we get them! From friend of the pod Robert Lukoshko Auto screenshot analysis which will take screenshots periodically and will send you a report of all you did that day, to Charlie Holtz live webcam narration by David Attenborough (which is available on Github!)

But I think there's 1 vision demo that takes the cake this week, by our friends (Steve Ruiz) from TLDraw, which is a whiteboard canvas primitive. They have added a sketch-to-code button, that allows you to sketch something out and GPT-4 Vision will analyze this, and GPT-4 will write code, and you will get live code within seconds. It's so mind-blowing that I'm still collecting my jaw of the floor here. It also does coding, so if you ask it nicely to add JS interactivity, the result will be interactive 🤯

GPT4-V Is truly as revolutionary as I imagined it to be when Greg announced it on stage 🫡

P.S - Have you played with it? Do you have cool demos? DM me with 👁️‍🗨️ emoji and a cool vision demo to be included in the next ThursdAI

AI Art & Diffusion & 3D

In addition to the TLDraw demo, one mind-blowing demo after another is coming this week from the AI Art world, using the LCM (Latent Consistency Model) + a whiteboard. This is yet another see it to believe it type thing (or play with it)

(video from Linus)

Dear friends from Krea.ai were the first to implement this insanity, that allows you to see real time AI art generation almost as fast as you type your prompts, and then followed up by the wizards at Fal to get the generations down to several mili-seconds (shoutout Gorkem!), the real time drawing thing is truly truly mind-blowing. It's so mind-blowing that folks add their webcam feeds into this, and see almost real time generation on the fly of their webcam feeds.

Meta announcing new Emus (Video & Edit)

Meta doesn't want me to relax, and during the space, announced their text-to-video and textual-editing models.

Emu Video produces great videos from a prompt, and emu-edit is really interesting, it allows you to edit parts of images by typing, think "remove the tail from this cat" or "remove the hat from this person"

They have this to say, which... dayum.

In human evaluations, our generated videos are strongly preferred in quality compared to all prior work– 81% vs. Google’s Imagen Video, 90% vs. Nvidia’s PYOCO, and 96% vs. Meta’s Make-A-Video. Our model outperforms commercial solutions such as RunwayML’s Gen2 and Pika Labs

It's really compelling, can't wait to see if they open source this, video is coming ya'll!

Audio & Sound

Deepmind + Youtube announced Lyria (blogpost)

This new music model is pretty breathtaking, but we only got a glimpse, not even a waitlist for that one, however, check out the pre-recorded demoes, folks at deep mind have a model you can hum into, sing into, and it'll create a full blown track for you, with bass, drums, and singing!

Not only that, it will also license vocals from mucisians (al-la Grimes) and will split the revenue between you and them if you post it on Youtube!

Pretty cool Google, pretty cool!

Agents & Tools

Look, I gotta be honest, I'm not sure about this category, Agent and Tools, if to put them into one or not, but I guess GPTs are kinda tools, so I'm gonna combine them for this one.

GPTs (My Visual Weather, Simons Notes)

This week, the GPT that I created called Visual Weather GPT has blown up, with over 5,000 chats opened with it, and many many folks using this and texting me about this. super cool way to just like check all the capabilities of a GPT. If you remember, I thought of this idea a few weeks ago when we got a sneak preview to the "All tools" mode, but now I can share it with you all in the form of a GPT, that will browser the web for real time weather data, and create a unique art piece for that location and weather conditions!

It's really easy to make as well, and I do fully expect everyone to start making their own versions very soon, and I think we're inching towards the era of JIT (just in time) software, where you'll create software as you require it, and it'll be as easy as talking to a chatGPT!

Speaking of, friend of the pod Steven Tey from Vercel (who's dub.sh I use and love for thursdai.news links) has released a GPT link shortener, called chatg.pt and you can register and get your own cool short link like https://chatg.pt/artweather 👏 And it'll give you analytics as well!

Pro tip for weather GPT, you can ask for a specific season or style in parentesises and then those as greeting cards for your friends. Happy upcoming Thanks giving everyone!

Speaking of Thanks

giving, we're not taking a break, next ThursdAI, November 23, join us for a live discussion and podcast recoding! We'll have many thanks, cool AI stuff, and much more!

ThursdAI TL;DR - November 23

TL;DR of all topics covered:

* OpenAI Drama

* Sam... there and back again.

* Open Source LLMs

* Intel finetuned Mistral and is on top of leaderboards with neural-chat-7B (Thread, HF, Github)

* And trained on new Habana hardware!

* Yi-34B Chat - 4-bit and 8-bit chat finetune for Yi-34 (Card, Demo)

* Microsoft released Orca 2 - it's underwhelming (Thread from Eric, HF, Blog)

* System2Attention - Uses LLM reasons to figure out what to attend to (Thread, Paper)

* Lookahead decoding to speed up LLM inference by 2x (Lmsys blog, Github)

* Big CO LLMs + APIs

* Anthropic Claude 2.1 - 200K context, 2x less hallucinations, tool use finetune (Announcement, Blog, Ctx length analysis)

* InflectionAI releases Inflection 2 (Announcement, Blog)

* Bard can summarize youtube videos now

* Vision

* Video-LLaVa - open source video understanding (Github, demo)

* Voice

* OpenAI added voice for free accounts (Announcement)

* 11Labs released speech to speech including intonations (Announcement, Demo)

* Whisper.cpp - with OpenAI like drop in replacement API server (Announcement)

* AI Art & Diffusion

* Stable Video Diffusion - Stability releases text2video and img2video (Announcement, Try it)

* Zip-Lora - combine diffusion LORAs together - Nataniel Ruiz (Annoucement, Blog)

* Some folks are getting NERFs out from SVD (Stable Video Diffusion) (link)

* LCM everywhere - In Krea, In Tl;Draw, in Fal, on Hugging Face

* Tools

* Screenshot-to-html (Thread, Github)

Ctrl+Altman+Delete weekend

If you're subscribed to ThursdAI, then you most likely either know the full story of the crazy OpenAI weekend. Here's my super super quick summary (and if you want a full blow-by-blow coverage, Ben Tossel as a great one here)

Sam got fired, Greg quit, Mira flipped then Ilya Flipped. Satya played some chess, there was an interim CEO for 54 hours, all employees sent hearts then signed a letter, neither of the 3 co-fouders are on the board anymore, Ilya's still there, company is aligned AF going into 24 and Satya is somehow a winner in all this.

The biggest winner to me is open source folks, who got tons of interest suddenly, and specifically, everyone seems to converge on the OpenHermes 2.5 Mistral from Teknium (Nous Research) as the best model around!

However, I want to shoutout the incredible cohesion that came out of the folks in OpenAI, I created a list of around 120 employees on X and all of them were basically aligned the whole weekend, from ❤️ sending to signing the letter, to showing how happy they are Sam and Greg are back!

Yay

This Week's Buzz from WandB (aka what I learned this week)

As I’m still onboarding, the main things I’ve learned this week, is how transparent Weights & Biases is internally. During the whole OAI saga, Lukas the co-founder sent a long message in Slack, addressing the situation (after all, OpenAI is a big customer for W&B, GPT-4 was trained on W&B end to end) and answering questions about how this situation can affect us and the business.

Additionally, another co-founder, Shawn Lewis shared a recording of his update to the BOD of WandB, about out progress on the product side. It’s really really refreshing to see this information voluntarily shared with the company 👏

The first core value of W&B is Honesty, and it includes transparency outside of matters like personal HR stuff, and after hearing about this during onboarding, it’s great to see that the company lives it in practice 👏

I also learned that almost every loss curve image that you see on X, is a W&B dashboard screenshot ✨ and while we do have a share functionality, it’s not built for viral X sharing haha so in the spirit of transparency, here’s a video I recorded and shared with product + feature request to make these screenshot way more attractive + clear that it’s W&B

Open Source LLMs

Intel passes Hermes on SOTA with a DPO Mistral Finetune (Thread, Hugging Face, Github)

Yes, that intel, the... oldest computing company in the world, not only comes out strong with the best (on benchmarks) open source LLM, it also does DPO, and has been trained on a completely new hardware + Apache 2 license!

Here's Yam's TL;DR for the DPO (Direct Policy Optimization) technique:

Given a prompt and a pair of completions, train the model to prefer one over the other. This model was trained on prompts from SlimOrca's dataset where each has one GPT-4 completion and one LLaMA-13B completion. The model trained to prefer GPT-4 over LLaMA-13B.

Additionally, even tho there is custom hardware included here, Intel supports the HuggingFace trainer fully, and the whole repo is very clean and easy to understand, replicate and build things on top of (like LORA)

LMSys Lookahead decoding (Lmsys, Github)

This method significantly improves the output of LLMs, sometimes by more than 2x, using some jacobian notation (don't ask me) tricks. It's copmatible with HF transformers library! I hope this comes to open source tools like LLaMa.cpp soon!

Big CO LLMs + APIs

Anthropic Claude comes back with 2.1 featuring 200K context window, tool use

While folks on X thought this was new, Anthropic actually announced Claude with 200K back in the May, and just gave us 100K context window, which for the longest time was the largest context window around. I was always thinking, they don't have a reason to release 200K since none of their users actually wants it, and it's a marketing/sales decision to wait until OpenAI catches up. Remember, back then, GPT-4 was 8K and some lucky folks got 32K!

Well, OpenAI releases GPT-4-turbo with 128K so Anthropic re-trained and released Claude to gain an upper hand. I also love the tool use capabilities.

Re: longer context window, there were a bunch of folks testing if 200K context window is actually all that great, and it turns out, besides being very expensive to run (you pay per tokens) it also loses a bunch of information at lengths over 200K using needle in the haystack searches. Here's an analysis by Greg Kamradt that shows that:

* Starting at ~90K tokens, performance of recall at the bottom of the document started to get increasingly worse

* Less context = more accuracy - This is well know, but when possible reduce the amount of context you send to the models to increase its ability to recall.

I had similar issues back in May with their 100K tokens window (source)

Voice & Audio

ElevenLabs has speech-to-speech

Creating a significant jump in capabilities, ElevenLabs now allows you to be the actor behind the voice! With speech to speech, they would transfer the pauses, the intonation, the emotion, into the voice generation. Here's my live reaction and comparison:

* Notable: Whisper.CPP now supports a server compatible with OpenAI (Announcement, Github)

AI Art & Diffusion

Stable Video diffusion - text-2-video / img-2-video foundational model (Announcement, Hugging Face, Github, DEMO)

Stable has done it again, Stable Video allows you to create increidbly consistent videos with images or just text! They are short now, but working on extending the times, and they videos look incredible! (And thanks to friends at Fal, you can try right now, here)

And here’s a quick gif I created with DALL-E 3 and Fal to celebrate the Laundry Buddy team at OAI while the outage was happening)

Tools

Screenshot to HTML (Github)

I… what else is there to say? Someone used GPT4-Vision to … take screenshots and iteratively re-create the HTML for them. As someone who used to spend month on this exact task, I’m very very happy it’s now automated!

Happy Thanksgiving 🦃

I am really thankful to all of you who subscribe and come back every week, thank you! I would have been here without all your support, comments, feedback! Including this incredible art piece that Andrew from spacesdashboard created just in time for our live recording, just look at those little robots! 😍

See you next week (and of course the emoji of the week is 🦃, DM or reply!)

🎶 Happy birthday to you, happy birthday to you, happy birthday chat GPT-eeeeeeee, happy birthday to you.

Hey everyone, welcome to this special edition of ThursdAI where you're probably gonna have two emails and two podcast episodes today and you can choose which one you want to but we actually recorded both of them live it just they went a little long.

ThursdAI - The only podcast that brings you yearly recaps since chatGPT was released (😂)

This one is the more celebratory one, today is one year from the release of chat GPT and we (and by we I mean I, Alex) decided to celebrate it by recapping not just the last week in AI but the last year (full timeline posted at the bottom of this newsletter)

Going month by month with a swoosh sound in the editing and covering the most important thing that happened in LLM and open source LLMs since chatGPT was released and imagination unlocked the capability for everyone!

We also covered Meta stepping in with Lama and then everything that happened since then in the multi modality and vector databases and agents and everything everything everything, it was a one hell of an hour and a half, we had almost 1K audience members! and so I recommend you listen to this one first and then the week updates later because there were some incredible releases this week as well! (as there are every week)

I think it's important to do like a Spotify wrapped type thing for AI, for something like a one year for chat GPT and I think we'll be doing this every year so hopefully in the year we'll see you here on November 30th covering the next year in AI.

And hopefully the next year in AI system will actually help me summarize all this because it's a lot of work but with that I will just leave you with the timeline and no notes and you should listen to everything because we talked about everything live!

I hope you enjoy this special birthday celebration! (OpenAI sure did, check out this incredibly cute little celebration video they just posted)

Here’s the full timeline with everything important that happened month by month that we’ve covered:

* December 2022 - ChatGPT becomes the fastest growing product in history

* GPT3.5 with 4K context window, instruction finetuning and conversational RLHF

* January

* Microsoft invests additional $10B into OpenAI (Jan 23, Blog)

* February

* LLaMa 1 - Biggest Open Source LLM (February 24 - Blog)

* No commercial license

* 30% MMLU

* No instruction fine-tuninig (RL;HF)

* ChatGPT unofficial APIs exist

* March (the month of LLM superpowers)

* ChatGPT API (March 1, announcement)

* Developers can now build chatGPT powered apps

* All clones so far were completion based and not conversation based

* LLama.cpp from ggerganov + Quantization (March 10, Blog)

* Stanford - Alpaca 7B - Finetune on self-instruct GPT3.5 dataset (March 13, Blog)

* GPT4 release + chatGPT upgrade (March 14 - GPT-4 demo)

* 67.0% HumanEval | 86.4% MMLU

* 8K (and 32K) context windows

* Anthropic announces Claude + Claude instant (March 14 - Blog)

* 56.0% HumanEval

* Folks previously form OAI leave to open Anthropic as research, then pivot from research into commercial

* LMSYS Vicuna 13B - Finetuned based on shareg.pt exports (March 30, Blog)

* April (Embedings & Agents)

* AutoGPT becomes the fastest github starred project + writes it's own code (April 1, Blog)

* Agents start to pop up like mushrooms after the rain

* LLaVa - Multimodality open source begins (April 18, Blog)

* CLIP + Vicuna smushed together to get LLMs eyes

* Bard improvements

* May (Context windows)

* Mosaic MPT-7B with 64K context, 1T parameters, commercial license (May 5, Blog)

* Anthropic updates Claude with 100K context window (May 11, Blog)

* LLongBoi summer begins (Context windows are being stretched)

* Nvidia shows Voyager agents that play Minecraft + Memory stored in Vector DB (May 27, Blog)

* June

* GPT-3.5-turbo + functions API (June 6, Blog)

* GPT3.5 and 4 got a boost in capabilities and steer-ability

* Price reduction on models + 75% reduction on ada embeddings model

* LLaMa context window extended to 8K with RoPE scaling

* AI Engineers self determination essay by swyx

* July

* Code Interpreter GA - ChatGPT can code (July 11, Blog)

* Anthropic Claude 2 - (July 11 - Blog)

* 200K context window

* 71% HumanEval

* LLaMa 2 (July 18 - Blog)

* Base & Chat models (RLHF)

* Commercial license

* 29.9% Human Eval | 68.9% MMLU

* August

* Meta releases Code-LlaMa, code finetune models

* September

* DALL-E 3 - Adds multi-modality on output and chat to image gen (Sep 20, Blog)

* Mistral 7B top performing open source LLM via torrent link (Sep 27, Blog)

* GPT4-V (vision & voice) - Adds multimodality on input (Sep 27, Blog)

* October

* OpenHermes - Mistral 7B finetune that tops the charts from Teknium / Nous Research (Oct 16, Announcement)

* Inflection PI gets connected to the web + supportPi mode (Oct 16, Blog)

* Adept releases multimodal FuYu 8B (Oct 19, blog)

* November

* Grok from Xai - with realtime access to all of X content

* OpenAI dev day

* Combined mode for MMIO (multi modal on input and output)

* GPT-4 Turbo with 128K context, 3x cheaper than GPT-4

* Assistants API with retrieval capabilities

* Share-able GPTs - custom versions of GPT with retrieval, DALL-E, Code Interpreter and vision

* Chatbots with real business use-cases, for example WandBot (that we just launched today! Blog)

* Has vector storage memory

* Available via Discord/Slack

* And custom GPT!

* Microsoft has copilot everywhere in office

Aaaand now we’re here!

What an incredible year, can’t imagine what the next year holds for all of us, but 1 thing is for sure, ThursdAI will be here to keep you all up to date!

P.S - If you scrolled all the way to here, DM me the 🎊 emoji so I know you celebrated with us! It really helps me to know that there is at least a few folks out of the thousands that get this newsletter that scrolls all the way through!

ThursdAI December 7th TL;DR

Greetins of the day everyone (as our panelist Akshay likes to sometimes say) and Happy first candle of Hannukah for those who celebrate! 🕎

I'm writing this newsletter from the back of an Waymo self driving car, in SF, as I'm here for just a few nights (again) to participate in the Open Source AI meetup, that was co-organized by Ollama and Nous Research, Alignment Labs and hosted by A16Z in their SF office.

This event was the highlight of this trip, it was quite a packed meetup in terms of AI talent, and I got to meet quite a few ThursdAI listeners, mutuals on X, and AI celebs

We also recorded the podcast this week from the arena, thanks to Swyx and Alessio from latentspace pod for hosting ThursdAI this week form their newly built out pod studio (and apologies everyone for the rocky start and the cutting out issues, luckily we had local recordings so the pod version sounds good!)

Google finally teases Gemini Ultra (and gives us Pro)

What a week folks, what a week, as I was boarding the flight to SF to meet with Open Source folks, Google announced (finally!) the release of Gemini, their long rumored, highly performant model with a LOT of fanfare!

Blogposts authored by Sundar and Demis Hassabis, beautiful demos of unseen before capabilities, comparisons to GPT-4V which the Ultra version of Gemini outperforms on several benchmarks, and rumors that Sergey Brin, the guy who's net worth is north of 100Bn is listed as the core contributor on the paper and reports on benchmarks (somewhat skewed) show Ultra beaing GPT-4 on many coding and reasoning evaluations!

We've been waiting for Gemini for such a long time, that we spend the first hour of the podcast discussing it and it's implications basically. We were also fairly disillusioned by the sleight of hand tricks Google marketing department played with the initial launch video, where it purportedly shows Gemini being a fully multi-modal AI, that reacts to a camera feed + user voice in real time, when in fact, it was quickly clear (from their developer blog) that it was not video+audio but rather images+text (the same two modalities we already have in GPT-4V and given some prompting, it's quite easy to replicate most of it. We've also discussed how we again, got a tease, and not even a waitlist for the "super cool" stuff, while getting a GPT3.5 level of a model today in Bard upgrade.

To me, the most mind-blowing demo video was actually one of the other ones in the announcement, which showed that Gemini has agentic behavior in understanding user intent, asks for clarifications, creates a PRD (Product Requirement Document) for itself, and then, generates Flutter code to create a UI on the fly, based on what the use asked it! This is pretty wild, as we all should expect that Just In Time UI will come to many of these big models!

Tune in to the episode if you want to hear more takes, opinions and frustrations as none of us actually got to use Gemini Ultra, and the experience with Gemini Pro (which is now live on Bard) was at least for me, underwhelming

This weeks buzz (What I learned in Weights & Biases this week)

I actually had a blast talking about W&B to many of the open source and fine-tuners community this and past week. I already learned that W&B doesn't only help huge companies (like OpenAI, Anthropic, Meta, Mistral and tons more) to train their foundational models, but is widely used by the open source fine-tuners community as well. I've met with folks like Wing Lian (aka Caseus), maintainer of Axolotl, who uses W&B together with Axolotl, and got to geek out about W&B, met with Teknium and LDJ (Nous Research, Alignment Labs) and in fact, got LDJ to walk me through some of the ways he uses and used W&B in the past, including how it's used to track model runs, show artifacts in the middle of runs, and run mini-benchmarks and evaluations for LLMS as they finetune.

If you're interested in this, here's an episode of a new “series” of me learning publicly (from scratch) so if you want to learn from scratch with me, welcome to check it out:

Open Source AI in SF meetup

This meetup was the reason I flew in to SF, I was invited by dear friends in the open source community, and couldn't miss it! There was such a talent density there, it was quite remarkable. Andrej Karpathy who's video about LLM I just finished re-watching, Jeremy Howard, folks from Mistral, A16Z, and tons of other startups, open source collectives, and enthusiasts, all came together to listen to a few lightning talks, but mostly to mingle and connect and share ideas.

Nous Research announced that they are a company (not anymore just a discord collective of rag tag open sourcers!) and that they are working on Forge, a product offering of theirs, that runs local AI, has a platform for agent behavior, and is very interesting to keep an eye for.

I've spent most of my time going around, hearing what folks are using (Hint: a LOT of axolotl), what they are finetuning (mostly Mistral) and what is the future (everyone's waiting for next Llama or next Mistral). Funnily enough, there was not a LOT of conversation about Gemini there at all, at least not among the folks that I talked to!

Overall this was really really fun, and of course, being in SF, at least for me, especially now as an AI Evangelist, feels like coming home! So expect more trip reports!

Here's a recap and a few more things that happened this week in AI:

* Open Source LLMs

* Apple released MLX - machine learning framework on apple silicon

* Mamba - transformers alternative architecture from Tri Dao

* Big CO LLMs + APIs

* Google Gemini beats GPT-4V on a BUNCH of metrics, shows cool fake multimodal demo

* Demo was embellished per the google developer blog

* Multimodal capabilities are real

* Dense model vs MOE

* Multimodel on the output as well

* For 5-shot, GPT-4 outperforms Gemini Ultra on MMLU

* AlphaCode 2 is here and Google claims it performs better than 85% competitive programmers in the world and it performs even better, collaborating with a competitive programmer.

* Long context prompting for Claude 2 shows 27% - 98% increase by using prompt techniques

* X.ai finally released grok to many premium+ X subscribers. (link)

* Vision

* OpenHermes Vision finally released - something there was not right there, back to drawing board

* Voice

* Apparently Gemini beats Whisper v3! As part of a unified model no less

* AI Art & Diffusion

* Meta - releases a standalong EMU AI art generator websites https://imagine.meta.com

* Tools

* Jetbrains finally releases their own AI native companion + subscription

That's it for me this week, this Waymo ride took extra long as it seems that in SF, during night rush hour, AI is at a disadvatage against human drivers. Maybe I'll take an Uber next time.

P.S - here’s Grok roasting ThursdAI

See you next week, and if you've scrolled all the way here for the emoji of the week, it's hidden in the middle of the article, send me that to let me know you read through 😉

Wow what a week. I think I’ve reached to a level that I’m not phased by incredible weeks or days that happen in AI, but I… guess I still have much to learn!

TL;DR of everything we covered (aka Show Notes)

* Open Source LLMs

* Mixtral MoE - 8X7B experts dropped with a magnet link again (Announcement, HF, Try it)

* Mistral 0.2 instruct (Announcement, HF)

* Upstage Solar 10B - Tops the HF leaderboards (Announcement)

* Together -Striped Hyena architecture and new models (Announcement)

* EAGLE - a new decoding method for LLMs (Announcement, Github)

* Deci.ai - new SOTA 7B model

* Phi 2.0 weights are available finally from Microsoft (HF)

* QuiP - LLM quantization & Compression (link)

* Big CO LLMs + APIs

* Gemini Pro access over API (Announcement, Thread)

* Uses character pricing not token

* Mistral releases API inference server - La Platforme (API docs)

* Together undercuts Mistral with serving Mixtral by 70% and announces OAI compatible API

* OpenAI is open sourcing again - Releasing Weak-2-strong generalization paper and github! (announcement)

* Vision

* Gemini Pro api has vision AND video capabilities (API docs)

* AI Art & Diffusion

* Stability announces Zero123 - Zero Shot image to 3d model (Thread)

* Imagen 2 from google (link)

* Tools & Other

* Optimus from Tesla is coming, and it looks incredible

This week started on Friday, as we saw one of the crazier single days in the history of OSS AI that I can remember, and I’ve been doing this now for .. jesus, 9 months!

In a single say, we saw a new Mistral model release called Mixtral, which is a Mixture of Experts (like GPT4 is rumored to be) of 8x7B Mistrals, and beats GPT3.5, we saw a completely new architecture that competes with Transformers called HYENA from Tri Dao and Together.xyz + 2 new models trained with that architecture, we saw a new SoTA 2-bit quantization method called QuiP from cornell AND a new 3x faster decoding method for showing tokens to users after an LLM has done “thinking”.

And the best thing? All those advancements are stackable! What a day!

Then I went to NeurIPS2023 (which is where I am right now, writing these words!), which I cover at length at the second part of the podcast, but figured I’d write about it here as well, since it was such a crazy experience.

NeurIPS is the biggest AIML conference, I think they estimated 15K people from all over the world attending! Of course this brings many companies to sponsor, raise booths, give out swag and try to record!

Of course with my new position at Weights & Biases I had to come as well and experience this for myself!

Many of the attendees are customers of ours, and I was not expecting this amount of love given, just an incredible stream of people coming up to the booth, and saying how much they love the product!

So I manned the booth, did interviews and live streams, and connected with a LOT of folks and I gotta say, this whole NeurIPS thing is quite incredible from the ability to meet people!

I hung out with folks from Google, Meta, Microsoft, Apple, Weighs & Biases, Stability, Mistral, HuggingFace and PHD students and candidates from most of the top universities in the world, from KAIST to MIT and Stanford, Oslo and Shaghai, it's really a worldwide endeavor!

I also got to meet many of the leading figures in AI, all of whom I had to come up to and say hi, shake their hand, introduce myself (and ThursdAI) and chat about what they or their team released and presents at the conference! Truly an unforgettable experience!

Of course, This Weeks’ Buzz is that, everyone here loves W&B, from the PHD students, to literally every big LLM lab! They all came up to us (yes yes, even researches at Google who kinda low-key hate their internal tooling) and told us how awesome the experience was! (besides Xai folks, Jimmy wasn’t that impressed haha) and of course I got to practice the pitch so many times, since I manned the W&B booth!

Please do listen to the above podcast, there’s so much detail that’s in there that doesn’t get up on the newsletter, as it’s impossible to cover all, but it was a really fun conversation, including my excited depiction of this weeks NOLA escapades!

I think I’ll end here, cause I can go on and on about the parties (There were literally 7 at the same time last night, Google, Stability, OpenAI, Runway, and I’m sure there were a few more I wasn’t invited to!) and about New Orleans food (it’s my first time here, I ate a soft shell deep fried crab and turtle soup!) and I still have the poster sessions to go to and workshops! I will report more on my X account and the Weights & Biases X account, so stay tuned for that there, and as always, thanks for tuning in, reading and sharing ThursdAI with your friends 🫡

P.S - Still can’t really believe I get to do this full time now and share this journey with all of you, bringing you all with me to SF, and now NeurIPS and tons of other places and events in the future!

— Alex Volkov, AI Evangelist @ Weights & Biases, Host of ThursdAI 🫡

Hey everyone, happy ThursdAI!

As always, here's a list of things we covered this week, including show notes and links, to prepare you for the holidays.

TL;DR of all topics covered:

* Open Source AI

* OpenChat-3.5-1210 - a top performing open source 7B model from OpenChat team beating GPT3.5 and Grok (link, HF, Demo)

* LAION 5B dataset taken down due to CSAM allegations from Stanford (link, full report pdf)

* FLASK - New evaluation framework from KAIST - based on skillset (link)

* Shows a larger difference between open/closed source

* Open leaderboard reliability issues, vibes benchmarks and more

* HF releases a bunch of MLX ready models (LLama, Phi, Mistral, Mixtral) (link)

* New transformer alternative architectures - Hyena & Mamba are heating up (link)

* Big CO LLMs + APIs

* Apple - LLM in a flash paper is making rounds (AK, Takeaways thread)

* Anthropic adheres to the messages API format (X)

* Microsoft Copilot finally has plugins (X)

* Voice & Audio

* AI Music generation Suno is now part of Microsoft Copilot plugins and creates long beautiful songs (link)

* AI Art & Diffusion

* Midjourney v6 is out - better text, great at following instructions (link)

Open Source AI

We start today with a topic I didn't expect to be covering, the LAION 5B dataset, was taken down, after a report from Stanford Internet Observatory found instances of CSAM (Child Sexual Abuse material) in the vast dataset. The outlined report had identified hundreds to thousands of instances of images of this sort, and used something called PhotoDNA by Microsoft to identify the images by hashes, using a sample of NSFW marked images.

LAION 5B was used to train Stable Diffusion, and 1.4 and 1.5 were trained on a lot of images from that dataset, however SD2 for example was only trained on images not marked as NSFW. The report is very thorough, going through the methodology to find and check those types of images. Worth noting that LAION 5B itself is not an image dataset, as it only contains links to images and their descriptions from alt tags.

Obviously this is a very touchy topic, given the way this dataset was scraped from the web, and given how many image models were trained on it, the report doesn't allege anything close to influence on the models it was trained on, and outlines a few methods of preventing issues like this in the future. One unfortunate outcome of such a discovery, is that this type of work can only be done on open datasets like LAION 5B, while closed source datasets don't get nearly to this level of scrutiny, and this can slow down the advancement of multi-modal open source multi modal models while closed source will continue having these issues and still prevail.

The report alleges they found and validated between hundreds to a few thousand of CSAM verified imagery, which considering the size of the dataset, is infinitesimally small, however, it still shouldn't exist at all and better techniques to clean those scraping datasets should exist. The dataset was taken down for now from HuggingFace and other places.

New version of a 7B model that beats chatGPT from OpenChat collective (link, HF, Demo)

Friend of the pod Alpay Aryak and team released an update to one of the best 7B models, namely OpenChat 7B (1210) is a new version of one of the top models in the 7B world called OpenChat with a significant score compared to chatGPT 3.5 and Grok and with very high benchmark hits (63.4% on HumanEval compared to GPT3.5 64%)

Scrutiny of open source benchmarks and leaderboards being gamed

We've covered State of the art models on ThursdAI, and every time we did, we covered the benchmarks, and evaluation scores, Whether that's the popular MMLU (Multi-Task Language Understanding) or HumanEval (Python coding questions) and almost always, we've referred to the HuggingFace Open LLM leaderboard for the latest and greatest models. This week, there's a long thread on the hugging face forums that HF eventually had to shut down, that alleges that a new contender for the top, without revealing methods, used something called UNA to beat the benchmarks, and folks are suggesting that it must be a gaming of the system, as a model that's trained on the benchmarks can easily top the charts.

This adds to the recent observations from friend of the pod Bo Wang from Jina AI, that the BGE folks have stopped focusing on the MTEB leaderboard (Massive Text Embedding Benchmark) benchmarks as well, as those are also seem to be gamed (link)

This kicked off a storm of a discussion about different benchmarks and evaluations, ability to score and check wether or not we're advancing, and the openness of these benchmarks. Including one Andrej Karpathy that chimed in that the only way to know is to read the r/LocalLlama comment section (e.g. vibes based eval) and check the ELO score on the LMSys chatbot arena, which pits 2 random LLMs behind the scenes and lets users choose the best answer/score.

LMsys also has a leaderboard, and that one only includes models they have explicitly added to their Arena, and also merges 3 different scores, the ELO score by human raters, the MTBench score and the MMLU.

This is the latest benchmark, showing that Mixtral is the highest ranking open source model at this point, and that a few other Apache 2.0 models like OpenChat (the previous version, the one from today should score even higher) and OpenHermes are inching closer as well and have honorable mentions given their license and size!

However, given the latest releases in HuggingFace lineage, where you could track the model finetunes to what models they were fine-tuned on, it's still a good place to check out those leaderboards, just... self evaluation and running models on your own tasks is always a good idea! Also a good idea is additional benchmarks, like the one proposed by KAIST this week called FLASK that shows quite a significant distance between closed source models and open source ones based on several skills.

This weeks Buzz (What I learned this week in Weights & Biases)

This week we kicked off a buildweek internally, which unfortunately I wasn’t able to be a super active participant in, due to lying on my couch with a fever for most of the week, but regardless, I noticed how important is it to have these build weeks/hack weeks from time to time to actually use some of the new techniques we often talk about, like chain-of-density prompting techniques, or agent fine-tunes. I also got paired with my colleague Anish on our project, and while we work on our project (to be revealed later) he gave a kick ass webinar on the famous deeplearning.ai platform on the topic of enhancing performance for LLM agents in automation that more than 5K folks tuned into! Anish is a wealth of knowledge, so check it out if this topic interests you 👏

Big CO LLMs + APIs

Apple - LLM in a Flash + MLX stuff

Apple has been more and more in the AI news lately, having recently released MLX framework for running models directly on apple silicon devices, without a lot of dependencies, which was always possible, but is not optimized. This got many folks to start converting models to an MLX compatible format and there's no even a new tag on HF for those converted models

But the main news this week don't stop there, folks from Apple also released the LLM in a flash paper, which shows advances in running LLMs in hardware restricted environments like smartphones, where memory is limited, and shows interesting promise, and also a glimpse that Apple is likely moving towards on device or partially on device inference at some point if we combine the MLX stuff and this paper attempts.

Anthropic moves towards messages API

Anthropic Claude finally gives us some DX and introduces a similar to OpenAI messages API.

Voice

Microsoft copilot now has plugins and can create songs!

Microsoft copilot (FKA Bing Chat) now has Plugins (probably not new from this week, but we haven't yet reported on this) and one of the coolest ones is SUNO, which is an audio generation platform that has been around. And now it's super easy to create whole songs, directly from the Microsoft Copilot interface!

Here’s my 1 shot attempt and creating a holiday jingle for ThursdAI, it’s not good, but it’s fun 😂

And I’ve seen some quite decent examples like return to monkey

AI Art & Diffusion

Midjourney v6 looks stunning and follows prompts very well

Midjourney finally dropped their version 6, and it looks, really really good. Notably, it's likely the highest quality / fidelity diffusion model out there that we can use, has better support for text, and follows prompts closely. DALL-E is still very impressive for folks given that the iteration via chatGPT interface is very easy and convinient, but still ,just look at some of these MJv6 generations 😻

Nick gave it a very details prompt with 8 specific color assingments and besides the image looking insane, MJ nailed the super complex prompt!

35mm film still, two-shot of a 50 year old black man with a grey beard wearing a brown jacket and red scarf standing next to a 20 year old white woman wearing a navy blue and cream houndstooth coat and black knit beanie. They are walking down the middle of the street at midnight, illuminated by the soft orange glow of the street lights --ar 7:5 --style raw --v 6.0

And just for fun, here’s a comparison of all previous versions of MJ for the same prompt, just to… feel the progress 🔥

Thanks for reading all the way through, I think I got more than I bargained for during NeurIPS and I came back with a fever and was debating wether to even record/send this weeks newletter, but now that I’m at the end of it I’m happy that I did! Though, if you listen to the full recording, you may hear me struggling to breathe a bit 😅

So I’ll go rest up before the holidays, wishing you merry Christmas if you celebrate it 🎄 See you next week 🫡

Hey hey hey (no longer ho ho ho 🎄) hope you had a great Christmas! And you know that many AI folks have dropped tons of OpenSource AI goodies for Christmas, here’s quite a list of new things, including at least 3 new multi-modal models, a dataset and a paper/technical report from the current top model on HF leaderboard from Upstage.

We also had the pleasure to interview the folks who released the Robin suite of multi-modals and aligning them to “good responses” and that full interview is coming to ThursdAI soon so stay tuned.

And we had a full 40 minutes with an open stage to get predictions for 2024 in the world of AI, which we fully intent to cover next year, so scroll all the way down to see ours, and reply/comment with yours!

TL;DR of all topics covered:

* Open Source LLMs

* Uform - tiny(1B) multimodal embeddings and models that can run on device (HF, Blog, Github, Demo)

* Notux 8x7B - one of the first Mixtral DPO fine-tunes - (Thread, Demo)

* Upstage SOLAR 10.7B technical report (arXiv, X discussion, followup)

* Capybara dataset open sourced by LDJ (Thread, HF)

* Nous Hermes 34B (finetunes Yi34B) - (Thread, HF)

* Open Source long context pressure test analysis (Reddit)

* Robin - a suite of multi-modal (Vision-Language) models - (Thread, Blogpost, HF)

* Big CO LLMs + APIs

* Apple open sources ML-Ferret multi-modal model with referring and grounding capabilities (Github, Weights, Paper)

* OpenAI & Microsoft are getting sued by NewYorkTimes for copyright infringement during training (Full Suit)

* AI Art & Diffusion & 3D

* Midjourney v6 alpha is really good at recreating scenes from movies (thread)

Open Source LLMs

Open source doesn't stop even during the holiday break! Maybe this is the time to catch up to the big companies? During the holiday periods?

This week we got a new 34B Nous Hermes model, the first DPO fine-tune of Mixtral, Capybara dataset but by far the biggest news of this week was in Multimodality. Apple quietly open sourced ml-ferret, an any to any model able to compete in grounding with even GPT4-V sometimes, Uform released tiny mutli-modal and embeddings versions for on device inference, and AGI collective gave NousHermes 2.5 eyes 👀

There's no doubt that 24' is going to be the year of multimodality, and this week we saw an early start of that right on ThursdAI.

Ml-Ferret from Apple (Github, Weights, Paper)

Apple has been in the open source news lately, as we've covered their MLX release previously and the LLM in a flash paper that discusses inference for low hardware devices, and Apple folks had 1 more gift to give. Ml-Ferret is a multimodal grounding model, based on Vicuna (for some... reason?) which is able to get referrals from images (this highlighted or annotated areas) and then ground the responses with exact coordinates and boxes.

The interesting thing about the referring, is that it can be any shape, bounding box or even irregular shape (like the ferred in the above example or cat tail below)

Ferret was trained on a large new dataset called GRIT containing over 1 million examples of referring to and describing image regions (which wasn't open sourced AFAIK yet)

According to Ariel Lee (our panelist) these weights are only delta weights and need to be combined with Vicuna weights to be able to run the full Ferret model properly.

Uform - tiny (1.5B) MLLMs + vision embeddings (HF, Blog, Github, Demo)

The folks at Unum have released a few gifts for us, with an apache 2.0 license 👏 Specifically they released 3 vision embeddings models, and 2 generative models.

Per the documentation the embeddings can yield 2,3x speedup improvements to search from Clip like models, and 2-4x inference speed improvements given the tiny size. The embeddings have a multi-lingual version as well supporting well over 20 languages.

The generative models can be used for image captioning, and since they are tiny, they are focused on running on device, and are already converted to ONNX format and core-ML format. Seen the results below compared to LLaVa and InstructBLIP, both at the 7B range

I've tried a few images of my own (you can try the demo here), and while there was hallucinations, this tiny model did a surprising amount of understanding for the size!

Also shoutout to Ash

Robin suite of multimodal models (Thread, Blogpost, HF)

The folks at the CERC-AAI lab in MILA-quebec have released a suite of multi-modal models, that they have finetuned and released a fork of NousHermes2.5 that can understand images, building on top of CLIP, and SigLIP as the image encoder.

In fact, we did a full interview with Irina, Kshitij, Alexis and George from the AGI collective, that full interview will be released on ThursdAI soon, so stay tuned, as they had a LOT of knowledge to share, from fine-tuning the clip model itself for better results, to evaluation of multimodal models, to dataset curation/evaluation issues and tips from Irina on how to get a government supercomputer compute grant 😈

Big CO LLMs + APIs

OpenAI is being used by NYT for copyright infringement during training (Lawsuit)

New York times is suing OpenAI and Microsoft for copyright infringement, seeking damages (amount unclear) and removal of NYT data from OpenAI models. The full lawsuit is a worthwhile read, and in includes a whopping 100 pages of examples of GPT4 completing NYT articles verbatim. I personally wasn't able to reproduce this behavior in the chatGPT app, but some folks on X suggested that it's possible in the OpenAI playground, with the right prompt and NYT URL in the prompt.

This lawsuit came after a round of attempted negotiations between NYT and OpenAI, which apparently failed, and it's worth noting a few things. First, OpenAI (with almost every other AI company) have a "Copyright shield" feature, where they protect the user of these services from getting sued for copyright violations. So there is no direct exposure for customers of OpenAI. Additional thing of note is, the NYT information is compiled not by OpenAI directly, rather, OpenAI (and almost every other LLM) have used the CommonCrawl dataset (among others) which did the crawling and collection of text itself.

Per the CommonCrawl license, OpenAI should have reached out to each individual URL in that dataset and worked out the copyright on their own, which is a bit difficult to do, as CommonCrawl includes 3-5 billion pages collected each month.

Regardless of the claims, the hottest takes I saw in regards to this are, that by the time anything moves with this lawsuit, we will be on GPT-6 or so and it won't matter by then, or that OpenAI will have to retrain a model without NYT data, which I find quite ludicrous personally and very unlikely to happen.

If this lawsuit actually sets a precedent, this will IMO be a very bad one for the US, considering other countries like Japan are already getting ahead of this, declaring all scraped data as fair us if used for training (source)

Of course, all of X became IP experts overnight, and the debates are very interesting, some are confusing technical terms, some are claiming that OpenAI will just cave and pay NYT, while some super libertarian ones take it all the way down to: if AI has human rights, and if it does, then preventing it learning from copyright material is like preventing people to read Hemingway.

This weeks buzz (What I learned in WandB this week)

This week, we sent out our annual emails of wrapped cards for everyone who used Weights & Biases to train models this year. This is a yearly tradition, similar to Spotify, however, for ML purposes, and this year the cards were generated with stable diffusion XL, generating hundreds of thousands of images based on autogenerated model run names!

The interesting thing I noticed also, is just how many folks shared their stats screenshots right from the email we send, including not only how many hours they spend training models this year, but also how many other features they used, like reports and sweeps. And I noticed just how many folks don't use reports, which is a shame, as it's such a cool feature! WandB literally has a built in blogging platform for all your ML needs and it includes live widgets of every metric you're tracking in your runs, it's really great.

AI Art & Diffusion

Midjourney v6 is incredible at recreating actual movie stills and scenes (Thread)

Another potential lawsuit is waiting to happen? We already saw lawsuits against StabilityAI for supposed copyright infringement and stability did a lot of work to exclude proprietary art from their training datasets, however, the new incredible version of Midjourney, shows just.. a mind-blowing accuracy in recreating scenes from movies, and cartoon styles. Just look at some of these examples (collected by some folks on X)

This + the above lawsuit news coming for OpenAI & Microsoft from New York Times is setting up 24' to be the year where copyright law and AI finally meet for real. And we'll keep reporting on the outcomes.

Predicitons for 24'

In the last 20 minutes of the pod recording we opened up the floor to folks giving us their predictions for AI developments in the year 2024, and I also asked this question on X itself. The idea was, to come back next year during our yearly summary and see which predictions we hit, and which predictions we were not even remotely thinking about!

Here's a list of predictions with their category (Thanks to AI to help me sort these from different sources and transcription)

* Open Source LMs

* 1GB models with Mix Trail performance levels - Nisten

* Continual pretraining and building on top of each other's work - Irina Rish

* Smaller models trained on more data - Irina Rish

* Consolidation and standardization of models - Irina Rish

* Agents running on 7B models with capabilities like web search and code interpretation - Shroominic

* End of dominance of transformer architecture - Far El

* Marriage of reinforcement learning and language models - Far El

* New benchmarking standards - Far El

* Plug and play weights for expertise - Umesh

* Self-improving pipeline framework - Umesh

* Big Companies/APIs

* Mistral to become a major player, surpassing companies like Anthropic - Alex Volkov

* Apple AI device with multimodal capabilities - Umesh

* Google Gemini Pro commoditizing APIs - Umesh

* Model that can ace undergrad computer science curriculum - George Adams

* Extremely good but expensive model (~$1 per response) - Shroominic

* Apple spatial computing + AI product innovation - John Doe

* Real-time multilingual translation app/device - Umesh

* Vision/Video

* AI-generated full length feature film - Umesh

* Artist AI model galleries for art generation - Umesh

* Real-time video understanding and multimodal models - Alex Volkov

* Public release of high quality, fast voice cloning tech - Alex Volkov

* 3D model/animation generation for video games - tobi

* Meta will outperform most companies in video AI and mixed reality - Alex Volkov

* Other

* Localized national AI models - Ravi

* Rise in use of deepfakes - Ravi

* Surge in metadata embedding for ownership identification - R.AI.S.E

* Advances in AI for biology/healthcare - Ravi, Ash Vardanian

* A model capable of completing an undergrad CS curriculum at an A level by the end of the year - George Adams

* AI device, fully capable of multimodal capabilities, from Apple - Educated Guess

* Development in domain-specific LMs for bio applications, especially in synthetic biology - Ravi

* Twitter Prediction

* CodeInterpreterAPI V2 - Shroominic

* Gemini will NOT outperform ChatGPT - Alex Northstar

* Tech slowdown in mass adoption, human creativity as bottleneck - “charles harben”

* Biology and Robots - Sinan

* Code LLMs near junior developer productivity - Karthik Kannan

* Tokenizers will work - Geronimo

* LLM curve plateaus, focus on refining and multimodal, OpenAI settles with NYT - hokiepoke

* Fully generated, rigged, voiced game characters, minimal human intervention - Rudzinski Maciej

* AI affects politics - 𝕄𝕏𝕊ℍℝ🤖

* Audio reaches DallE3 level, video and 3D advancements, new cool modality - Darth thromBOOzyt

* Synthetic data will be huge - Leo Tronchon

Ok now that our predictions are here, we'll come back here next year and see who hit what predictions!

If you have predicitons if your own, please reply to this email/substack and post them here as well, so we'll have a record 🫡

With that, I want to wish you a happy new year, and as always, see you here next week 👏

Here’s a TL;DR and show notes links

* Open Source LLMs

* New WizardCoder 33B V1.1 - 79% on HumanEval (X, HF)

* Tekniums Hermes 2 on SOLAR 10.7B (X, HF)

* Microsoft - E5 SOTA text embeddings w/ Mistral (X, HF, Paper, Yams Thread)

* Big CO LLMs + APIs

* Samsung is about to announce some AI stuff

* OpenAI GPT store to come next week

* Perplexity announces a $73.6 Series B round

* Vision

* Alibaba - QWEN-VL PLUS was updated to 14B (X, Demo)

* OCU SeeAct - GPT4V as a generalist web agent if grounded (X, Paper)

* Voice & Audio

* Nvidia + Suno release NeMo Parakeet beats Whisper on english ASR (X, HF, DEMO)

* Tools & Agents

* Stanford - Mobile ALOHA bot - Open source cooking robot (Website, X thread)

Open Source LLMs

WizardCoder 33B reaches a whopping 79% on HumanEval @pass1

State of the art LLM coding in open source is here. A whopping 79% on HumanEval, with Wizard Finetuning DeepSeek Coder to get to the best Open Source coder, edging closer to GPT4 and passing GeminiPro and GPT3.5 👏 (at least on some benchmarks)

Teknium releases a Hermes on top of SOLAR 10.7B

Downloading now with LMStudio and have been running it, it's very capable. Right now SOLAR models are still on top of the hugging face leaderboard, and Hermes 2 now has 7B (Mistral) 10.7B (SOLAR) and 33B (Yi) sizes.

On the podcast I've told a story of how this week I actually used the 33B version of Capybara for a task that GPT kept refusing to help me with. It was honestly kind of strange, a simple request to translate kept failing with an ominous “network error”.

Which only highlighted how important the local AI movement is, and now I actually have had an experience myself of a local model coming through when a hosted capable one didn’t

Microsoft releases a new text embeddings SOTA model E5 , finetuned on synthetic data on top of Mistral 7B

We present a new, easy way to create high-quality text embeddings. Our method uses synthetic data and requires less than 1,000 training steps, without the need for complex training stages or large, manually collected datasets. By using advanced language models to generate synthetic data in almost 100 languages, we train open-source models with a standard technique. Our experiments show that our method performs well on tough benchmarks using only synthetic data, and it achieves even better results when we mix synthetic and real data.

We had the great please of having Bo Wang again (One of the authors of the Previously SOTA Jina embeddings and a previous podcast gust) to do a deepdive into embeddings and specifically E5 with it's decoder only architecture. While the approach Microsoft researchers took here are interesting, and despite E5 claiming a top spot on the MTEB leaderboard (pictured above) this model doesn't seem to be super practical for most purposes folks use embeddings right now (RAG) for the following reasons:

* Context length limitation of 32k, with a recommendation not to exceed 4096 tokens.

* Requires a one-sentence instruction for queries, adding complexity for certain use cases like RAG.

* Model size is large (14GB), leading to higher costs for production use.

* Alternative models like bge-large-en-v1.5 are significantly smaller (1.35GB).

* Embedding size is 4096 dimensions, increasing the cost for vector storage.

Big CO LLMs + APIs

OpenAI announces that the GPT store is coming next week!

I can't wait to put the visual weather GPT I created and see how the store prompts it and if I get some revenue share like OpenAI promised during dev day. My daughter and I are frequent users of Alice - the kid painter as well, a custom GPT that my Daughter named Alice, that knows it's speaking to kids over voice, and is generating coloring pages. Will see how much this store lives up to the promises.

This weeks Buzz (What I learned with WandB this week)

This week was a short one for me, so not a LOT of learnings but I did start this course from W&B, called Training and Fine-tuning Large Language Models (LLMs).

It features great speakers like Mark Sarufim from Meta, Jonathan Frankle from Mosaic, and Wei Wei Yang from Microsoft along with W&B MLEs (and my team mates) Darek Kleczek and Ayush Thakur and covers the end to end of training and fine-tuning LLMs!

The course is available HERE and it's around 4 hours, and well well worth your time if you want to get a little more knowledge about the type of stuff we report on ThursdAI.

Vision

SeeAct - GPT4V as a generalist web agent if grounded (X, Paper)

In June OSU NLP released Mind2Web which is a dataset for developing and evaluating web acting agents, LLMs that click buttons and perform tasks with 2350 tasks from over 130 website, stuff like booking flights, finding folks on twitter, find movies on Netflix etc'

GPT4 without vision was terrible at this (just by reading the website html/text) and succeeded at around 2%.

With new vision LMMs, websites are a perfect place to start training because of the visual (how website is rendered) is no paired with HTML (the grounding) and SeeAct uses GPT4-V to do this

SeeAct is a generalist web agent built on LMMs like GPT-4V. Specifically, given a task on any website (e.g., “Compare iPhone 15 Pro Max with iPhone 13 Pro Max” on the Apple homepage), the agent first performs action generation to produce a textual description of the action at each step towards completing the task (e.g., “Navigate to the iPhone category”), and then performs action grounding to identify the corresponding HTML element (e.g., “[button] iPhone”) and operation (e.g., CLICK, TYPE, or SELECT) on the webpage.

SeeAct achieves a 50% score on the Mind2Web evaluation task!

QWEN-VL was updated to PLUS (14B) and it's pretty good compared to GPT4V

Capabilities include: image captioning, visual question answering, visual grounding, OCR, visual reasoning. We had a chat with Junyang Lin, the tech lead for Qwen with Alibaba on the pod, and he mentioned specifically that they noticed that adding a larger "brain" (as in, LLM) to vision models, significantly increases the performance and vision understanding of the LMMs.

While this model is not yet released, you can demo it here, and Junyang told us that it is coming to a release, like the previous QWEN models did before.

I noticed the advanced OCR capabilities and understanding, this example was really spot on. Notice the "logo for Browser company" , the model understood that this text was in fact a logotype! (which even GPT4V failed at in my test)

Voice

Parakeet from NVIDIA beats Whisper on English with a tiny model (blog)

Brought to you by @NVIDIAAI and @suno_ai_, parakeet beats Whisper and regains its first place. The models are released under a commercially permissive license! The models inherit the same FastConformer architecture and come in 2 flavors: 1. RNNT (1.1B & 0.6B) 2. CTC (1.1B & 0.5B) Each model is trained on 65K hours of English data (40K private proprietary data by Suno & NeMo teams) over several hundred epochs. Key features of the parakeet model: 1. It doesn't hallucinate (if the audio sample has silence, the output is silent). 2. It is quite robust to noisy audio (if the audio sample has non-vocal sounds, it outputs silence).

We had the great please to have VB from the Audio team at HuggingFace, and he went in depth into the way in which Parakeet is better than Whisper (higher quality transcriptions while also being much much faster), it was trained on only 65K hours vs a few million with whisper, and we also covered that because of this different architecture, Parakeet is not able to receive any guidance for words that are hard for it to understand. For example, with whisper, I often provide ThursdAI in initial_prompt parameter to help guide whisper to know what it should say.

Regardless, having a model that's superfast, and can beat whisper, and is commercially licensed to build on top of is incredible! Here's a demo for you to try it out and it's available with the NVIDIA NeMO framework.

Coqui shuts down :(

We've had Josh from Coqui on our pod before, when they released XTTS, and they have been friends ever since. It's sad to see Coqui shut down, and we want to wish all the team an easy and great transition 👏 You guys did a great job and we're rooting for each and every one of you.

* Coqui is closing down.

* The team is praised for being small yet impactful, competing with big tech despite limited resources.

* Coqui began as the Machine Learning Group at Mozilla, creating DeepSpeech, Common Voice, and TTS.

* Spun out as Coqui in 2021 to accelerate their mission.

* Major achievement: XTTS, with openly released model weights for versions 1 and 2.

* 2021: Coqui STT v1.0 released, Coqui Model Zoo and SC-GlowTTS launched.

* 2022: YourTTS became viral, numerous open-source releases, team expansion.

* 2023: Coqui Studio webapp and API launched, XTTS open release, first customers acquired.

* Acknowledgment of the community, investors, customers, and partners for their support.

* Partners include HuggingFace, Mozilla, Masakhane, Harvard, Indiana University, Google, MLCommons, Landing AI, NVIDIA, Intel, and Makerere University.

* Future of generative AI in 2024 predicted to grow, with open-source playing a significant role.

* Coqui TTS remains available on Github for further innovation.

Tools

Stanford Mobile ALOHA bot open sources, shows cooking

Back in March, Stanford folks introduced ALOHA, (A Lowcost Open Hardware system for Bimanual Teleoperation)

Basically a 4 arm robot, that a human operator can operate tasks and do fine motor skills like break an egg or tie ziptie. Well now, just 10 months later, they are introducing the Mobile version. A mounted ALOHA gear, that uses the human to perform tasks like cooking, calling the elevator and is able to learn from those actions, and then perform them.The operating gear can be easily detached for self operation, it's mobile so compute and battery pack are on the wheel base.

Recently Meta released a huge dataset of first person operations called Ego-Exo 4D which combines first person and third person perspective for a big variety of tasks, such as cooking, cleaning, sports, healthcare and rock climbing, and this open hardware from Stanford is an additional example of how fast robotics advances into the physical world

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

And just like that, the first ThursdAI of the year is done! 🫡 Thank you for being a subscriber, see you next week 👏

Hey hey everyone, how are you this fine ThursdAI? 👋 I’m gud thanks for asking!

I’m continuing my experiment of spilling the beans, and telling you about everything we talked about in advance, both on the pod and in the newsletter, so let me know if this is the right way to go or not, for the busy ones it seems that it is. If you don’t have an hour 15, here’s a short video recap of everything we chatted about:

ThursdAI - Jan 11 2024 TL;DR

TL;DR of all topics covered + Show notes

* Open Source LLMs

* 🔥 Donut from Jon Durbin is now top of the LLM leaderboard (X, HF, Wolframs deep dive and scoring)

* OpenChat January Update - Best open source 7B LLM (X, Hugging Face)

* Our friends at NousResearch announce a seed round of 5.2M as their models pass 1.2 million downloads (X)

* Argilla improved (Distillabeled?) the DPO enhanced Neural Hermes with higher quality DPO pairs (X)

* New MoEs are coming out like hotcakes - PhixTral and DeepSeek MoE (X, Omar Thread, Phixtral Thread)

* Microsoft makes Phi MIT licensed 👏

* Big CO LLMs + APIs

* OpenAI adds personalization & team tiers (Teams announcement)

* OpenAI launches GPT store (Store announcement, Store link)

* Mixtral medium tops the LMsys human evaluation arena, is the best LLM overall after GPT4 👏 (X)

* Hardware

* Rabbit R1 is announced, $200/mo without a subscription, everybody has a take (X)

* This weeks Buzz from Weights & Biases

* Hackathon with Together, Langchain and WandB (and ME!) this weekend in AGI house (X, Signup)

* Video

* Bytedance releases MagicVideo-V2 video gen that looks great and passes Pika labs in human tests (X)

* AI Art & Diffusion & 3D

* Luma launched their online version of Genie and it's coming to the API (X)

* Show notes and links mentioned

* MergeKit (github)

* Jon Durbins Contextual DPO dataset (HuggingFace)

* Phixtral from Maxime Lebonne (X, HuggingFace)

* WandGPT - out custom Weights & Biases GPT (GPT store)

* Visual Weather GPT by me - https://chatg.pt/artweather

* Ask OpenAI to not train on your chats - https://privacy.openai.com/policies

AI Hardware

It seems that the X conversation had a new thing this week, the AI hardware startup Rabbit, showcased their new $200 device (no subscriptions!) at CES and everyone and their mom had an opinion! We had quite a long conversation about that with (his first time on ThursdAI 👏) as we both pre-ordered one, however there were quite a few red flags, like for example, GPUs are costly, so how would an AI device that has AI in the cloud just cost a 1 time 200 bucks??

There were other interesting things they showed during the demo, and I’ll let you watch the full 30 minutes and if you want to read more, here’s a great deeper dive into this from .

UPDATE: Ss I’m writing this, the CEO of Rabbit (who’s also on the board of Teenage Engineering, the amazing company that designed this device) tweeted that they sold out the initial first AND second batch of 10K unites, netting a nice $2M in hardware sales in 48 hours!

Open Source LLMs

Mixtral paper dropped (ArXiv, Morgans take)

Mistral finally published the paper on Mixtral of experts, the MoE that's the absolutel best open source model right now, and it's quite the paper. Nisten did a full paper reading with explanations on X space, which I co-hosted and we had almost 3K people tune in to listen. Here's the link to the live reading X space by Nisten.

And here's some notes courtecy Morgan McGuire (who's my boss at WandB btw 🙌)

Strong retrieval across the entire context window

Mixtral achieves a 100% retrieval accuracy regardless of the context length or the position of passkey in the sequence.

Experts don't seem to activate based on topic

Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic. For instance, at all layers, the distribution of expert assignment is very similar for ArXiv papers (written in Latex), for biology (PubMed Abstracts), and for Philosophy (PhilPapers) documents.

However...

The selection of experts appears to be more aligned with the syntax rather than the domain

Datasets - No info was provided to which datasets Mixtral used to pretrain their incredible models 😭

Upsampled multilingual data

Compared to Mistral 7B, we significantly upsample the proportion of multilingual data during pretraining. The extra capacity allows Mixtral to perform well on multilingual benchmarks while maintaining a high accuracy in English

Mixtral Instruct Training

We train Mixtral – Instruct using supervised fine-tuning (SFT) on an instruction dataset followed by Direct Preference Optimization (DPO) on a paired feedback dataset and was trained on @CoreWeave

Jon Durbin Donut is the 🤴 of open source this week

6 of the top 10 are donut based models or merges of it. If you remember Auroborous, Donut includes that dataset, and there are two varieties there, the DPO and the non DPO versions of Bagel, including two merges from Cloudyu, which are non trained merges with mergekit, based on Donut. Jon pro tip for selecting DPO vs Non DPO models is

FYI, the DPO version is more factual, truthful, better at math, etc., but is not great for RP, creative writing, etc. Use non-DPO for those tasks!

Donut includes an impressive amount of dataset mixed together, which are all linked from the model card but here they are:

"ai2_arc, airoboros, apps, belebele, bluemoon, boolq, capybara, cinematika, drop, emobank, gutenberg, lmsys_chat_1m, mathinstruct, mmlu, natural_instructions, openbookqa, pippa, piqa, python_alpaca, rosetta_code, slimorca, spider, squad_v2, synthia, winogrande, airoboros 3.1 vs airoboros 2.2.1, helpsteer, orca_dpo_pairs"

Jon also shared his end of the year WandB report nad has trained a whopping 917 models this year for a total of ~2500 hours and is in the top 10% of the top active users (among 800K or so users)

I didn't know that Jon is going to join, but was so happy that he joined the live recording that we ended up chatting for 20 minutes, and there was so many nuggets in that conversation, about how to prepare DPO datasets, which other ones Jon has been releasing, and just a bunch more gold, that I decided to CUT that out and post it as a separate special deepdive episode that's going to get released on the Sunday special. Stay tuned for that!

Nous Research announces $5.2 million funding seed round as they cross 1.1 million model downloads on the hub

Congrats to Karan, Emozilla, Teknium, Bowen, Shivani and the rest of the Nous team on this great news! 👏 We expect to hear more from them in the coming year, with a consistent commitment to open source, keep open sourcing the best models, and the upcoming Forge news!

With investors like Balaji, OSS capital, Vipul from Together, Nous completes the $5.2M seed round, and we had Karan (one of the co-founders of Nous) on the pod to chat to use about what they are planning to do with that money and what are their continuous commitments to open source!

In addition, they just recently passed 1.1 million downloads on the hub with Nous-Hermes-2-34B being their best model! 🤴

OpenChat Jan update becomes the leading open source 7B model (X, Hugging Face)

This update mainly enhanced training methodology, in-context learning & coding skills, outperforming the last 1210 release on 7 out of 8 benchmarks! and scores 71.3 on HumanEval, 65.8% on MMLU 👏

The previous version of OpenChat trails just behind OpenHermes on the human evals on Lmsys arena, but both are incredible 7B models.

Argilla

- Argilla used their Distilabel tool to build a preference dataset from ratings and critiques of AI response pairs, taking around 3 hours 

- The original dataset assumed the GPT-4/3.5 responses were always best, but Argilla found this was not always the case

- Their dataset confirmed ~4,000 pairs had the same rating, 7,000 pairs were unchanged, and ~2,000 times the rejected response was preferred  

- Improving existing DPO datasets with higher quality pairs is important for model fine-tuning

- They are releasing an improved version of the popular Orca Pairs DPO dataset from Intel, and a new OpenHermes model outperforming baselines with 54% fewer DPO pairs

Big CO LLMs + APIs

OpenAI has a big week, launches GPTs store and team pro accounts (Blog)

Things of note about the store:

* My GPTs are getting feedback and crossed 10K chats , was #6 on lifestyle and the disappeared, but has gained 2x more chats in 24 hours since the store has launched!

* Discoverability is great, trending GPTs are shown clearly, and folks are getting a lot of exposure

* Copycats already started copying a bunch of the great GPTs, see this example of what happens when you search for Gymstreak, most of the top GPTs are already being copy-catted.

Team accounts:

$25/mo per user for annual plans and at least 2 teams

The biggest confusion was from folks who didn't understand that OpenAI trains on Pro conversations, and there's an option to Opt-out!

This weeks Buzz (What I learned with WandB this week)

Weights and Biases (and ME!) are going to AGI house to lead a Rag vs Finetune hackathon with cool prizes!

There's still time to RSVP, will incredible guests speakers, this Hackathon is organized together with... LangChain, TogetherCompute and AGI house - If you're in the SF area, and you wanna hack on some cool RAG things and get awesome prizes (and meet me!) join the waitlist here https://partiful.com/e/AlntdLtxh9Jh1J6Pcsma

Vision & Video

Luma released GENIE on Web and IOS, if you remember, we covered the GENIE text-to-3d model they first released on discord a while ago, and now it's incorporated into the luma website, and is significantly higher quality 3D assets.

The generations are free for now, and they look awesome! Here are some of mine, I created a Bee holding a Wand (get it? WandB? 😆) and a polish bear (internal joke) and they look so cool!

Friend of the pod and recent LUMA hire Arthur Islamov jumped on and also told us that this is coming to the API, so developers would be able to automate asset creation and generate tons of 3D objects programmatically, and use cool prompt techniques to make sure they are a bit better every time maybe? Great news!

AI Art & Diffusion

Bytedance announces MagicVideo-V2 (Arxiv, Project)

We didn't get anything besides quite a few cherry picked videos and a paper, so we can't use this yet, but wow some of these videos look incredible!

MagicVideo-V2 that integrates the text-to-image model, video motion generator, reference image embedding module and frame interpolation module into an end-to-end video generation pipeline. Benefiting from these architecture designs, MagicVideo-V2 can generate an aesthetically pleasing, high-resolution video with remarkable fidelity and smoothness. It demonstrates superior performance over leading Text-to-Video systems such as Runway, Pika 1.0, Morph, Moon Valley and Stable Video Diffusion model via user evaluation at large scale

Lastly, I had the greatest time to interview my new friend João Moura, the creator of Crew AI, which been popping off, was the #1 trending on Github and #2 of the day on Product hunt, and is essentially an AI framework that lets you create a crew of AI agents to do tasks for you. I will be polishing up that conversation and post it together with the deep dive with Jon, so stay tuned, but here’s a sneak preview of how cool this is and expect that episode to drop soon!

ThursdAI - Sunday special deep dive, interviews with Joao, and Jon, AI agent Crews and Bagel Merges.

Happy Sunday dear reader,

As you know by now, ThursdAI pod is not a standard interview based podcast, we don't focus on a 1:1 guest/host conversation, but from time to time we do! And this week I was very lucky to have one invited guest and one surprise guest, and I'm very happy to bring you both these conversations today.

Get your Crew together - interview with João Moura, creator of CrewAI

We'll first hear from João Moura, the creator of Crew AI, the latest agent framework. João is a director of AI eng. at Clearbit (acquired by Hubspot recently) and created Crew AI for himself, to automate many of the things he didn't want to keep doing, for example, post more on Linkedin.

Crew has been getting a lot of engagement lately, and we go into the conversation about it with João, it's been trending #1 on Github, and received #2 product of the day when Chris Messina hunted this (to João's complete surprise) on Product Hunt.

CrewAI is built on top of Langchain, and is an agent framework, focusing on Orchestration or role-playing, autonomous agents.

In our chat with João we go into the inspiration, the technical challenges and the success of CrewAI so far, how maintenance for crew is now partly a family effort and what's next for crew

Merges and Bagels - chat with Jon Durbin about Bagel, DPO and merging

The second part of today's pod was a conversation with Jon Durbin, a self described AI tinkerer and software engineer. Jon is a Sr. applied AI researcher at Convai, and is well known in our AI circles as a master finetuner and dataset curator.

This interview was not scheduled, but I'm very happy it happened! If you've been following along with the AI / Finetuning space, Jon's Airoboros dataset and set of models have been often mentioned, and cited, and Jon's latest work on the Bagel models took the lead on HuggingFace open LLM leaderboard

So when I mentioned on X (as I often do) that I'm going to mention this on ThursdAI, Jon came up to the space and we had a great conversation, in which he shared a LOT of deep insights into finetuning, DPO (Direct Preference Optimizations) and merging.

The series of Bagel dataset and models, was inspired by the Everything Everywhere All at Once movie (which is a great movie, watch it if you haven't!) and is alluding to, Jon trying to throw as many datasets together as he could, but not only datasets!

There has been a lot of interest in merging models recently, specifically many folks are using MergeKit to merge models with other models (and often a model with itself) to create larger/better models, without additional training or GPU requirements. This is solely an engineering thing, some call it frankensteining, some frankenmerging.

If you want to learn about Merging, Maxime Labonne (the author of Phixtral) has co-authored a great deep-dive on Huggingface blog, it's a great resource to quickly get up to speed

So given the merging excitement, Jon has set out to create a model that can be an incredible merge base, many models are using different prompt techniques, and Jon has tried to cover as many as possible. Jon also released a few versions of Bagel models, DPO and non DPO, that and we had a brief conversation about why the DPO versions are more factual and better at math, but not great for Role Playing (which is unsurprisingly what many agents are using these models for) or creative writing. The answer is, as always, dataset mix!

I learned a TON from this brief conversation with Jon, and if you're interested in the incredible range of techniques in the Open Source LLM world, DPO and Merging are definitely at the forefront of this space right now, and Jon is just at the cross-roads of them, so definitely worth a listen and I hope to get Jon to say more and learn more in future episodes so stay tuned!

So I'm in San Francisco, again...

As I've mentioned on the previous newsletter, I was invited to step in for a colleauge and fly to SF to help co-host a hack-a-thon with friends from TogetherCompute, Langchain, in AGI house in Hillsborough CA. The Hackathon was under the Finetune VS RAG theme, because, well, we don't know what works better, and for what purpose.

The keynote speaker was Tri Dao, Chief Scientist @ Together and the creator of Flash Attention, who talked about SSM, State space models and Mamba.

Harrison from Langchain gave a talk with a deepdive into 5 techniques for knowledge assistants, starting with basic RAG and going all the way to agents 👏

I also gave a talk, but, I couldn't record a cool gif like this for myself, but thanks to Lizzy I got a pic as well 🙂 Here is the link to my slides if interesting (SLIDES)

More than 150 hackers got together to try and find this out, and it was quite a blast for me to participate and meet many of the folks hacking, hear what they worked on, what worked, what didn't, and how they used WandB, Together and Langchain to achieve some of the incredible results they hacked together in a very short time.

The projects showcased a range of creative applications leveraging RAG, finetuning, and other large language models. Several projects like Magic RAG, CareerNavigator-AI, and CompetitionAI used RAG for document retrieval and knowledge enhancement. Others like rags2pizza and Naturalist DALL-E focused more on finetuning models for specific domains. Some projects compared finetuning and RAG, finding that combining both gives superior performance over using either alone but that result wasn't conclusive.

My vote as a judge (which I did not expect to be) eventually went to the team that built the OptiMUS project, they had generated a systentic dataset, cleaned it up, finetuned a model on it, and showed that they want to optimize AI agents. They used WandB to track their work and I hope they take this project forward and keep making advancements in AI. Congrats for the win Ali and Shayan, hope you enjoy the WandB branded Airpods (even I don't have those) and the Meta Quest, well deserved!

Thank you for tuning in! See you next week!

Full Transcription :

[00:00:00] Alex Volkov: Hi. Welcome back to Thursday. The Sunday special episode. This is Alex Volkov. And I'm recording this in. A gorgeous space. In San Francisco. Where I was. Invited to judge hackathon. And now I'm hanging out with a few friends from cerebral valley. So thank you. Valley folks. For letting me use this place for recording and Today, we have a special episode for you. As If you hear this on Sunday. Today's not a Thursday. We often times have special guests on the pod. Where conversations. Or deeper.

[00:00:45] Alex Volkov: And usually I reserve that slot for a Sunday special release. So this is what you're hearing now. In today's episode, we actually have two conversations. Although I only planned on one. And the first part is the planned part that you hear from Joao Maura. He is a director of AI in Clearbit, and now acquired by HubSpot. And he's also the creator of Crew AI and the Gentek AI framework that can run. By orchestrating.

[00:01:14] Alex Volkov: c

[00:01:15] Alex Volkov: The digital AI agents and have them work together.

[00:01:19] Alex Volkov: And I think you'll hear from, Joao why this peaked interest. For many folks. Specifically. Because as we caught up with. Wow.

[00:01:29] Alex Volkov: Crew AI was trending on GitHub and getting number two on product hunt at the same time. And it's a really cool framework. And I think the underlying. Power of this is that it can use open source, local models. A lot of previous agent attempts used GPT4 For example, and the crew AI can use things like Mistral or Mixtral running in LM studio or Ollama on your Mac, which I think is super cool.

[00:01:55] Alex Volkov: And I think on device AI, plus something like this framework is going to be very, very powerful. It was a great conversation was wow. And surprising to me, the second guest was not planned. However you may have heard from the previous Thursday that the. Bagel series of models from a. Self-proclaimed AI, tinker, John Durbin. Have taken over the leaderboards on hung and face. Including a bunch of mergers and we haven't. Done a deep dive into merges and merge good and Franklin state models.

[00:02:32] Alex Volkov: But if you've been to Thursday for awhile, you probably heard about them. Merging is a technique to take a model or different models. And without any computation, great, bigger or different models using a dissection and some computing. Process of the layers of those models just based on weights without any training or continuing to fine tuning, which is incredibly interesting.

[00:02:58] Alex Volkov: And John goes into this a little bit and he created. Bagel. Based on the inference of what I'll let you hear this at the end. And it's a very fascinating conversation. I took a lot from it and unfortunately we didn't have time for a long, deep dive, but I learned a lot from John and hopefully he'll come from the podcast and we'll be able to deep even dive even deeper and talk with John about. How to create data sets, why DPO is better than PPO and all of these great things. So we had two great guests. And I. Had a blast having them on the bud and I probably should do more of these deep dives.

[00:03:37] Alex Volkov: So please let me know what you think. Don't forget to subscribe to the newsletter or I sent a summary and in the newsletter, you'll find my. Trip report, quote unquote for the hackathon. There was co-sponsor with together, AI. And Lang chain and Harrison was there and I gave a brief talk as well. And the, sorry, I that a bunch of pictures.

[00:03:57] Alex Volkov: So if you're hearing this in your car, check out the newsletter afterwards on Thursday, either.

[00:04:02] Alex Volkov: And with that, I give you our first guests as well. Maura. All right, everyone. Welcome back to ThurdsAI. And we have a great guest today. João Moura from I want to say clear a bit. If I'm not mistaken. Or Joao, could you please introduce yourself and what you do and then we're going to talk about the thing we're here to talk about.

[00:04:36] Joao Moura: A hundred percent. Thank you for having me. First of all, you got my name right, it's hard to pronounce. I go by Joao, make it easier for everyone. I work at Clearbit, yes, but we just got acquired by HubSpot. I'm not sure. I'm Joao from Clearbit, from HubSpot, and from Crew AI. Everything at once.

[00:04:54] Alex Volkov: Awesome.

[00:04:58] Alex Volkov: Eye. I think it's your first time here on stage. Welcome. We've met In San Francisco, at the Ollama open source event, and I think like Teknium was there and a bunch of other folks, Ollama, and I met you and we had like a brief conversation, and you mentioned CREW. AI to me, and it sounds like super, super interesting, and then, this week and the previous week, there was like an explosion of interest in CREW.

[00:05:17] Alex Volkov: AI, so I would love to hear from you how your like last few weeks have been going, definitely the time that we spent like together since then. A lot of stuff happened to Kurei. Could you just, without saying what Kurei is, could you just, like, recap on your experience for the past two weeks?

[00:05:33] Joao Moura: A hundred percent, a hundred percent and first of all, that Oyama event the other day was so good. Had so much, so much fun on it.

[00:05:41] Alex Volkov: It was

[00:05:41] Joao Moura: last couple of weeks have been intense I gotta tell you, kind of like, the thing. Got, like, blown up out of proportion. Like, I have a lot of DMs, a lot of messages, a lot of issues, and not that many requests, I want to say, but but it has been a lot of fun.

[00:05:59] Joao Moura: Kriyai just like, seems to have a lot of interest in from different people. I think this idea of building like AI agents is something that captivate most of the tinkerers out there, like how you can automate your life away. And it seems that have been resonating with a bunch of engineers out there.

[00:06:16] Joao Moura: The last couple of weeks has been very intense in terms of Writing code, like at late nights having like to spare a few hours to insert DMs and help with the Discord community. And actually, I actually ended up recruiting my wife to help me with that. So if you see Bianca on Discord or over GitHub issues, that's my wife helped me out, make sure that I get it all covered.

[00:06:41] Alex Volkov: Definitely shout out Bianca, thanks for helping. And uh, as well so, now trending on GitHub, I think number one , I think somebody submitted this to Product Hunt as well?

[00:06:50] Joao Moura: That was a thing. So I have been working on this and like as an engineer working on an open source project, you don't, you don't think about this project as products necessarily from the get go, but as it starts to get more traction it got the interest of like this one guy that seems to be like a I don't know if it's a big deal or not, but it seems that he hunts a lot of products and product hunt.

[00:07:14] Joao Moura: And for some reason he got like. The super fun thing is that I have been part of like, and I have seen other like product, product launches, and I know how much effort goes into preparing those and to be ready for it and have like a, like social media ready to post a lot of content about it. And I had none of that.

[00:07:36] Joao Moura: I woke up in the morning and there was a message from a VC saying like, Hey, congratulations on your launch. And I was like, What is this guy talking about? I have no clue. It was very interesting because I, I opened like Product Hunt's website and I'm searching like, how do I cancel this? Like, I, I didn't want to launch this, at least not right now.

[00:07:58] Joao Moura: And on Product Hunt's like documentation, they mentioned that you have two options, either, You send them a message like super urgent so that they can pull like the, the brakes on it, or you run with it.

[00:08:13] Joao Moura: And at the end of the day, I was like, I'm just going to run with it. I'm going to see how it goes. And turns out we end up the day as [00:08:20] number two.

[00:08:20] Joao Moura: And that was, that was something else. Thanks.

[00:08:25] Alex Volkov: number one hunter. I think he hunted like most of the products on ProductHunt, so shout out Chris. And definitely, I saw this and what a surprise to wake up to and then get like the product number two. Definitely helped the stats probably. Right. So I think, I think with this excitement, let's talk about like why it's so exciting.

[00:08:43] Alex Volkov: Could you give us like a brief primer on Crew AI? We've talked about agents before. We obviously talk about like auto GPT previously and GPT engineer from, from Anton Musica and like a bunch of other very interesting projects. Could you give us the brief primer on like a crew AI, what it is?

[00:08:57] Alex Volkov: And then we're going to talk about like why you built it and the orchestration stuff.

[00:09:02] Joao Moura: percent Crew I is a very thin framework. It's a Python framework. It's in the process of being converted to TypeScript as well, but it's a Python framework that allows you to build a group of AI agents. You can think about it as if it AutoGem and ChatBath had a child.

[00:09:21] Joao Moura: That's the way that I usually describe it. So you're going to have a group of AI agents that are role playing in order to. perform a complex series of tasks. And you can do all sorts of automations on it and you can plug it to all sorts of different systems out there. I think that's the easiest way to describe it right now.

[00:09:43] Alex Volkov: Awesome. And could you, you briefly mentioned this, GPT, could you talk about like the, the inspiration here? what made you start this as Clearbit was getting acquired and, or, or around this area, at least I think what made you work on this? There's a bunch of other , orchestration platforms out there, the bunch of agents what made you write your own instead of like taking something off the shelf on open source?

[00:10:06] Joao Moura: So turns out that this is a fun story. There was so you're back into my wife again, always propping me up. I love her. She's so great. She was she was telling me, Hey, you have been doing all this amazing work at Clearbit. Because at Clearbit, we have been doing work with LLMs for the past one year.

[00:10:22] Joao Moura: And at a scale that I believe not many have. And she was like, You should be sharing more about this. Like, you're leading these efforts and you're doing all these complex systems at scale. And this could definitely help and benefit other people. So she was telling me that I should do a better job at posting online in things like LinkedIn and Twitter.

[00:10:41] Joao Moura: And Twitter, I, I think like I'm okay with, but LinkedIn was always hard to me. I feel like there is a, there is a harder threshold, like a higher threshold for how well your idea must be before you post it on LinkedIn. So I was considering like how, how I can do better LinkedIn posting. And because I was so excited about AI agents, I was like, can I build like a couple of agents that will actually help me out with this, where I can like shoveling my, like, like my, my draft and rough ideas.

[00:11:11] Joao Moura: And it's going to come up with like some guidance and a better post that I can just edit and post. It turns out that I could and that's, that's how I started QueryAI. I looked into AutoGem and I was not a huge fan on how they, like, one, they didn't have the option to execute tasks sequentially. They also have a lot of assumptions on how this agent should work together.

[00:11:34] Joao Moura: And I think The way that they work together should vary depending on the tasks that you're trying to accomplish. I was not a huge fan of it. Chat dev on the other side, I saw a lot of like good stuff on it, but it just didn't feel like a production system, right? Like it has like a game like UI, something that you would experiment with, but not something that you would deploy in production.

[00:11:56] Joao Moura: So that's, that's how I came up with this idea of like, maybe I should do something myself so I can build this LinkedIn automation. And if that works, then I can build other sorts of automations. And that's how I started to create AI. I viewed it. Five agents from A social network researcher all the way to a chief content officer to help me create great ideas so that I can post them on LinkedIn.

[00:12:23] Joao Moura: And it works great. I went from never posting on LinkedIn to post like three to four times every week. And I love what I post and it seems other people do as well. So from that point on. I decided that I want to create more automations and that's how CREATE. AI came to be. I just abstracted what I learned from that experience into this framework that I could then use to build other sorts of automations and things took off from there.

[00:12:50] Alex Volkov: Wow, that's incredible. Incredible story. As a lot of the engineering stories happen when people create like cool things, laziness is somewhere there. Like I want to automate something that I don't want to do, but I definitely need done. I definitely have a bunch of those as well, at least for Thursday. The collection stuff and the other stuff that I would love to just like happen for me.

[00:13:10] Alex Volkov: So definitely. Want to check out KuroAI for that and create like a Thursday collection thing. Could you, could you mention like the, like, like technical challenges here? You did mention that it's based on LengChain, if I'm not mistaken. You mentioned that there's not a lot of like, pull requests for people to help out with Could you talk about like the, the technical challenges you ran into?

[00:13:30] Joao Moura: Yes so basically when I start to build this out, I realized pretty quickly that Agents are just as useful as how many tools you can connect them with. And when I was looking online, I realized that both YamaIndex and LinkChain already had all these amazing tools that you could, you could run with.

[00:13:52] Joao Moura: So I wanted to make sure that I could, people could use those tools too. And build, like, Crews that use them. Because of that, I took the decision to build CREAI around LinkChain. So that if anyone wants to hook that up with their GitHub or their Gmail, there are already tools that were built out for that, and they're pretty easy to plug in and just work.

[00:14:15] Joao Moura: And it seems Lemma Index tools also work. I'm putting together some experiments around that to share with more people. But basically that was some of the initial decision that that will lead to this design. I think some of the technical challenges that came from it is It's just realizing that as people start creating all these different curls for these different use cases, there's so many edge cases, right?

[00:14:38] Joao Moura: You know that you can try to, like, steer LLMs your way, but especially if you're using, like, open source LLMs and smaller LLMs, they have a harder time just sticking with, like, a given.

[00:14:54] Joao Moura: I started to add a bunch of guardrails in Cree AI that actually makes it way better than what you would get with any other agent framework out there, where if it's For example, one of them is if you're running out of iterations, like you're, like your, your agent is stuck on a cycle or taking too long to come up with an answer it's gonna force it to come up with an answer if it goes over a certain number of iterations that you could define.

[00:15:21] Joao Moura: Another one is if it tries to use the same two in a row, it's going to prevent it to do that and guide there towards moving on. Another one is it has caching. So every two any agent uses is going to be cached so that if any other agent in the group decides to use the same two they don't need to actually execute it.

[00:15:41] Joao Moura: So I think a lot of the challenges come from like how I can add all these guardrails to make sure that Independently of what the use case and what the person is building a group of agents for, that's still going to run smoothly. And that's, that's where a lot of the work has been, has been put, been putting on.

[00:16:01] Joao Moura: So you mentioned local modals as well

[00:16:04] Alex Volkov: we mentioned, we met in the OLAMA event, and OLAMA is a CLI, a shout out, OLAMA folks, is a CLI to be able to download and run open source models on your hardware, basically. Many of the previous agent attempts, Auto GPT like different ones, they use maybe GPT 4 or something.

[00:16:20] Alex Volkov: We're getting to the tools and we heard previously in the space we heard from John Durbin that there are models now that are like better for specific tasks like function calling as well. Jao, could you speak a little bit about the difference that you see? Could Crue AI work with both, right? Open source and also like, API ones.

[00:16:39] Alex Volkov: And could you [00:16:40] talk about a little, the difference that you see between like the open source models as we have them right now versus kind of the, the online models and which ones would you prefer for your tasks?

[00:16:50] Joao Moura: Turns out that I think that the fact that crew AI supports local models is some like thing that, that. Make it take off because that's something that I wanted from the get go. Like these agents, especially if you're trying to automate complex tasks, they can become rather costly if you want to run them like 24 7.

[00:17:09] Joao Moura: But with like the ability to use local models, you can basically just set and forget, and they're going to keep doing work for you. So I wanted to make sure to support local models because of that. Guru AI supports like any of the vendors that you're going to find support in link chain. So you can use any of the open source models out there, a drawback, GPT, you name it.

[00:17:30] Joao Moura: And you can also use Zolyama, you can also use LM studio whatever is the best way that you have to run your models locally, you can use that. I. Specifically, like personally, love Olym. Olym is amazing. I love the guys that built it as well. And I think it's so easy to use that I ended up using that. And I have been using some of the smaller models.

[00:17:51] Joao Moura: Shout out to Nose Research. I love that OpenARMS 2. 5 model. It's just amazing and so small. Like I can't believe how good it is. And that's one that I use a lot for like when I'm using I'm using OpenARMS 2. 5 just because of how well it works, but I also tried with Mistro, I also tried with Solar, I also tried with Nexus so many models out there, so good.

[00:18:19] Joao Moura: One thing that I want to call out as well is that These local models, they definitely struggle a little bit more when compared to GPT 4 in terms of sticking with a given format. I'm also collecting all my executions data so that I can fine tune agentic models. Similar to how you have like instruct models and chat models, I want to make sure that we start to see more agentic models out there.

[00:18:46] Joao Moura: I have seen some closed source ones that are not like, You're not able to touch on. So I'm building an open source data set that I can then use to fine tune those models. And then you basically are going to have these agents run on local models without a glitch. That would be at least the end goal.

[00:19:05] Joao Moura: That's incredible, incredible specifically because

[00:19:08] Alex Volkov: we've, we've had interviews with a bunch of folks who build agentic stuff. So one, one of the more successful episodes of last year for Thursday, I was in an interview with Killian Lucas from Open Interpreter and the open source community here definitely opened the thread with Killian specifically to say, Hey, when the users run a bunch of this stuff, we would love to have.

[00:19:27] Alex Volkov: Users opt in maybe for some like telemetry or analytics to be able to build the data sets for the tasks that were completed or not completed. I don't know if you have this plan, but definitely this is a benefit to the community if you do have a way for folks to like, log their stuff. I also mentioned that like, I probably should reach out to you separately to see if like, these runs for these agents in crew could be logged in Weights Biases with the integration.

[00:19:50] Alex Volkov: Would be definitely more than happy to like participate and see if we can like look at the execution stuff of your agent on Weights Biases. As well, I think before I let Umesh wanted to have like a bunch of questions for you as well. He's been running and he, he does agents of his own. I want to say

[00:20:06] Alex Volkov: what's the next plans for crew? Where are you planning to take this? Many of these projects, suddenly people ask for a UI because maybe they don't want to do like, installing and, and doing like Python stuff. So you already mentioned TypeScript. Could you give us a little bit of a future sense of like, where are you planning to take this?

[00:20:23] Joao Moura: think what we are getting now is a bunch of feature requests from most bunch of different sites. So there is some prioritization going on so that I can figure out what to focus next. One thing that seems to be a no brainer to me though is that we need to have a UI for this.

[00:20:37] Joao Moura: I think this would be pretty cool and unlock a lot of use cases for people out there. I know there are other people that have been building UIs for their, like their businesses that are being built around this. I, I just think like an open source version would be better. So I'm definitely already working on the UI for this.

[00:20:53] Joao Moura: We're going to be able to. Put your agents together, bring your, all your cartoons together, and then you can basically have these agents like run by yourselves. I, I might look into offering an option where you, like, we can even host it for you, and I'm still figuring out what that would look like.

[00:21:10] Joao Moura: Maybe that's too far ahead. But but yeah, I think like the UI for it makes a lot of sense. Also another thing is that it seems a lot of the use cases kind of like go back into very similar tools over and over again. And even though you can hook them up with like link chain or lemma index tools, those might still require some like configuration.

[00:21:30] Joao Moura: It might not be as straightforward for some people. So we might take an opinionated take. On a tool specific repository and package that you can basically use to bring, let's say, let's say they want to create an agent that does reg you might be able to do that with one line versus having to be like a custom.

[00:21:51] Joao Moura: Like two cents for that. So that's another thing that we have been looking at as well. I think there's so many use cases. One thing that I'm trying to do more now is just kind of like chat with more people that are using this. Especially on the business side of things to understand like what other use cases we could support there.

[00:22:08] Joao Moura: But yeah, a lot of interesting things cooking.

[00:22:11] Alex Volkov: I'm looking forward to hear more about Kuru AI and upcoming things. I think Umesh Arkohos here has been doing Agents for a while and has a few questions as well. Umesh, go ahead.

[00:22:23] Umesh Rajiani: Yeah. Hey, Joe thank you for, for coming in. We are almost 80, 80, 90 percent of our workflow now is agentic workflow. So we are employing the generative AI library of. I think that's pretty much it for the introduction of Google for Gemini and also a lot of work using Autogen.

[00:22:41] Umesh Rajiani: And we got introduced to Crue AI, I think, four weeks ago through one of my engineers and found it pretty interesting. There are going to be a lot of pull requests coming in from us, actually, because we are thinking about a few things. I just wanted to ask you one particular question about the process part.

[00:22:59] Umesh Rajiani: Your current library, as I understand, is is a linear process library and what we have is what we are employing with Autogen is, is also. Bit of a, a graph of actions as well as the dag approach as well. Dag approach, can be implemented using your process. But do you have a, a, a graph of actions, workflow in planning or something that is coming up?

[00:23:24] Joao Moura: Yes, so this idea of processes, I want this to be like one of the cornerstones for our career AI. I, my understanding is that a lot, as I said earlier, like a lot of the different outcomes that you're going to get, a lot of the magic happens when you define true what processes these agents are going to work together, right?

[00:23:43] Joao Moura: And there are so many options out there. Like you can have them work like sequentially, you can have them work like in a group, like if they're in a meeting, you can have like a consensus strategy where they can kind of like bet to see who is going to take on the task and even evaluate the results.

[00:23:59] Joao Moura: So there's just a A lot of different processes that can be implemented there. And the idea is to implement all these processes so that people can have some work happen in parallel if they want to, or sequentially or whatnot. About a graph specific API, I I'm not sure how much I can tell about it, but we have been talking with link chain folks about it.

[00:24:19] Joao Moura: And there's, there's some things that have been cooking there.

[00:24:23] Umesh Rajiani: Enough said. This last question. So currently it is all Python but most of our implementations now because of the latency and everything and complexity of. The workflows that we are implementing, mostly our applications are enterprise applications.

[00:24:36] Umesh Rajiani: We are employing a lot of Rust to, for, for a compiled workflow. So do you have any plans of porting it to Rust or you're looking for kind of a support in that area or something?

[00:24:47] Joao Moura: Yeah. So we are, we are porting it to TypeScript right now, and there's some work being done in to build like an API where you might be able to just spin it off as like a service.

[00:24:58] Joao Moura: And you can then like [00:25:00] add agents, create agents, outrun API. So you don't have to create one yourself. You just need to figure out how you want to host it. I haven't thought about porting in trust yet, but I would be open to that idea. For sure. If I can get enough people to help out, I create a repository and we can get things working for sure.

[00:25:16] Umesh Rajiani: I'll, I'll reach out to you separately. Thanks Alex for, for allowing me to ask questions. Of course I have many questions, but I'll reach him out on his Discord.

[00:25:23] Alex Volkov: Yeah, thank you Umesh, and João, I just want to like recap on the awesome success of Kuru AI. I agree with you. I think the fact that, like, we've had many frameworks like this, we've talked about many frameworks like this, the ability to run this completely on your machine, the ability to, like, not pay for somebody else the ability to like, use Olama.

[00:25:43] Alex Volkov: I didn't know that you also support LM Studio. Shout out LM Studio, a friend of the Pada, hopefully we're, we're going to get on, on the next Thursday, I so I didn't know that I can, like, open up a local model on LM Studio and, and then the crew would use this API. Definitely. Definitely want to play with this now.

[00:26:00] Alex Volkov: I want to say, I want to give you a few minutes to just like talk to the community. A lot of things are happening in this world. I find it very interesting where kind of the AI engineers, the kind of the traditional software engineer background folks, they're building the tools, they're building the rag systems, let's say they use the link chain.

[00:26:17] Alex Volkov: From the other side, we have a bunch of machine learning folks who are Building the models, fine tuning the models, and working on that space, and reading the papers. And I do see a connection between, and obviously my role in Ways and Biases specifically is to connect these two worlds. I do want to see more people that train models also kind of like think about the agentic behaviors as well.

[00:26:37] Alex Volkov: We heard John Durbin before talk about like, hey, there's specific data sets for RAG, there's specific data sets for execution and function. I think Eroboros has The, the data set has a bunch of like function calling as well. So definitely I want to see a connection here. João, please feel free to talk to the community in terms of like what you need to make crew the best crew ever.

[00:26:57] Alex Volkov: Where can they find you, what you can get help with the floor is yours. Feel free to take over and ask everything. Community will provide.

[00:27:06] Joao Moura: A hundred percent. And just to tap into what you said there, I agree. I think like there's something magical that happened like last year with like GPT taking the world by the storm is that it like it connected two groups of engineers that in the past didn't talk very much.

[00:27:22] Joao Moura: And that was like AI and ML engineers with. regular software engineers. I have managed teams in both areas in the past, and I definitely have seen like that there isn't much interaction there, but right now it's, it's amazing to see all the amazing stuff that have been coming up from like those two groups to interacting more together.

[00:27:40] Joao Moura: It has been a lot of fun. About, about CREATE. AI. Yes, I would say give me a follow on Twitter or X, I would say now, so give me a follow on X and I definitely will keep posting and share more about CRE AI and all the things related to LLMs, Agents, and everything else. You can know more about CRE AI by looking into its GitHub.

[00:28:00] Joao Moura: So you can go into my profile slash Guru AI. I probably add the link to my ex account as well. From that, if you have follow up questions or if you want to like see what people have been cooking with it, I would say join the Discord community. We have around 500 people there and has been growing daily.

[00:28:18] Joao Moura: So if you join that, you might be able to see other use cases and things like that. If you're curious about it, but you're just like, you're, you're not sure what you could build with it there's a bunch of examples in the README and even some videos that I recorded crews doing like, stock analysis or tree planners and all that.

[00:28:38] Joao Moura: There is there's a lot of content there that you can consume in order to get your ideas. And if you do decide to give it a try, don't miss out on the custom GPT. It's also linked in the README and it can help you write the code. It can help you with ideas for the agents, ideas for the roles or for the tasks or anything around using QrooAI.

[00:28:58] Joao Moura: If you're also curious at contributing to the project. GitHub has a bunch of issues. My wife, again, has been flagging and tagging all of them. So thank you so

[00:29:07] Joao Moura: much.

[00:29:07] Alex Volkov: out, Bianca.

[00:29:08] Joao Moura: can find like all the ones that are tagged with help wanted or the ones that are related with questions And you can help answer them as well And we're gonna be writing new documentation from the scratch So this might be a great opportunity to help with like more simpler stuff as well if you're into that

[00:29:24] Alex Volkov: Awesome and I think I saw something, I don't know if I have a link

[00:29:28] Alex Volkov: to, to the generous documentation on the fly from, from just the, the code itself. And it looks super cool. I'll, I'll try to send this to you. Joao, thank you so much for joining Thursday. I, this is your first time here. Hopefully not the last.

[00:29:40] Alex Volkov: Congrats on the success of Kru AI and it's been great meeting you and then having you on definitely thank you for coming and folks should definitely check out Kru AI, give Joao a follow and we will expect more. I can't wait to like run a few Kru myself to help me with Thursday night tasks, especially on local, local models.

[00:29:58] Alex Volkov: It was super cool. Thank you for coming, man.

[00:30:01] Joao Moura: I love it. Thank you so much for having me catch you folks online.

[00:30:04] Alex Volkov: Awesome, and your audio quality was great by the way, thanks for testing out your mic.

[00:30:07]

[00:30:11] Bagel models the leaderboard from Jon Durbin

[00:30:11] Alex Volkov: We're moving forward into the top open source on the LLM leaderboard and the creator. So if you guys open the open source LLM leaderboard, which we often talk about. On HuggingFace we, we've talked about kind of the, the difference between human evaluation and the automatic evaluations that OpenLLM leaderboard runs.

[00:30:32] Alex Volkov: You will see a bunch of models. The top three ones are from CloudU and they're, they're like, I think merges of Yee34 and then the Mixtroll34b as well, but it's not based on Mixtroll. And then the rest of the is like a bunch of John Durbin Bagel examples. And, so all of those, there's like six models there that are based basically on the John's Bagel DPO versions.

[00:31:00] Alex Volkov: And I just wanted to shout this out and shout out Durbin for, for working this hard and releasing these models.

[00:31:06] Alex Volkov: Let's see if we can hear from the man himself. Hey, John.

[00:31:09] Jon Durbin: Hey, how's it going?

[00:31:10] Alex Volkov: Good. Thanks for joining us. I don't remember if you've ever been on stage. So feel free to briefly introduce yourself to the audience who doesn't know you. And definitely they should and they should follow you as well.

[00:31:22] Jon Durbin: Yeah, I'm a software engineer. I'm an AI tinker. I've been doing synthetic stuff since I guess maybe April with Aragoros project. It's been tons of fun. Lately I've been mostly working on the bagel models. If you're wondering what the bagel name came from, it's from Everything, Everywhere, All at Once.

[00:31:37] Jon Durbin: Great movie. Yeah, so that, that's the kind of the premise of the model is like all the prompt formats. Yeah. All the data sources, all the training techniques, there's Neptune, there's DPO yeah, just fun stuff there. As far as the leaderboard, that wasn't really my goal. If you look at the actual, like, token count per data set, I think the highest And then the last amount of tokens is actually probably the Cinematica dataset, which is movie scripts converted to roleplay format.

[00:32:07] Jon Durbin: So it's, it's interesting that it does so well, but really I was targeting the model for general purpose as a merge base because I know that, MergeKit is so popular now. So I was trying to come up with a base model that has a little bit of everything and every prompt format so that anyone who wants to do this, alchemy with MergeKit.

[00:32:28] Jon Durbin: Can use the Bagel series as a base, because I should, if you have an alpaca based model and a vicuña based model, they're not going to merge very well. It'll have, weird stray user tokens or whatever. The idea with Bagel is to be a good base.

[00:32:42] Alex Volkov: I also saw quite a lot of work you're doing on new DPO data sets. Could you talk about those?

[00:32:48] Jon Durbin: And then, yeah, I keep cranking out new DPO datasets to enhance the stuff that's lacking right now.

[00:32:54] Jon Durbin: I think even the YI 34B. Might be a little bit overcooked. I used QLORA for both the supervised fine tuning stage and DPO. And it turns out DPO, you really need to use an incredibly low learning rate. I was even using, like, maybe 50x smaller learning rate for the DPO phase than the Then the supervised fine tuning phase, and even then [00:33:20] I stopped the run about halfway through and killed it because the eval started spiking all over the place.

[00:33:26] Jon Durbin: Yeah, still, still lots of stuff to learn and I'd love to do a full weight fine tune of the E34B. I'm probably going to work on a Solar 10. 7B version of it next and maybe a DeepSeq 67B. I'm curious if the DeepSeq's, deeper network is actually going to improve things in any sort of way. But

[00:33:46] Alex Volkov: awesome. John, thank you so much for joining and thank you so much for the deep dive. So I have two questions for you real quick. I did not expect you to join. So this is not a full blown interview, but I'm very happy that I have you. First of all, you mentioned that there's like two versions, DPO and non DPO, of Bagel.

[00:34:01] Alex Volkov: And you mentioned the differences between them. You said like DPO version is more factual and truthful, but not great for RP. I wasn't sure what RP is. Roleplay?

[00:34:10] Jon Durbin: Roleplay,

[00:34:11] Alex Volkov: Yeah. And then creative writing. Could you give us like a little bit of a, of a sense of like, what's like DPO versus non DPO version? Is that just dataset based or is there something more going on behind the scenes that like makes the one model behave differently than the other?

[00:34:27] Jon Durbin: Yeah, so really all of the Bagel series, you basically have two phases of training. There's the super, regular supervised, fine tuning stage where I just, you can look at the Bagel repository. Everything is completely open source and reproducible. But in the supervised fine tuning phase it's just a ton of data sets and and then I take that fine tuned model, fine tuned model, and then I apply DPO, direct preference optimization to it.

[00:34:52] Jon Durbin: And I have quite a few DPO datasets in there, but really, the DPO landscape is sparse right now. You basically have DPO datasets from NVIDIA, the Helpsteer database, which is a human annotated one where they ran a bunch of gen a bunch of prompts against LLMs and then had humans rank them.

[00:35:14] Jon Durbin: Then there's like the LIMSYS, 1, 000, 000, where you can find the exact same prompt sent to multiple models. And so you can take like the GPT 4 answers. Use that as the preferred answer, and then the Kunyu 33 or something as the rejected answer, because you're assuming the GPT 401 is better.

[00:35:31] Jon Durbin: Same with there's Orca DPO pairs. I know Argya just did a new release of that, which is better. But we don't have a ton of DPO datasets that are specifically for creative writing tasks and stuff. I made one which is actually based on the Eroboros 2. 2 compared to the Eroboros 3 series where I actually rewrote most of the creative writing prompts with a different prompt and some other stuff.

[00:35:59] Jon Durbin: I actually used the March version of GPT 4 which is better. So in that case you get Basically like three to four times the number of tokens in the output. So there's that DPO data set, which I make myself in the Bagel Code. But otherwise there's really no role play focused data in any of the DPO data sets.

[00:36:21] Jon Durbin: So what happens is you take that supervised or, fine tuned model from the first phase. And you apply DPO to it, and it kind of experiences, forgetting of what it learned during the fine tuning of some of the stuff like creative writing and role play. Yeah same with code. So if you look at, my Twitter feed, you can see that I've released there's a Python DPO dataset that'll hopefully fix some of that stuff.

[00:36:44] Jon Durbin: I just released another contextual question answering DPO dataset for better RAG performance after the DPO phase. I put out just a few minutes ago Gutenberg DPO, which is basically I parse maybe 14 or 15 books from Project Gutenberg that are public domain into chapters and then create prompts to actually write those chapters and then I create summaries so you have like the previous chapter summary inside the prompt and then I use that to prompt one of the local LLMs so I used Dolphin, eChat, and Lama 213b. To get the rejected values the outputs from these models are fine in some cases, but they're short and they, you'll notice with the LLM, like most of the LLMs, when you write a story, it's always a happy ending and it, and it ends with like, and they walked into the forest lived happily ever after.

[00:37:37] Jon Durbin: It's boring and cliche. My hope with the Gutenberg stuff is that when you actually prompt it to write a chapter of a book, it's gonna be, from human writing that are popular books. They're a little bit old timey because they have to be to be public domain, but,

[00:37:52] Alex Volkov: Yeah.

[00:37:53] Jon Durbin: hopefully it will improve the writing and create creativity of the late whatever bagel models I do in the future with So I'm trying to kind of improve, improve that, but still a lot of stuff I need to do. I think the next thing I'll do before I actually make another bagel model is use something like the Goliath 120B to make a role play centric dataset for DPO. That way it doesn't completely forget how to do that as well.

[00:38:15] Alex Volkov: Awesome. And I'm just looking at the number of data sets that, like you said, everything, everywhere, all at once. And this is why it's called Bagel, Everything Bagel. It's just like an insane amount of data sets. I'm just gonna run real quick. AI2, Arc, Error Bores Belly Belly, Blue Moon.

[00:38:30] Alex Volkov: You have Capybara in there, Cinematica. Imo Bang, Gutenberg, LMsys chat, like, like tons, tons of stuff. It's incredible how well the model performs. John, one thing that I wanted to follow up on before we move on. You mentioned something that's better for RAG as well. You mentioned a DPO data set that's better for RAG.

[00:38:45] Alex Volkov: Is that the contextual DPO that you released?

[00:38:49] Jon Durbin: Yep.

[00:38:50] Alex Volkov: What, what makes it better for, for RAG purposes? Could you, could you like maybe give two sentences about this?

[00:38:56] Jon Durbin: And this is actually something you can reproduce with the AeroBoros tool as well if you wanted to generate your own data, but I have this instructor in there called Counterfactual Contextual, and what that does is it makes a bunch of fake facts, like it'll say, the Battle of Midway happened in the Civil War, something like that and it'll put that into context and then ask a question about it.

[00:39:19] Jon Durbin: And then it'll have the real version of the fact as well, World War II, Battle of Midway and then the idea is that you want to train the model to always attend to the context and not try to base the answers on what it knows from the base pre training. For example, if you're doing I don't know, like a virtual, you have a different planet where the sky is purple.

[00:39:41] Jon Durbin: And you ask the model, what color is sky, is the sky based on your lore book or whatever. You want to make sure that the model always obeys your context and, and answers accordingly, and not says the sky is blue, because I know the sky is blue. So the, the data set that I put in there has a bunch of those kinds of things.

[00:39:59] Jon Durbin: You can't just put in the fake facts, because then the model will just You know, learn to answer incorrectly. So for every, for every fake version of the context, you have to put in a real version of the context as well. The other thing that makes it better for RAG is I actually stuff more than one piece of context into it because Like with RAG, the retrieval accuracy is the hardest part, so you want to retrieve more than one document.

[00:40:23] Jon Durbin: So suppose you want to retrieve ten documents. If you want to stuff all ten of those into a single prompt and then you want to provide references to the user, you have to know which segment of the prompt it came from. This data set also includes, like, you can put metadata into the prompt for each section that you retrieve, and then when you ask for references in the output, it'll actually only reference that segment.

[00:40:47] Jon Durbin: A bunch of stuff like that, yeah, I, I put in irrelevant context as well to make, try to confuse them all because retrieval is very noisy. All of that kind of stuff is in there.

[00:40:57] Alex Volkov: First of all, I think from the whole community, thank you a lot for everything that you do and your work. And I really appreciate your time here on Thursday. You're more than welcome to always join us. And I didn't expect you to be here when I talked about.

[00:41:09] Alex Volkov: The stuff that you just released, but it's really, really awesome when people from the community who work on the stuff that they do also come and have a chance to speak about them. So John, you're always welcome on Thursday. I would love to invite you again and talk deeper.

[00:41:20] Alex Volkov: And as you release the next stuff that you're working on, I know you're working on a bunch of next stuff more than welcome to come here and, and, and discuss, or even like DM me before. So we'll know what to chat about. I will. Definitely mentioned the, the DPO datasets in the fine tuning hackathon that I'm going to this week.

[00:41:35] Alex Volkov: And so thank you for that. That, that was why I wanted to do a little bit of a deep dive. [00:41:40] And also I want to shout out you as the, one of the most active users of Weights Biases. You posted your like recap that we sent and you have two reports there. And you're part of like the top 10 percent of most active users with 2, 500.

[00:41:53] Alex Volkov: Hours trained in 23 and like 900 plus models. So that's, that's incredible. I just wanted to shout this out.

[00:42:02] Jon Durbin: Yeah, I'm a little addicted.

[00:42:03] Alex Volkov: Yeah, it's amazing. It's amazing. And I, I appreciate everything that you do and I think the community as well

👋 Hey there, been quite a week, started slow and whoah, the last two days were jam-packed with news, I was able to barely keep up! But thankfully, the motto of ThursdAI is, we stay up to date so you don’t have to!

We had a milestone, 1.1K listeners tuned into the live show recording, it’s quite the number, and I’m humbled to present the conversation and updates to that many people, if you’re reading this but never joined live, welcome! We’re going live every week on ThursdAI, 8:30AM pacific time.

TL;DR of all topics covered:

* Open Source LLMs

* Nous Hermes Mixtral finetune (X, HF DPO version, HF SFT version)

* NeuralBeagle14-7B - From Maxime Labonne (X, HF,)

* It's the best-performing 7B parameter model on the Open LLM Leaderboard (when released, now 4th)

* We had a full conversation with Maxime about merging that will release as a standalone episode on Sunday!

* LMsys - SGLang - a 5x performance on inference (X, Blog, Github)

* NeuralMagic applying #sparceGPT to famous models to compress them with 50% sparsity (X, Paper)

* Big CO LLMs + APIs

* 🔥 Google Deepmind solves geometry at Olympiad level with 100M synthetic data (Announcement, Blog)

* Meta announces Llama3 is training, will have 350,000 H100 GPUs (X)

* Open AI releases guidelines for upcoming elections and removes restrictions for war use (Blog)

* Sam Altman (in Davos) doesn't think that AGI will change things as much as people think (X)

* Samsung S24 has AI everywhere, including real time translation of calls (X)

* Voice & Audio

* Meta releases MAGNet (X, HF)

* AI Art & Diffusion & 3D

* Stable diffusion runs 100% in the browser with WebGPU, Diffusers.js (X thread)

* DeciAI - Deci Diffusion - A text-to-image 732M-parameter model that’s 2.6x faster and 61% cheaper than Stable Diffusion 1.5 with on-par image quality

* Tools & Hardware

* Rabbit R1 announces a deal with Perplexity, giving a full year of perplexity pro to Rabbit R1 users and will be the default search engine on Rabbit (link)

Open Source LLMs

Nous Research releases their first Mixtral Finetune, in 2 versions DPO and SFT (X, DPO HF)

This is the first Mixtral finetune from Teknium1 and Nous team, trained on the Hermes dataset and comes in two variants, the SFT and SFT+DPO versions, and is a really really capable model, they call it their flagship!

This is the fist Mixtral finetune to beat Mixtral instruct, and is potentially the best open source model available right now! 👏

Already available at places like Together endpoints, GGUF versions by the Bloke and I’ve been running this model on my mac for the past few days. Quite remarkable considering where we are in only January and this is the best open chat model available for us.

Make sure you use ample system prompting for it, as it was trained with system prompts in mind.

LMsys new inference 5x with SGLang & RadixAttention (Blog)

 LMSys introduced SGLang, a new interface and runtime for improving the efficiency of large language model (LLM) inference. It claims to provide up to 5x faster inference speeds compared to existing systems like Guidance and vLLM. 

SGLang was designed to better support complex LLM programs through features like control flow, prompting techniques, and external interaction. It co-designs the frontend language and backend runtime.

- On the backend, it proposes a new technique called RadixAttention to automatically handle various patterns of key-value cache reuse, improving performance. 

- Early users like LLaVa reported SGLang providing significantly faster inference speeds in their applications compared to other options. The LMSys team released code on GitHub for others to try it out.

Big CO LLMs + APIs

Meta AI announcements (link)

These #BreakingNews came during our space, Mark Zuckerberg posted a video on Instagram saying that Llama3 is currently training, and will be open sourced!

He also said that Meta will have 350K (that’s not a typo, 350,000) H100 GPUs by end of the year, and a total of ~600,000 H100 equivalent compute power (including other GPUs) which is… 🤯 (and this is the reason why I had to give him double GPU rich hats)

Deepmind releases AlphaGeometry (blog)

Solving geometry at the Olympiad gold-medalist level with 100M synthetic examples

AlphaGeometry is an AI system developed by Google DeepMind that can solve complex geometry problems on par with human Olympiad gold medalists

It uses a "neuro-symbolic" approach, combining a neural language model with a symbolic deduction engine to leverage the strengths of both

The language model suggests useful geometric constructs to add to diagrams, guiding the deduction engine towards solutions

It was trained on over 100 million synthetic geometry examples generated from 1 billion random diagrams 

On a benchmark of 30 official Olympiad problems, it solved 25 within time limits, similar to the average human medalist

OpenAI releases guidelines for upcoming elections. (Blog)

- OpenAI is taking steps to prevent their AI tools like DALL-E and ChatGPT from being abused or used to spread misinformation around elections

- They are refining usage policies for ChatGPT and enforcing limits on political campaigning, impersonating candidates, and discouraging voting

- OpenAI is working on technology to detect if images were generated by DALL-E and labeling AI-generated content for more transparency  

- They are partnering with organizations in the US and other countries to provide users with authoritative voting information through ChatGPT

- OpenAI's goal is to balance the benefits of their AI while mitigating risks around election integrity and democratic processes

Microsoft announces copilot PRO

Microsoft announced new options for accessing Copilot, including Copilot Pro, a $20/month premium subscription that provides access to the latest AI models and enhanced image creation.

Copilot for Microsoft 365 is now generally available for small businesses with no user minimum, and available for additional business plans.

This weeks Buzz (What I learned with WandB this week)

Did you know that ThursdAI is not the FIRST podcast at Weights & Biases? (Shocking, I know!)

Lukas, our CEO, has been a long time host of the Gradient Dissent pod, and this week, we had two of the more prolific AI investors on as guests, Elad Gil and Sarah Guo.

It’s definitely worth a listen, it’s more of a standard 1:1 or sometimes 1:2 interview, so after you finish with ThursdAI, and seeking for more of a deep dive, definitely recommended to extend your knowledge.

AI Art & Diffusion

Zero shot face adapted image gen - 3 different tech approaches

What used to take ages, now takes seconds with 0 shot, there are quite a few approaches to generate images with real human faces, in 0 shot capacity, providing just a few faces. Gradio folks call it Zero-shot face-adapted image generation and there are 3 tools to generate those:

1⃣IPAdapter

2⃣PhotoMaker

3⃣InstantID

Here’s a great summary thread from Gradio folks for this fast advancing field! Remember we had to finetune on faces for a long time? Dreambooth and then LORAs, and now we have this exciting development.

Tools & Hardware

Rabbit R1 partners with Perplexity

The R1 device that was just announced, is about to sell through it’s first 50K in just a few days, which is remarkable. I definitely pre-ordered one, and can’t wait to get my hands on it. Jesse the founder has been all over X, getting incredible recognition, and after a few conversations with Aravind Srinivas, they agreed to make a deal right on X.

Today they hopped on a space and announced that all the first 100K early buyers of Rabbit are going to get a full year PRO subscription of Perplexity (one of the best AI search engines out there) for free! I sure as heck didn’t expect it, but the email was sent just a few minutes after the X space, and now guess who uses perplexity pro?

Here’s an example of a perplexity searching ThursdAI content (it doesn’t always get it right tho)!

I guess that’s it for today, as I’m writing this, there are incredible other stuff getting released, Codium open sourced AlphaCodium (here’s a link to the founder talking about it) but I didn’t have a second to dive into this, hopefully will bring Imatar to ThursdAI next time and chat about it!

Have a great weekend all 🫡 (please give us a good review on Apple Itunes, apparently it really helps discovery!)

Full Transcription for convenience:

[00:00:02] Alex Volkov: Hey everyone, happy Thursday. My name is Alex Volkov. I'm an AI evangelist with Weights Biases, and this is Thursday AI.

[00:00:13] Alex Volkov: We had such a great show today, over 1100 of you tuned in to the live recording, which is incredible.

[00:00:30] I also wanted to say that if you're not subscribed to thursdai.news newsletter, please go ahead and do because I send a full blog with the links to the show notes and to the speakers that we have on stage, and you should be able to follow up.

[00:00:46] Alex Volkov: There's a bunch of multimedia, like videos, that are not coming through in the audio only podcast format. So please subscribe to ThursdayEye. News as well. This live recording, we also hosted Maxime Lebon, who's a senior machine learning scientist with J.

[00:01:04] Alex Volkov: P. Morgan, and the author of several models, and Merged models, lately the Neural Beagle model that we've talked about. We had a great conversation with Maxime. And that full episode will be posted as a Sunday special evergreen content episode. So please stay tuned for that.

[00:01:29] Alex Volkov: It's been an incredibly illuminating conversation in the world of merging and merge kit and everything else that Maxim does and it was a super cool conversation. So that's coming soon.

[00:01:41] Alex Volkov: And, as I've been doing recently, the following is going to be a 7 minute segment, from the end of the live recording, summarizing everything we've talked about.

[00:01:54] Alex Volkov: I hope you've been enjoying these TLDR intros. Please let me know in the comments if this is something that's helpful to you.

[00:02:05] ThursdAI Jan18 TL;DR recap by Alex

[00:02:05] Alex Volkov: Alright we started with talking today, Thursday I, January 18th. We was talking about n News imis, the Mixt mixture fine tune that came out from Teo and the folks at News. It, it was of the first fine noon of mixture, the mixture of experts model from a mistral that came from the news research folks.

[00:02:35] Alex Volkov: And it released in two versions, the DPO only version SFT plus DPO version. Given different data sets they was trained on and actually different capabilities. It looks based on the community, the DPO version is like very well performing. I've been running this on my Macbook with LM studio and it really performs well.

[00:02:53] Alex Volkov: So shout out and folks should try this. This is By far the best, looks like the best new Hermes model based on just benchmarks. They're trained on the best open source model that's currently Mixtro. Mixtro is number 7th in the world based on LMCS Arena, and that's an open source model that we all get to use.

[00:03:10] Alex Volkov: Then we've covered the Neural Beagle 14. 7b from Maxim Le Bon. Maxim also joined us for a full interview that you can hear as part of the a podcast episode and Maxim released a Neural Beagle, which is a merge plus a DPO fine tune. And it's one of the top performing 7 billion parameters on the OpenLM leaderboard.

[00:03:30] Alex Volkov: When released in a few days ago, now it's fourth. So the speed with which things change is quite incredible. We then covered the LMSYS. SGLang attempt is a 5x performance inference bunch of techniques together on the front end and the back end called Radix attention on the back end and the SGLang way to run through inference code on the front end that combines into almost a 5x performance on inference.

[00:03:56] Alex Volkov: 5x is incredible Nistan mentioned that it does less than 5x on like longer sequences and then we had a conversation about Where it could improve significantly, which is agents and agents are sending short sequences. Alignment Labs told us that this could be significant improvement in that area.

[00:04:13] Alex Volkov: So our agents are about to run way faster. A 5x improvement is just incredible. And we also mentioned that at the same day when this was released, another Optimization was shouted out by Tim Ditmers from the Qlora fame called Marlin that also improves by 4x some significant inference techniques.

[00:04:34] Alex Volkov: And I wonder if those can be compiled together in some way. Quite impressive. We also covered neural magic doing spars, pacification and sparse. And we did in a deep dive into a short, deep dive. Thank you. Alignment and thank you Austin for what's spars, pacification means. And they do in this for like major models and they compress them with specification to around 50% sparsity.

[00:04:55] Alex Volkov: It's zeroing. Out the weights that you don't actually use. And it makes the models like significantly smaller. We covered Desilang a little bit. We didn't actually get to the diffusion. I'll just read out those updates as well. Then we covered the OpenAI had new guidelines for upcoming elections, and they're trying to add techniques for folks to identify daily generated images.

[00:05:18] Alex Volkov: And they're adding, restrictions to how their LLMs are used in the context of voter suppression, etc. We then talked about DeepMind and AlphaGeometry, where DeepMind released And open sourced looks like a model called Alpha Geometry that uses neuro symbolic approach with two models that solves geometry at almost a gold medal at the Olympiad level.

[00:05:42] Alex Volkov: So Geometry Olympiads and quite impressive this release from from DeepMind and shout out. It was trained on a hundred million synthetic data set sources. A source from like more than one billion. Or so random examples and it's quite impressive. So shout out DeepMind as well. We also briefly mentioned Samsung that has a Samsung S24, the flagship phone that now Apple is needed to compete with, that has AI everywhere, uses the new Qualcomm chip and has AI in.

[00:06:10] Alex Volkov: Pretty much summarization everywhere. There's like a button with the sparkles with AI. And one cool thing that we haven't mentioned, but I saw MKBHD on Twitter review is that they added real time translation of calls. So you can literally call some people with a different language and on device translation, after you download the model on device, we'll actually be able to translate this in real time.

[00:06:30] Alex Volkov: So you can read what the other person said in different language, but also hear it. And that's like quite cool. Then we had a deep interview with Maxim Lebon, the author of many things. Recently, we've talked about Fixtral or Fixtral, the mixture of experts of the five models. We've talked about merges.

[00:06:46] Alex Volkov: Maxim had a great explanation on, on, on his blog. And then on the Hug Face blog about what merges, what MergeKit does and how that. Plays into the whole ecosystem, the top LLM leaderboard now has been taken over by merges, specifically, likely because merging models does not require additional computer, additional training, and that's fairly easy to do with just the code merges takes and combines.

[00:07:11] Alex Volkov: With different, using different algorithms like SLURP and other algorithms it combines different models and different weights from different models, including potentially building models of novel sizes. So we've seen 10 billion parameter models, like 120 billion parameters so you can use those techniques to Combine models or merge models into different ways.

[00:07:31] Alex Volkov: There's also Frankenmerge that uses different models to combine into one. So we dove into that and what the inspiration for merging and what it actually does. Maxim also released like Lazy Merge Kit, which is a thin wrapper on top of the merge kit from Charles Goddard. So shout out to Charles.

[00:07:47] Alex Volkov: So we had a very interesting interview about merging and thank you, Maxim, for joining us. Definitely worth a listen as well. And then we had breaking news from BigZuck and the meta team that talked about he gave an update about the number of GPUs that they have. And by the end of this year, they're talking about 350, and overall 600, 000 H100s or equivalents of compute which they're going to use for AI and Metaverse.

[00:08:14] Alex Volkov: And Definitely a great update. They're training Lama 3 right now. The stuff that we didn't get to, but I wanted [00:08:20] to update, there's a, and I will add in show notes. There's a stable diffusion code that runs 100 percent in the browser with WebGPU and Diffusers. js, a thread from ClipDrop, the CEO Cyril Diagne.

[00:08:32] Alex Volkov: And there's also, we've talked about DeciEye, the company that releases a bunch of models. They release DeciDiffusion, a text to image model with only 370, the 300. Sorry, 732 million parameters. It's twice as fast and 61 percent cheaper than Stable Diffusion with the same image quality, so that's getting improved.

[00:08:51] Alex Volkov: But I think they're talking about Stable Diffusion 1. 4, so not SDXL or the new one. And Desi, I also released Desi Coder, and we also covered the Stable Diffusion Coder that is a coding model that runs closer on device, a 3 billion parameter model that beats Code Llama 7b. I think that's most of the stuff we talked about.

[00:09:09] Alex Volkov: And then one of the major things that Umesh brought we've talked about corporate drama, maybe a new segment in Thursday Eye where Microsoft, Did some things that actually disrupted workflows and companies actual products built on top of Microsoft, which is considerably not great and led to a fight.

[00:09:30] Alex Volkov: Hopefully not, but potentially a legal battle as well, and that's not something that should be done by a cloud provider such as Microsoft. Very ugly. In addition to this, we also talked about Microsoft announcing the CoPilot Pro that's now open for small businesses for 20 bucks a month with no minimum seats as well.

[00:09:46] Alex Volkov: And I think that's most of the things that we've mentioned

[00:09:49] Alex Volkov: Let's go.

[00:09:51] Sounds: to all of you.

[00:09:57] Alex Volkov: from, I guess

[00:09:59] Sounds: all of you. Namaskaram to

[00:10:07] Alex Volkov: 2024, we all need to get used to say 2024 at this point we have a bunch of AI news. My name is Alex Volkov, I'm an AI evangelist with Weights Biases, and I'm joined on stage here with dear friends, co hosts of Thursday AI. Podcast, newsletter, live X recording, community, I don't know, a bunch of other stuff as well.

[00:10:29] Alex Volkov: Nishten does paper readings, is a semi part of this as well. Welcome everyone. Welcome.

[00:10:33] Introduction to the Session's Structure

[00:10:33] Alex Volkov: I will just say a few things before we get started. So first of all, for those of you who are new, who are listening to this for the first time first of all, welcome.

[00:10:41] Alex Volkov: It's great that you have found us. Please DM me with like how you found us. I would love to know as I'm looking into the channels, et cetera. However, I will say that we've been here every week, pretty much at the same time. I don't think we've changed time since the summer. So 8.

[00:10:55] Alex Volkov: 30 AM Pacific and we try to do this every Thursday. I think we missed one or two. I was sick once, apologies. But other than that, we're here to talk about the AI every week. And what happens often is as we as we talk about things, different breaking news happened and folks announced different stuff on Thursday., and we cover pretty much everything. A very broad spectrum in AI changes. So I know there's like spaces to talk about diffusion, specifically art spaces as well. So we cover diffusion to an extent, but we try to focus on I guess our main focus is open source LLMs. We love those. We have a bunch of folks here on stage. They're training and fine tuning the greatest kind of open source models and definitely follow up on the different how should I say, different techniques, like the merging stuff that we're going to talk to at length later, and we, we hopefully get to hear about them first before they take over hug and face which was the case, I think with some of the models and some of the techniques.

[00:11:54] Alex Volkov: And I see two more folks joining us as well from different areas of the open source community. So I will say welcome LDJ and welcome alignment, LDJ. You've been missing in action. I was just saying, how are you, man? Welcome back.

[00:12:08] Luigi Daniele: Yeah, I'm doing good. Glad to be

[00:12:10] Alex Volkov: Yeah. And also we have Austin AKA Alignment Lab. What's up Austin?

[00:12:16] Alignment Lab: Oh, dude, I'm doing great. I was actually just in a call with LDJ and he was like, oh, Thursday Eye is starting and I was like, let's go.

[00:12:22] Alex Volkov: Yeah that's exactly what I like to hear that the calendar events is popping off and Thursday is starting.

[00:12:27] Open Source AI: Nous Hermes Mixtral Finetune + DPO deep dive

[00:12:27] Alex Volkov: So with that, I think it's time for the open source stuff.

[00:12:44] Sounds: Open Source AI, let's get it started.

[00:12:48] Alex Volkov: All right, so welcome to probably the biggest, the most fun, the most Contentful section of Thursday ai, where we talk about open source, LLMs and lms. I guess we should also start mentioning because a bunch of these models that we see are also multimodal, and I guess we'll start with.

[00:13:08] Alex Volkov: , News Hermes Fine Tune on Mixtral we've been waiting for this, Mixtral was released I want to say a month or so ago, a month and a half ago, and now we're getting one of the top kind of data sets and fine tunes trained on Mixtral, and we're getting this in multiple formats.

[00:13:25] Alex Volkov: Again, shout out Technium. If you guys don't follow Technium yet what are you even doing showing up on Thursday? I definitely give Technium a follow. But Mixtral fine tune is available and it comes in two variants and SFT and then DPO and SFT only. So SFT is a supervised fine tuning and DPO, direct preference optimization.

[00:13:45] Alex Volkov: This is like a, not a new technique, but definitely has been around for a while. Many people are using DPOs at this point. We've talked about DPO multiple times. I think we also saw, Nistan, correct me if I'm wrong, the actual mixtural instruct is also DPO, right? We saw this in the paper.

[00:14:00] Alex Volkov: So DPO is everywhere. And this is not the first time that the SFT and DPO pair is getting released separately. I think we've chatted with John Durbin who's, shoutout John, is in the audience. And that conversation is on the feed. So definitely check out the conversation with John.

[00:14:16] Alex Volkov: And the Bagel models were also released separately with SFT and the DPO version as well. And I think John back then mentioned that each one has Different different things it's good at. And I also would love to figure out which one of the new, Neus Ermis Mixtural Fine Tunes is best at what.

[00:14:33] Alex Volkov: Technium has a bunch of stuff in in, in the thread, so I'll link this below for examples. And I will say that the comparisons to Mixed Real Instruct. Technium posted a bunch of comparisons to Mixed Real Instruct. And it's interesting that not all of the benchmarks look like on improvements.

[00:14:51] Alex Volkov: There's a few, I think on GPT4ALL and HelloSwag. The base model, at least the non DPO base model, still wins just by a little bit. But everything else, like ARX, AGI, EVAL, and MMLU are significant improvements. And we're gonna probably continue to see those improvements. Shoutout. If you have tried it, please let me know.

[00:15:08] Alex Volkov: I will say this last thing, that finally, after setting up LM Studio again, shoutout to LM Studio we'll get to chat with LM Studio at one point. Hopefully soon, I am now, the first thing I do is download these models because it's super, super easy. Both of them, Studio and Allama, and there was a tiny, I think, quantization thing in the beginning, and now there isn't, and now it works great.

[00:15:33] Alex Volkov: And these models, I've loaded them up on my Mac before, before a flight. And I was just able to chat with this AI with no internet connection or like poorly internet connection. It was really something. I know we've talked about this multiple times. Hey, put this on a a thumb drive and then have all of human knowledge, quote unquote.

[00:15:51] Alex Volkov: I'm not really saying it's all human knowledge, but I've been actually able to do this before my flight and it was really cool.

[00:15:57] Alex Volkov: And I think the last thing to mention here is that Technium suggests to make liberal use of system prompts. So all of Hermes models, which is, there's now a bunch of Hermes models flying around, definitely the most. At least the famous one is Hermes, I think, 7B, but also the YI version, and this seems to beat the YI version as far as our friend Wolfram Raven, Wolfram Loco Lama tested.

[00:16:22] Alex Volkov: This is probably the best news model out of them all. So far, obviously it's based on the best. Open source model called Mixtro and definitely liberal use of system prompts. Yeah, roleplay is suggested setting expectations, specifications and everything else you can think of. Very easy to do with Elm Studio.

[00:16:39] Alex Volkov: I haven't [00:16:40] dove into like actually how to steer these models for exactly the task that I do. Luigi, you said LDJ, you said that you want to Tell me how to use LM studio in regards on this. So I would love to hear from you. First of all, have you had a chance to try these models specifically? And second of all let's talk about system prompts in LM studio a little bit, because I think it's a part that people are definitely missing.

[00:17:02] Luigi Daniele: Yeah. A lot of the latest models like Hermes and I think maybe Dolphin too, trained with system prompts. So if you really want to get the best use out of it definitely use that and it's just same thing with chat GPT really, where it's give instructions of how you maybe want to have it respond to you, or maybe add in a few threats of, of what you would do to the AI if it does not respond correctly, and so surprisingly that seems to actually sometimes.

[00:17:28] Luigi Daniele: Give good results, I personally try to always say please and thank you, but yeah yeah. And there's also prefix and suffixes, which I think I talked to you about, Alex,

[00:17:36] Alex Volkov: You briefly mentioned this, but maybe worth like a given a little bit of a heads up for folks.

[00:17:41] Luigi Daniele: yeah I think it really is worth maybe just a sit down and just a video with me and you actually going through it, because,

[00:17:47] Alex Volkov: Sure.

[00:17:47] Luigi Daniele: it's a decent amount to go through, but, yeah on the model card of most models, if you just look at something called prefix or suffix that is usually described in the model card, then You apply that to the LM Studio settings on the right panel in the chat settings.

[00:18:03] Luigi Daniele: And yeah, you just make sure you have those things right. If you don't, there's a good chance you're not actually using the model correctly. And it's not going to give you the best results.

[00:18:10] Alex Volkov: And they differ from the base model as well. Like we've seen like different base models have different things that you want to you want to add there. And you may getting like the same performance, but getting under performed a little bit. I'll also say for folks who are using Mac the Silicon, Apple Silicon, there's a little hidden checkbox there that I don't know if it's like, it's by default already.

[00:18:30] Alex Volkov: It's called use Apple Metal. And definitely make sure that's on for you. Significant improvement in performance and inference. All so I think NeuralRMS, anything else on folks here on stage that want to talk about this model and how it was trained and the difference in DPO? Folks, feel free to chime in.

[00:18:45] Alignment Lab: There's the cool thing about DPO is It's so it's a reinforcement learning technique. I don't know if anyone else has had a chance to read the paper about it, but essentially what occurred was that some researchers found that the, that transformers already have a baked in optimal reward function.

[00:19:03] Alignment Lab: And so what DPO is really doing is just training the model on that reward function, just biasing it towards the selected. Like good example when you give it a good and bad example pairs not directly unique to to the, to this model, but it is super interesting because it really opens up a whole bunch of possibilities for what you can do with the model now that you can give it negative examples and get more performance for it.

[00:19:27] Alex Volkov: DPO is ranking different outputs in terms of like preference, . So can you talk about the pairs stuff? Everybody says DPO pairs, like what do they mean by pairs? Could you say this about this?

[00:19:38] Alignment Lab: instead of training on like typically what you would do is you would build your data set. And that would be like your good data set. You'd have a weaker model that you, than the one that you use to synthesize the dataset or just bad examples of responses for every single example in the dataset.

[00:19:54] Alignment Lab: So if you have one that's like, how do I make a cup of tea? And then instructions about how to make a cup of tea, then you'd also have that paired with a negative example of, a response to how do I make a cup of tea? And then, the response is something else, like how to build a Lego house or whatever.

[00:20:08] Alignment Lab: And when you go to actually train it, you show it both at once, and you tell it which one is the positive and which one's the negative, and you just bias it towards the positive. It's quite similar, conceptually, to the way that OpenChat does the CRLFT training, although OpenChat actually has a specific token for the good and bad examples that it has weighted.

[00:20:34] Alignment Lab: But functionally, it's, the idea is the same. You're just doing reinforcement learning which lets you take data where you may have bad examples in there, and rather than having to remove them and waste data, you can now make a good example and get more out of it than you would have been by just replacing it.

[00:20:50] Alignment Lab: So it lets you recoup extra performance out of bad data.

[00:20:54] Alex Volkov: Thanks for the explanation. And definitely we've seen at least in my game plays with the bigger model and the DPO version of noose. RMS mixture this feels like the DPO at least behaves a little bit. Actually don't know how to attribute this to the technique or to the datasets, but it's really good.

[00:21:13] Alignment Lab: Yeah, we've noticed if we do a regular supervised fine tune first, like a just normal fine tuning, and then we DPO over that we, the models push just much further than either thing alone, too. I don't know if that's unilaterally true, because we do a fairly, specific kind of model when we make these big releases, but it seems, at least for the case of just general reasoning skill it helps a lot.

[00:21:37] Alex Volkov: Yeah, it's super cool. And I guess the downside of this, not the downside, but the outcome of some of this is that folks now have, folks who want to just use a model and are trying to maybe tune in to Thursday Eye to know which model is good to use, or maybe they're reading the local Lama stuff.

[00:21:53] Alex Volkov: There's now so many choices, including so many configurations. So maybe we should do Like a recap and also a simplification LDJ for like system messages and the prefixes alignment with DPO versus SFT. Just simplify and say, Hey folks, use this. Because right now there's so many, you can choose between quantization methods.

[00:22:11] Alex Volkov: There's at least four or five different ones for you to choose from. And LM studio says in a few of them, use this is recommended, but it says recommended for five, five different ones. There's different quantization providers as well, right? So the bloke is obviously the most familiar one,

[00:22:26] Alex Volkov: there's now a choice between DPO or SFT or DPO plus SFT, and We haven't even begun to talk about merges, which is coming as well. So there's a lot of choice and we need to simplify this for folks. So definitely just to simplify the Hermes models are usually very well behaved and great for role play as well.

[00:22:43] Alex Volkov: Try them out. If you have the room to run Mixtrl for your stuff, Mixtrl is definitely by far the best open source models that we have. Go ahead, Levent.

[00:22:52] Alignment Lab: Yeah, so Mixtrel is, that model is the architecture is very similar to a really old, comparatively old architecture that's been tried and true before. And so because of that, there's a lot of efficiencies that we just haven't integrated into the modern stack, but that will come.

[00:23:09] Alignment Lab: And there's a bunch of new ones that people have been making. And between the new quantization methods that you can do with Mixtro, because since it's sparse MOE, it doesn't actually, need all of its weights as much as it, as as each other. So some of them are, like, less important. It lets you quantize those quite a lot without actually hurting the model's performance very much.

[00:23:27] Alignment Lab: And you can also offload these layers when they're not being used. And then you can do like expert pre caching, where you predict some experts ahead of time, which lets you get faster inference speed. And at the end of the day, if the sort of quick sharp, which is like 2 bit quantization method continues to prove out that it's as performant as it claims, We could end up running Mixtro on 4 gigs of VRAM, like on a laptop.

[00:23:58] Alex Volkov: And

[00:23:59] Nisten Tahiraj: We will.

[00:24:00] Alex Volkov: we will.

[00:24:00] Nisten Tahiraj: it to perform a bit better.

[00:24:02] Alex Volkov: So I guess this takes us to the next, I'll go ahead and stand, and it's going to take us to the next optimization stuff.

[00:24:09] Nisten Tahiraj: We could definitely have it run on on 4 gigs. I've had it a little above 4. However, but the point is to have it run well. The quantization, it still makes it a little bit unfit for anything other than very short conversations. And we'll get it there.

[00:24:30] Alex Volkov: All right. So in this, in, in this

[00:24:32] Nisten Tahiraj: we'll have Mixtro under 4 gigs very soon and it'll be good.

[00:24:37] Nisten Tahiraj: Yes.

[00:24:37] Alex Volkov: And that's a promise. That's a promise.

[00:24:39] LMsys SGlang - increased inference by 5X

[00:24:39] Alex Volkov: So what happens is once you go and put those bigger models on slower hardware, which is possible you then wait painfully a long time for inference to actually happen. But this takes us to the next thing from the folks from LMSys. They released a fast and expressive LLM inference with Radix attention and SG Lang.

[00:24:59] Alex Volkov: So folks from [00:25:00] LMSys, if you guys remember from Models like Vicuna that took Lama and trained it on additional datasets. and NMSIS Arena and all these places like we definitely trust them at least with some of the evaluation stuff. I think, is MMLU also in NMSIS's area? Or at least they test on MMLU. They released a inference optimization kind of collection of techniques.

[00:25:24] Alex Volkov: I don't think it's one specific technique because there's like Radix attention. Yeah, go ahead.

[00:25:28] Alignment Lab: It's where all this was going in the first place between all these sort of different prompting programming frameworks and inference engines. What they've done is they built out the back end with the end goal of having an extremely controllable, steerable compiling system for programming outputs from a, from like an AI in the way, like a Pydantic or in the way that you would typically use sort of structured grammars and sampling techniques.

[00:25:58] Alignment Lab: And way more. It's hard to explain in, in summary in a way that's very easily grokkable without getting too technical but it's a combination of many things that we've been doing individually, which were always gonna be one big thing, they just saw it first and did it first, and now, when you're looking at it, it seems very obvious that this is probably how things should look going forward

[00:26:17] Alex Volkov: so let's actually talk about

[00:26:18] Bluetooth: overall, just a

[00:26:19] Alex Volkov: they have. Yeah, they propose like different co designing the backend runtime and the frontend language, which is like Alain said, a structured domain specific language embedded in Python to control the inference generation process. It's called domain specific language, DSLs.

[00:26:35] Alex Volkov: I, I think many folks have been using some of this. I think DS p Ys as well from is being like mentioned in the same breath. And then this language like executed in the interpreter code or in compiler code. And on the backend they have this radix attention technique for automatic and efficient KV cache reuse.

[00:26:53] Alex Volkov: I don't know if that's like instance like MOE specific or not yet, but definitely. The combination of those two plus the code that they've released shows just incredible results. Like folks, we live in an age, and we've talked about multiple of those techniques. We live in the age where somebody like this can come up and say, Hey here's an example of a set of techniques that if you use them, you get.

[00:27:12] Alex Volkov: 5x improvement on inference. In the same breath that we're saying, Hey, we're going to take Mixtrel and put it in 4GB, and we've seen this obviously with Stable Diffusion, which we're going to mention that runs fully in the browser, we're now seeing releases like this from a very reputable place. A collection of techniques that have been used to some extent by some folks, and now all under one roof, under one like GitHub.

[00:27:35] Alex Volkov: Thing that actually improves the inference by 5x on all of the major evaluations, at least that they've tested, that we always talk about. So 5x on MMLU and HelloSwag is significantly more performant, all these things. Quite impressive. One thing that I would definitely want to shout out is that the maintainer of Lava the LMM, the kind of the visual Lama, is definitely also replied and said that the execution of Lama is actually, of Lava, is actually written in the report itself.

[00:28:07] Alex Volkov: And it improves lava execution by 5x as well. And by execution, I mean like inference speed, basically. So without going like too much into Radix attention, because honestly, it's way too heavy for the space. It's quite incredible that we get, do we get stuff like this from like places like LMCS, specifically in the area of running smaller models, sorry, running bigger models with smaller hardware.

[00:28:33] Alex Volkov: Go ahead, Nissan.

[00:28:36] Nisten Tahiraj: I'll say something. So it does automate a lot of the tricks that people have been pulling, and it works great for large amounts of smaller prompts. Once you go to longer prompts, the benefit is not that much compared to VLLM. I think it felt like five or ten percent faster when it came to VLLM. So again, I haven't taken a very deep dive into it.

[00:29:01] Nisten Tahiraj: Just want to just make people aware that it's fantastic for smaller prompts and stuff. But for longer ones, you don't necessarily need to switch your whole stack to it. VLLM still works fine. Yeah, I think for if you're doing like what you would normally be doing with VLLM, which is like processing like large amounts of data or serving for just general purposes.

[00:29:24] Nisten Tahiraj: Probably, there's no need to switch your stack. I think for, specifically what it feels optimized for is Asian frameworks, in which you have many models communicating short strings back to each other. One model wearing many hats. And the optimizations just while we're on the topic, is crazy right now.

[00:29:43] Nisten Tahiraj: There's still three papers with major inference optimizations for MixedRole alone, as well as for VLLM, and that seem to compose everything pretty well. Having an alternative to VLM that's similarly. Performance is huge because VLM is a big bottleneck on a lot of stacks because of the way that it handles attention off on the CPU.

[00:30:00] Nisten Tahiraj: It feels a lot like when llama CPP got like offloading the same week that speculative decoding came out with hugging face transformers and. Everything just got a hundred times faster, like a half a year ago or so.

[00:30:12] Alex Volkov: Yeah, I would also it definitely felt like that day when LMS released the SG Lang optimization that we just now talking about I don't have a link for this, but also LES released from IST Austria. Released Marlin, which is a 4 bit, I think the way I know it's cool is that, Tim Dittmers from QLOR retweeted this and said this is a huge step forward.

[00:30:33] Alex Volkov: And Tim Dittmers is the guy who in KUDO mode, the codes, KUDO kernels, within like a night or something, planning for 3 months and then finishing. So I know that Tim Dittmers, when he says something is a huge deal, he probably Probably knows what's up. So Marlin released the same day that like the SGLang released and it's a linear kernel for LLM entrants with near ideal.

[00:30:53] Alex Volkov: 4x speedup up to batch sizes of 16 to 32 tokens. And they came out pretty much the same day yesterday on January 17th. So I'm going to add this in the show notes. So Marlin is also like an exciting optimization. And Nostia, I fully agree with you where we see these breakthroughs or collections of method that suddenly are finally collected in the same way.

[00:31:11] Alex Volkov: A bunch of papers that haven't, released code as well or haven't played with different things. And it's very exciting to see them Keep coming out, we're only at the beginning of this year. And I think to the second point that you just mentioned, with agent frameworks Specifically, RAG, Retrieval Augmented Generation this benefit is significant like you said, because the short strings back and forth, these agents communicate with each other.

[00:31:34] Alex Volkov: Last week we've talked with one such author from Cru AI, Cru specifically is an orchestration of different agents that do different tasks and coordinate and talk to each other and improving inference there. Many of them run on GPT 4 and I haven't fully gotten into how to do this yet, but SGLang also say that they're like LLM programming can actually work with various backends.

[00:31:55] Alex Volkov: So OpenAI as well and Tropic and Gemini and local models. That's very interesting if they actually improve OpenAI inference in Python. But DSPY RAG, so RAG on DSPYs from Omar Khattab is definitely mentioned in the SGLANG report. I know I'm throwing like a lot of a lot of acronyms at you guys.

[00:32:14] Alex Volkov: So SGLANG is the stuff we talk about as the That's the new language from LMCS org that speeds up some stuff. DSPY I haven't talked about yet, so we'll cover but one of the tasks on, on, on DSPY's RAG, so retrieval is mentioned that it gets like a significant boost. Like Nissen and Austin said, not necessarily for longer context prompts.

[00:32:35] Alex Volkov: 30, 000 tokens for summarization, maybe this technique that caches a bunch of. Stuff between calls is not going to be super helpful, but for fast execution of multiple things is definitely significant 5x. And like I think Lyman said, it's only the beginning of optimization cycles that we see, and it's quite exciting to to see them come out.

[00:32:56] Alex Volkov: I think we've covered two optimization techniques, SGLang, and then Marlin as well. I'll put a link to the show notes as well.

[00:33:03] NeuralMagic, compressing models with sparcification

[00:33:03] Alex Volkov: And I think now it's time to move to Yeah, one, one, one thing that we're going to chat about is neuromagic and I definitely focus on stage. Feel free to talk about neuromagic because I saw [00:33:20] somebody told me it's cool, but I have no idea how to even simplify this.

[00:33:23] Alex Volkov: So if you want us and you want to take a lead on this one, definitely feel free.

[00:33:28] Alignment Lab: Okay Neural Magic. This is actually the first conversation I think that me and LDJ both geeked out really hard on we were talking, because we were both the only people the other person knew who even knew about this company. Neuromagic has been making miracles in the corner for years.

[00:33:44] Alignment Lab: I first got interested in them because they had made a BERT model that was initially, it was nearly like I think a gig on your computer to run and, it spoke English perfectly well and all this other stuff. And they had compressed it to the point that the full model completely On your computer was like 15 megabytes and it, and what blew my mind was like, how does that even know English?

[00:34:06] Alignment Lab: And it's it was at like 96 percent the original accuracy, despite all of that. They specialize in these like optimization and compression techniques. And so what they do typically is they have a stack, which they wrote a paper about a while ago, which I'll post in the comments here.

[00:34:22] Alignment Lab: It's called Overt Surgeon, which is basically a process in which they have a teacher model. In a student model, in the student model they use distillation in the the more traditional sense than I think it's more commonly used now, where you're just training on a model's output, and they use the actual logits during they basically load both models in during the training run, and train the smaller model to behave like the larger model, and while they're doing that, they're also pruning it, which is, Essentially, you reduce the weights that are not getting used during training to zero, which lets your computer not have to calculate them, so it moves much faster.

[00:34:58] Alignment Lab: And then they also quantize, which is where you reduce the accuracy. Basically, without getting too technical, you're literally summarizing the parameters of the model, such that it's literally a smaller file. And they do this all at once, which takes the larger model, And compresses it into the student model that's starting out smaller, and then they're quantizing the student model and pruning it, so it's both running faster and literally getting smaller, and they can, as far as I'm aware, there's nobody who's even coming close as far as being able to compress a model so much and recently I think about two months ago we first saw that they're integrating transformers with Sparsify Alpha, which is now just out and it's called Sparsify on the GitHub.

[00:35:43] Alignment Lab: Totally check it out. You can make a tiny llama and do all that stuff to it and make it microscopic. It's amazing. And

[00:35:49] Alex Volkov: here, Austin, just real quick. So we've been talking about quantization for folks who are like not following the space look super closely. Let's say there's different quantization techniques in, and some of them create like small files, but the performance or like the accuracy, is getting lowered.

[00:36:03] Alex Volkov: How is Sparsification different from quantization, at least on the basic level. Are they compatible? Will they be used could you use both of them on the same file? What is this thing, sparsification?

[00:36:15] Alignment Lab: so in reality, probably if it were like more accessible of a tool, we would all likely just be doing both every single training run. But since there's always new quantization techniques, it doesn't make sense to. But with sparsification, the specific difference is rather than taking the same model and reducing the accuracy of its, the calculations, but making it smaller, the model's staying the same size physically on your drive, but you're reducing the weights that aren't getting used to to a zero value.

[00:36:50] Alignment Lab: And what that does is just means your, your GPU just has to do less calculations for the model to do inference and it makes it just much faster.

[00:36:59] Alex Volkov: All

[00:36:59] Nisten Tahiraj: Also, we for the next Baklava version, Neural Magic did make a A clip model for us. So shout out to them. They were able to cut down the size by from about four times smaller.

[00:37:14] Nisten Tahiraj: So we'll we'll have that out soon. And yeah, also for anybody else that. wants to learn about sparsity, just look up Nir Shavit on on YouTube. N I R S H A V I T. He's the OG MIT professor that pioneered sparsity and has a lot of videos out, and Neuromagic is his company. And yeah, it's looking really promising in the future because they can optimize at a deep level for CPU inference.

[00:37:45] Nisten Tahiraj: And it's not necessarily just quantization, it's also They are reducing the amount of unused weights. So yeah, expect to see a lot more stuff about sparsity from the GPU poor side of the spectrum, , because that's where the benefits are yet to be read.

[00:38:02] Nisten Tahiraj: Anyway, shout out to Neural magic as well.

[00:38:04] Alex Volkov: shout out to Neer Shovit and Neural Magic, it looks cool, and they just got into sparsifying fine tuned models as well, I think they sparsified some new models, and I don't know if they got to open chat yet, but I think some folks are waiting for PHY sparsification, definitely. The area of smaller models running on smaller hardware is advancing super, super fast.

[00:38:26] Star Coder from Stability AI - 3B coding model bearing CodeLLama

[00:38:26] Alex Volkov: Let's move on, folks, because we've been in the open source area for quite a while, and then we also need to get to our to the end of our conversations here and start doing deep dives. So StarCoder was released from Stability. A brief review here is a 3 billion parameter language model.

[00:38:41] Alex Volkov: From Stability AI it does code completion and obviously it runs offline cause it's a small model and you can run it. They claim it can run on MacBook Airs as well. And they say something like without GPU. Interesting. Accurate completion across 18 languages at level comparable to models twice their size.

[00:38:57] Alex Volkov: This is a Code Llama. Interesting comparison to Code Llama at this point, because we've seen a bunch of other models already beat, I think, Code Llama on different metrics. But people still compare themselves to the big dog. And it's very interesting. They use the multi stage process, pre training in natural language.

[00:39:15] Alex Volkov: fine tuning on code datasets to improve programming language performance. And it supports fill in the middle and expanded contact sizes compared to previous versions of stable coder. And I think, oh yeah the stable diffusion now has like a commercial membership plan because everybody's thinking about, okay how was.

[00:39:33] Alex Volkov: Table going to make money. So they have this membership where you can use their models. So it's not like fully open source. I think you can use this models commercially if you participate in this membership, otherwise you can use them for research. So stable quarter, check it out. I think it's new on, on hug and face.

[00:39:48] Alex Volkov: I think from today I believe,

[00:39:50] Discussion on Neural Beagle 7B & Model merging

[00:39:50] Alex Volkov: And I think the last thing that I want to chat about in open source just briefly is Neural Beagle 7B from Maxim who's in the audience and is going to come up hopefully in the interview in a few.

[00:39:59] Alex Volkov: Minutes, I want to say maybe 20 minutes, Maxim. Neural Beagle back when I added this to my notes, was the top performing 7 billion parameter fine tune in, in, in open source LLM leaderboard. It's no longer the top performing, it was definitely number 4, at least.

[00:40:14] Alex Volkov: And it's a merge plus a DPO, that's what I saw from Maxim, a merge of Actually interesting what it's a merge of, so let's go into the model card and check this out.

[00:40:24] Alex Volkov: But Maxim looks like have a bunch of models and Neural Beagle, the, this Neural Beagle 14, 7 billion parameters has an average of 60 on the, all the scores, 46 on AGI eval. And yeah, it's one of the top performing models and it's a merge of different things. And it already has a demo space that I'll link in the show notes as well.

[00:40:43] Insights on Lazy Merge Kit

[00:40:43] Alex Volkov: Yeah, it uses Lazy Merge Kit, which is a collab that Maxim also we're going to chat about and figure out what this means, what this merging thing means but definitely, I think that this model triggered one of the Nathan's in AI that says, Hey, I wanted to ignore this merge business for a while, but I guess I can't anymore because, merges is not to be ignored at this point.

[00:41:04] Alex Volkov: And this is a merge of the Wunna And distilled Markoro. Slurp. So which is also a merge. So if you guys hear me and you're like confused, like what are all these things mean? Hopefully we'll be able to clarify this one. Maxim. Maxim also had a tweet where there's now a collab where you can take a model like this and basically map out the genealogy of these models.

[00:41:25] Alex Volkov: What is based on what? And it's quite cool to see. And what else should I say about this model? I think that's pretty much it. It's very performative. I actually haven't had the chance to use this, but it's right up there and it's a merge model. There is, there's the [00:41:40] checkbox, like we said, in the open LLM leaderboards.

[00:41:42] Alex Volkov: If you don't want for some reason to see the merge models and we'll see like more trained models, you will uncheck that. But definitely the merge models are competing for the top of the LLM leaderboards right now. Haven't seen a lot of them on the LMCs arena, so it's going to be interesting to see how they treat the merge models.

[00:42:02] Alex Volkov: And I think that's most on open source, and we've given this corner almost 40 minutes, so I think it's time to move on a little bit here, folks. So I'll, yeah, I don't have breaking news here, so I'll just do this, a small transition so I can take a breath, haha.

[00:42:17] Sounds: Namaskaram to all of

[00:42:22] Deep mind to Alpha Geometry

[00:42:22] Alex Volkov: LMs and APIs, and I think the biggest player in this whole, Aparigraha, Niyama, Shaucha, Satya, Ashtanga, Yama, Ashtanga, Niyama Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, Ashtanga, is deep mind, deep mind released, A Nature article, which they always do, they always publish in Nature, this time the link to Nature article didn't really work but hopefully they fix it by now, and they released Alpha Geometry, so they released like a bunch of stuff, Alpha Fold, if you remember, Alpha Go Alpha Zero, they had a model that, that self trains to play anything, not only chess, or, not only Go, and now they've released Alpha Geometry, that solves geometry, almost a gold medal Level at the at the Olympiad level, so they have this this how should I say, this nice chart that says the previous state of the art on this Olympia Gold Medallist Standard gotten to ten problem solved there's like time limits. I'm not sure what the time limits are actually are. I don't have it in my notes. But you have to solve these like very like difficult geometry levels. Folks compete for the gold medals in this Olympiad. And alpha geometry now comes very close to the gold medalist standard.

[00:43:29] Alex Volkov: So the gold medalist is answers 25.9 problems solved, and alpha geometry now answers 25, and they claim that the previous state of the art answered 10, just 10. So they more than doubled and they're getting close to the Olympiad. I think I saw like a tweet from Nat Friedman or somebody. That says they would offer a 1, 000, 000 prize for somebody who solves the Geometry Olympiad at the Golden Medal, and now we're getting there.

[00:43:53] Alex Volkov: They use the newer symbolic approach and they combine all of them with a symbolic deduction engine to leverage the strength of both. Which some folks compare to thinking fast and slow, where you have system 1, system 2 thinking, or at least the outline system 1, system 2 thinking.

[00:44:09] Alex Volkov: In this case, this does actually help. They have the neuro symbolic approach. I think they use this, the neuro symbolic approach. I don't think I've seen this before. And I think the most interesting parts It was trained on over a hundred million synthetic geometry examples generated from one billion random diagrams.

[00:44:27] Alex Volkov: Completely, solely synthetic geometry examples. This whole data set for training of this model that beats Humans at Geometry, which was previously very difficult, is fully synthetic. And I think that's super cool. We only began this year, but definitely this is going to be the year where full synthetic datasets are going to rule.

[00:44:49] Alex Volkov: And Yeah. Opinions, folks here on stage. Have you read about this? What's interesting to you? I would love to hear folks kind of chime in on this as well, because I think it's like super cool and kudos for them to releasing this. Also, I saw somebody said, I think Bindu said that they released this open source, but I haven't seen anything.

[00:45:06] Alex Volkov: Definitely Luigi Go and then Nistan.

[00:45:09] Luigi Daniele: Yeah it's funny that you brought up Nat Friedman having that bet up. Because I remember that too, and now I'm thinking, I wonder if he'd be willing to give up like the million dollars or whatever the money is to DeepMind. Ha

[00:45:20] Luigi Daniele: was done by Google DeepMind, so that'd be funny.

[00:45:25] Nisten Tahiraj: How has Google not discovered AGI yet and fallen so behind?

[00:45:30] Nisten Tahiraj: This almost feels like an internal illness or something. Something's going on. Because yeah.

[00:45:40] Alignment Lab: I don't think that Google needs to compete is the thing. I just don't think they're incentivized to release anything into the space because they don't have to. There's really not anything here except money to lose for them.

[00:45:51] Alignment Lab: They already have all the data and stuff. Yeah, and back to the geometry problems, I can't wait to test this, if they release it, as to how it does when given really random, very long numbers. If it still solves the problem, then that, that will be extremely impressive. And yeah, I've done those Math Olympias with geometry questions and they're not easy at all.

[00:46:18] Alignment Lab: You have to picture stuff in 3D. 4D and whatever in your head. They're very tricky problems. So yeah this is pretty huge. That's all. Yeah.

[00:46:26] Alex Volkov: Quite, quite huge and kudos on them. Umesh, I think you actually found the source, right? I just

[00:46:32] Umesh Rajiani: Yeah so there is GitHub repo on Google DeepMind. So if you go to Google DeepMind on GitHub and then alpha geometry, you can find the code repo for that. So Nistan, if you want to test it out, it's there for you. So I'm taking your

[00:46:47] Alex Volkov: hark on this just like for a little bit. Did Google release code for us finally? Did Google like open source something? Welcome back, Google.

[00:46:54] Umesh Rajiani: yeah, so this is like first release kind of thing, coming out of Google. So it's going to be, yeah, it is quite quite interesting.

[00:47:01] Alex Volkov: Definitely moves us towards like more generalist

[00:47:04] Bluetooth: I'll have it up in a sec.

[00:47:05] Alex Volkov: Yeah, listen, please put this and we'll add this to the show notes as well. Definitely the question, how have they not solved AGI yet? Solving math at the Olympiad level seems like moving us forward, definitely. This neuro symbolic approach where they combine language models with a symbolic deduction engine, which I have no idea what symbolic deduction means in this case.

[00:47:24] Alex Volkov: But leveraging strength of both, this seems like going towards the right path. We've seen, I think Similar things with vision as well, where you combine kind of vision heads into one model they can understand. I don't think this model was multi modal at all. Doesn't look like, but maybe I'm wrong here.

[00:47:42] Alex Volkov: And I think Yeah, the solutions for this thing is verifiable by machines. I saw this one tweet that will go down in history. Somebody said, computers has always been good for calculations. So I don't understand the big deal there, here. And I think I think it's really funny to like, keep this tweet behind the scenes.

[00:48:04] Alex Volkov: Alright, so shout out to DeepMind for this fairly incredible release. Hopefully some of the techniques they used will be then used by folks in other areas as well to get us AIs that are significantly better at the geometry and different things. Oh yeah, Umesh, just before, before we continue, you want to talk about this NeuroSymbolic thing? Cause we've talked about this. I think Daniel Jeffries talked about this last time we've talked about Rabbit.

[00:48:27] Alex Volkov: If you guys remember, this was at the end of the last space and we've talked about Rabbit LAM, Large Action Model. And Umesh, you just mentioned something that they also use NeuroSymbolic to an extent, right?

[00:48:39] Umesh Rajiani: Yeah, so the LAM Large Action Model, basically based on Neuro Symbolic Programming for when, specifically when they are talking about training the model from the actions that you're going to perform is basically they are encoding Neuro Symbolic Programming to train the model or capture the actions, basically.

[00:48:55] Umesh Rajiani: So that's what we're trying to do. Namaste. In theory, they are saying we have to see what comes out in practice.

[00:48:59] Alex Volkov: Yeah, and based at least on their examples, it looks like very compelling and potentially like being able to solve a bunch of stuff or like to remember based on your actions. So neuro symbolic not a new approach. I apologize. I will edit this. Definitely Rabbit said this, you're right and hopefully we're going to get to see this lamb thing.

[00:49:19] Alex Volkov: So back to OpenAI as elections are happening right now and everybody was fearing like, Hey, what's going to happen with deepfakes, et cetera. OpenAI released their guidelines toward election, as they prepare for elections, obviously, they're aware that they're happening. And I think the few interesting things there that they're taking steps to prevent their tools like Dalai and Shajipati from being abused.

[00:49:38] Alex Volkov: I don't know. We have open source, so I don't know if folks will go to the GPT 4 to generate let's say, propaganda. But DALI, for example, starts to integrate some cryptography to their images, which is very interesting. Cryptography solutions, which, again, In case you download the actual file and then send it, could be a thing.

[00:49:58] Alex Volkov: But I don't know if [00:50:00] somebody takes a screenshot of a Dalit generation, if that will apply at all. There are definitely like usage policies for like stuff like Chajapati enforcing limits on political campaigning and impersonating candidates and discouraging voting. And then they want to run ahead of what happened with Facebook and Cambridge Analytica, and like all these things they want to get ahead of us which, it makes sense.

[00:50:18] Alex Volkov: So the technology they use to detect images were generated by DALI I haven't seen any release on them that says, Hey, we'll build a tool for you to actually identify if those are generated images or not. It's going to be interesting because like with LLM writing all of these tools that you use to like dump AI text in there, they're all can be obscured with another LLM.

[00:50:38] Alex Volkov: I don't know if it's a futile attempt, but definitely a worthwhile one. And at least in the basic UI, I think blocking some attempts of destabilizing democracy, I think it's a good idea. And I think that's mostly it. I think there's one different mention that somehow silently they removed where the terms and conditions thing where their outputs is not to be used for war or weapon developing.

[00:51:04] Alex Volkov: And I think they removed that and I think they're also like signed something with Department of Defense, but I think that's all for OpenAI.

[00:51:11] Microsoft announces CoPilot pro

[00:51:11] Alex Volkov: And then I wanted to mention about Microsoft and Umesh, definitely feel free to chime in here as well, because the underlines the benefit for open source, but quickly Microsoft announced Copilot, we've talked about Copilot, the kind of previously BingChat, Copilot everywhere.

[00:51:25] Alex Volkov: So they've announced like different paid plans for Copilot Pro, 20 bucks a month premium, and then it does. Enhanced image creation, where we don't even get We don't even get in, in, in Dali like by default, and it's now generally available for small businesses with no user minimum. So if you guys remember, we've talked about Copilot before when Microsoft announced it for large enterprises it integrates into Microsoft 365 everywhere.

[00:51:49] Alex Volkov: And now the Copilots are also open for smaller businesses. And soon there's going to be like this Copilot Studio to build custom GPTs. Very cool for small businesses. We'll see how much actually folks will use this. And there's also some Microsoft Saga that they've changed some stuff in their pipeline.

[00:52:04] Corporate Drama - Microsoft Azure changing moderation flows and breaking products

[00:52:04] Alex Volkov: So Umesh, you mentioned this in the beginning. We'd love to hear from you what's been going on as you guys are big Azure users through Microsoft.

[00:52:11] Umesh Rajiani: Ooh happened

[00:52:15] Umesh Rajiani: day before yesterday. Actually, we got a call from one of our clients, which is one of the, one of a very big financial institution. And we have a deterministic pipeline, which was constructed using Azure studio, in fact. And we work together with very core Microsoft team actually to make sure that it is right.

[00:52:36] Umesh Rajiani: properly deterministic because there are some legal implications and everything. And and then the tool started failing and because we had some function calling, which would actually go into the knowledge base of the company. And that function calling was was getting extracted, getting triggered using what you call the deterministic intent from user's prompts, basically.

[00:52:56] Umesh Rajiani: And and that entire function calling was failing. Now, we carried out all types of work and everything it was very frantic because it was a front end tool and it started having some impact. And it was, remember, it was working for six months. So it's it worked without any problems for six months and suddenly it just stops working.

[00:53:14] Umesh Rajiani: And the reason was that there were two words that were in the definition of The tool, so that definition of tool was actually informing the pipeline what the tool is all about and that's how the tool was getting invoked and those two words basically were getting flagged into The OpenAI API.

[00:53:32] Umesh Rajiani: So we're basically Azure OpenAI API, not OpenAI's direct API. We are routing it through Azure and it's a separate separate instance of of GPT 4 and there are separate guidelines. They mimic some of the guidelines that are there in OpenAI, but Microsoft has its own guidelines and they change the guidelines without actually informing the clients. That basically triggered. Yeah. So we literally we literally had legal people and literally had fight. It was an open fight, literally, with Microsoft. If you were in that room, you would have you would have seen. It was really bad. And and then eventually there were talks about cases and stuff like that.

[00:54:08] Umesh Rajiani: And eventually, basically actually this company is actually modifying the contract with Microsoft. Where Microsoft will be liable to inform the company before they change any kind of guidelines. And you know what happened after that is, is the beauty because in the beginning of my startup, like beginning of the year, we implemented some solutions where we have a direct contract with Microsoft And we have implemented solution on the backing of those contracts.

[00:54:34] Umesh Rajiani: So in last two days, actually, I've gone back to those clients with whom we have implemented solutions so that they have a direct contract with Microsoft, because we don't want to be a party involved as far as the SLAs are concerned, because this is very dangerous if you're developing solutions for.

[00:54:49] Umesh Rajiani: For people and and if the core solution through which you are driving the entire application pipeline is getting changed without any kind of data contract backing, so to say. Yeah, this is a great learning for us and I've been always a proponent of. Open source solutions, and I think this has given one more kind of a booster to us because now we can go back to the new clients and say, Hey, guys if possible, if we give you the kind of solution that you're looking for, then let's go to open source solution rather than going for a closed source solution.

[00:55:20] Umesh Rajiani: So

[00:55:20] Alex Volkov: And this is like a huge, yeah, a huge like reason why, right? Getting, it's very interesting, like in this area we mentioned, definitely feel free to chime in on this a little bit more. The outputs of LLMs are usually non deterministic. And so this has to be built into understanding when you build tools on top of this.

[00:55:36] Alex Volkov: But this is not that. This is them adding not like a different model or something like a different that you can switch. They're adding something in between or some like policy thing without announcing this to the customers. And supposedly if you go to Azure instead of OpenAI, for example, you would go for the most stability as underlined by the fact that when OpenAI had downtime after Dev Day, Microsoft Azure, GPT for like endpoints, they were all fine.

[00:56:02] Alex Volkov: They were all green, right? So supposedly you would go for the stability and kind of the kind of the corporate backing. There's also like different ISO things and HIPAA compliances, like all these things that Microsoft Azure like proposes on top of OpenAI. But here we have a case where like underlines how.

[00:56:17] Alex Volkov: How important open models that you host yourself are, even if you host them, like maybe on Azure as well, because then nobody can change the moderation endpoints for you and suddenly decide that a few words in your prompt are not, to be used anymore.

[00:56:32] Umesh Rajiani: Yeah, but Alex this had nothing to do with the prompt, actually. It was actually the definition of the function that was there. And the key is like I would draw an analogy to what you call the data contracts. I don't know how many people are aware of data contracts, but when you have.

[00:56:47] Umesh Rajiani: Ownership of data within a very large organization, let's say 20, 000, 30, 000 people up you have data contracts where the data originates from a particular source and some other division is using that data. So you have a contract between those two and that data contract details the data definitions which are there and the contract sign, the signatory of the contract is responsible to ensure that if they change any kind of data structure or data definition.

[00:57:14] Umesh Rajiani: Then the receiver of the data or the client of the data contract is supposed to be informed. That is a part of your data contract. And that's how these large organizations function. And what we need is that kind of a framework where you have a data contract with the service provider.

[00:57:30] Umesh Rajiani: So even if you're going with an open source solution, and if your open source solution is hosted by someone, Then you need to have that kind of a contract in place. So it's not just that open source solution is a solution for everything. It's about the person who is providing the inference. So if you are controlling the inference, then you are secure because you are not going to make the changes without, understanding the repercussions of those changes.

[00:57:52] Umesh Rajiani: But if you are let's say hosting open source model on Amazon Bedrock, for example. And if they have a system prompt that lies in front of your prompt that goes to the the model, then you have to make sure that Amazon adheres to their responsibility in terms of giving you the required inference.

[00:58:12] Alex Volkov: Absolutely. Thanks for giving us the, first of all, like it's, it sucks that it happens and hopefully now Microsoft, like you said, they [00:58:20] changed their their approach here. Aniston, go ahead if you want to follow up.

[00:58:26] Nisten Tahiraj: Yeah. So for us, this has been amazing. I already have clients lining up to pay for the Baclav API. So I'll just say that first before it's even out. However It is extremely unfortunate for those that built, let's say, apps in a hospital or for a therapist because now those kinds of applications just had a moderation engine added, and they added apparently for their safety, and now whoever was relying on these applications, now they just stop working.

[00:59:02] Nisten Tahiraj: Out of nowhere, and this is an extremely immature thing to do this is something you expect from like a random startup with kids, not from freaking Microsoft, and it is pretty worrisome that this safety hysteria has gotten to the point where You're literally just breaking medical applications in production without modifying, without notifying people.

[00:59:27] Nisten Tahiraj: That's just, you lost people's trust now. You're not going to gain that back for a couple of years. And I hope they realize and don't do this again. Don't break production and make changes. To people in Prad that are relying on this for like SOC 2 or as in the case of UMass that have signed service level agreements.

[00:59:49] Nisten Tahiraj: Because now those people lose all their money if they don't, if they don't provide the service. And it's really bad. That's all I have to say. It's pretty bad.

[00:59:58] Alex Volkov: Yep. Very bad look from Microsoft. Even I think I remember like not entirely OpenAI, when they talked about Sunsetting some models and there was like a developer outcry that said, Hey, like we use those, we haven't had time to change how we work with different prompts, et cetera, for the newer models.

[01:00:15] Alex Volkov: And so OpenAI actually went back and said, Hey, we heard you and we're going to release we're going to deprecate deprecation is going to be pre announced in advance. It's going to be way longer Omesh let's yeah, let's go ahead.

[01:00:27] Umesh Rajiani: Yeah, very quickly I think you have raised a very valid point, Alex, that I think all the models that they actually put out of service, they actually should make them open source. I think that's the best solution.

[01:00:39] Alex Volkov: Nah, I wish this was the case. We're still waiting for potentially like open source GPT 2. 5. We haven't seen any open sources from OpenAI for a while. Besides like some GitHub code, I agree with you. There should be a way for folks to keep doing this, the same exact thing they're doing.

[01:00:52] Alex Volkov: I don't know, in my example, I use Whisper, no matter like what their API really says, what it's like, what they deem inappropriate to translate, the Whisper that I use is hosted and it will be the same version until I decide basically and test everything. All right, folks, we're moving forward, I think, just quickly.

[01:01:10] Alex Volkov: There's not a lot of stuff in the vision area. I will mention briefly we've been here for more than an hour. So I'll definitely like recap the space a little bit. If you're joining, let me just play the music and then I'll recap and then we'll get into the interview. So with with Hour 15, you're listening to Thursday Eye. Those of you who just joined us, welcome. If you haven't been here before, this is a weekly space all about AI, open source, as our friend of the pod, Jan, just tweeted out, everybody and everybody in LLM space and open source is in here, and very great to see.

[01:01:45] Alex Volkov: We've covered open source stuff, we've covered corporate drama right now, and then we're moving on to an interview. Thank you.

[01:01:53] This weeks Buzz from Weights & Biases

[01:01:53] Alex Volkov: And then we're going to talk about AI, art, and diffusion, if we're going to have time at the end of this. There's a brief mention that I want to say, but basically, let me just reintroduce myself.

[01:02:01] Alex Volkov: My name is Alex Volkov. I'm the AI Evangelist with Weights Biases. And we have a small segment here for Weights Biases that I want to choose to just bring. I just came back a few days ago from San Francisco Hackathon, the WeHub sponsor with TogetherAI and LengChain. It was a pretty cool hackathon.

[01:02:20] Alex Volkov: It was very brief, like a few hours with AGI House. But basically the theme was RAG versus FineTune. And I think the theme was versus, and I just promised I'll bring some learnings from this. So there's a bunch of projects that did different things. They used Together's endpoint for FineTune.

[01:02:35] Alex Volkov: So if you can FineTune. On your models and your GPUs that's one thing, but for many of the AI engineers, that's very difficult to do. So there's a bunch of startups together as one that they offer like very simple fine tuning. I'll definitely add my my Link to the show notes, to the presentation I gave there, which talks about how easy it is to fine tune using their endpoints.

[01:02:56] Alex Volkov: And the folks that won the hackathon, some folks won different prizes, basically used both Reg and FineTune. And it looks like also there was a paper released afterwards from some folks trying to identify what's better. Is it just doing RAG on top of Hindu models or just doing basic RAG?

[01:03:13] Alex Volkov: And I don't think we have a clear answer yet. Definitely this hackathon wasn't the end all of all answers. However it does look like doing RAG on top of a fine tuned model improves just a little bit on top of just basic RAG. And it looks like RAG wins on top of just a regular fine tuned for information retrieval tasks as well.

[01:03:30] Alex Volkov: So definitely do not skip RAG. And I think from the open source perspective, which we love here on Thursday Eye getting more RAG kind of Related models is definitely going to happen. I think we saw some from John Durbin. I think I saw Technium. You mentioned something about like function calling.

[01:03:47] Alex Volkov: Datasets are coming to, to, from news as well. So definitely that area is still to be explored. But it looks like the combination of FineTune and RAG wins just a little bit on top of just basic RAG. I think this is the outcome of that hackathon. Next week in this corner of 1B is going to be an interview with Jason.

[01:04:06] Alex Volkov: Stay tuned for that.

[01:04:07] BREAKING NEWS - Meta announces LLama 3 is training and will be pen source

[01:04:07] Alex Volkov: I think now we have, and many folks have been DMing me because right now we have breaking news. Breaking news actually happening right now.

[01:04:17] Sounds: AI breaking news coming at you only on Thursday ice.

[01:04:27] Alex Volkov: You know I love to use this sound. You know I love to use this sound, everyone. We have some updates from BigZuck. you guys see this because it's over on threads. And I don't know how many of us are on threads. I definitely know that I barely go there. We have some updates from BigZuck specifically around Training Lama 3.

[01:04:43] Alex Volkov: There's like key updates about the long term vision. I think the summary there is They have an insane amount of GPUs this year. So like literally he says at the end of this year, we'll have three, around 350, 000 NVIDIA H100s. I'm going to repeat this slowly for the people in the back. 350, 000 NVIDIA H100s and overall 600, 000 H100s or equivalents of compute if you include other GPUs.

[01:05:13] Alex Volkov: You remember those hats that people wear, like GPU poor, GPU rich hats? I think Zack can stack the GPU rich hats, like one on top of the other and it still won't be enough because 600, 000 H100 compute is just like ridiculous. And he talks about. Two major parts of their vision, AI and Metaverse are connected.

[01:05:32] Alex Volkov: I love how like it was Metaverse, and then suddenly AI started being a thing and now oh, they're connected. I definitely am expecting AI to exist in some form of virtual slash world, et cetera. But definitely he talks about Lama 3. And Lama 3 is coming. They're currently training it per BigZakh.

[01:05:48] Alex Volkov: We know that's coming or like we at least expected this, but I think now is like more of a confirmation. And I'm very excited about Lama 3. I will just mention that it's not been a year since Lama 1 yet. So we're in January Lama was released in like around February 12th, 13th or so.

[01:06:06] Alex Volkov: And it's not half, like it hasn't been a year yet. And here we are like training the third model on top of Lama. We've had just an incredible amount of like innovation on top of it. So definitely expecting and we're obviously going to cover this as much as possible. So this is I think most of it.

[01:06:23] Alex Volkov: Oh and this last thing that he added, Zak has added and I think it's Adding to Thursday as well where we have to start talk about hardware is that he says I think lots of people will talk to A. I. s frequently through the day using smart glasses like what we're building with Ray Ban Meta.

[01:06:38] Alex Volkov: And I think we've [01:06:40] talked about their smart glasses that they're like multi modal glasses. They have a camera built in them. You can press a button and actually pass the image into the LLM. They're making improvements in speed as well. I will say just like an additional one thing we've talked how Meta is adding a bunch of AI into every chat and nobody like necessarily used them.

[01:06:58] Alex Volkov: Recently, a friend of mine, maybe because, I'm an AI evangelist, so he felt free to do this in our chats. He just added an AI bot to our chat. Literally, just like my DM with a friend who has no, nothing about AI, like it's not part of his world. He does something else. Recently, he's Hey, let me add this thing.

[01:07:14] Alex Volkov: So Meta is definitely letting folks experiment with AI more than some other places. And he just added in the AI to our chat. It was super cool. So here's an update from Zack BigZack. Allama3 is training and then they have a lot of GPUs. They're like super GPU rich and, hopefully we'll get the benefit.

[01:07:30] Alex Volkov: Go ahead, Nissan. Yeah,

[01:07:36] Nisten Tahiraj: H100s? Yeah, they're going to need that if they're going to have visual stuff from people's glasses. But it's an insane amount. That's all. Yeah, I just ran some quick calculations. I got roughly similar numbers to what Nishtan just said. And if I'm doing my math I'm running just some numbers based off the alleged GPT 4 leaks of the amount of GPU hours that it might take, let's say if they used all those meta GPUs.

[01:08:08] Nisten Tahiraj: It's do a GPT 4 level model. I'm getting numbers it would take less than a week pretty much to train, yeah, this is an insane amount of GPUs for people that, don't have good references for this. Yeah.

[01:08:18] Alex Volkov: I think it's insane enough to maybe open a new category like on top of GPU rich. It's just quite incredible and like hopefully they're committed to the open source of this in Lemma 3. Omesh, you had a comment as well?

[01:08:32] Umesh Rajiani: Yeah, what if Lama 3 is going to be multi modal? Then they will need those GPUs.

[01:08:37] Alex Volkov: I'm really hoping it will. Like they're training the models, like multimodality is something they talked about. It's time. To move towards the LMM world and multimodality, and they will need all those GPUs to crank out. The vision part of this hopefully multimodal in other areas reminder meta has released like bull a bunch of attempts at multimodality in other areas, not only image.

[01:08:59] Alex Volkov: IMU motion units and they've talked about F-F-M-R-I signals they've talked about, like incredible stuff. But definitely modality, other modality like sounds like audio. Live video would be super cool, like I think this year is the year of live video, so not only, hopefully not only vision, and if it's vision, then hopefully it's like a live video.

[01:09:18] Alex Volkov: Alright folks, we're coming up on two hours,

[01:09:20] Alex Volkov: and with that, I think this is the summary of today's Thursday Eye. Thank you everyone for joining. If you haven't subscribed yet, definitely feel free to subscribe at ThursdayEye. News. I appreciate everyone's time and attention here. Thank you so much for the Co hosts and guests for today's pod and shallow with everyone.

[01:09:36] Alex Volkov: And I have to end this on the very happy note of the alchemy thing, because the one thing that came out from the conversation with with Maxim, who merges and Nistan and everything is that a lot of this is alchemy and a lot of this is like trying to see how things work when you combine and not continue to train models, they still perform better.

[01:09:55] Alex Volkov: So I have to end on this very happy tune, which will represent the alchemy that we're all doing. And we love it. Thank you everyone for joining this Thursday. I will see you next week. Cheers. And we'll add this banger to the show notes as well. Bye everyone.

Hey everyone, we have an exciting interview today with Maxime Labonne.

Maxime is a senior Machine Learning Scientist at JPMorgan, the author of Hands on GNNs book and his own ML Blog, creator of LazyMergeKit (which we cover on the pod) and holds a PHD in Artificial Intelligence from the Institut Polytechnique de Paris.

Maxime has been mentioned on ThursdAI a couple of times before, as he released the first Phi mixture-of-experts, and has previously finetuned OpenHermes using DPO techniques which resulted in NeuralChat7B

For the past couple of months, following AI on X, it was hard not to see Maxime's efforts show up on the timeline, and one of the main reasons I invited Maxime to chat was the release of NeuralBeagle7B, which at the time of writing was the top performing 7B model on the LLM leaderboard, and was specifically a merge of a few models.

Model merging

Model merging has been around for a while but recently has been heating up, and Maxime has a lot to do with that, as he recently checked, and his wrapper on top of MergeKit by Charles Goddard (which is the library that put model merging into the mainstream) called LazyMergeKit was in charge of >50% of the merged models on HuggingFace hub leaderboard.

Maxime also authored a model merging blogpost on Hugging Face and wrote quite a few articles and shared code that helped others to put merged models out.

Modern day Alchemy

This blogpost is a great resource on what model merging actually does, so I won't go into depth of what the algorithms are, please refer to that if you want a deep dive, but in a nutshell, model merging is a technique to apply algorithms to the weights of a few models, even a few instances of the same model (like Mistral7B) and create a new model, that often performs better than the previous ones, without additional training!

Since this is algorithmic, it doesn't require beefy GPUs burning power to keep training or finetuning, and since the barrier of entry is very low, we get some cool and crazy results as you'll see below.

Yeah, quite crazy as it sounds, this method can also create models of non standard sizes, like 10B or 120B models, since it's slicing pieces of other models and stitching them together in new ways.

If you recall, we had a deep dive with Jon Durbin who released Bagel, and Jon specifically mentioned that he created Bagel (based on everything everywhere all at once) as a good base for merges, that will include all the prompt formats, you can read and listen to that episode here

This merge frenzy, made HuggingFace change the leaderboard, and add a checkbox that hides model merges, because they are flooding the leaderboard, and often, and require much smaller effort than actually pre-training or even finetuning a model

And quite often the top of the leaderboard was overrun with model merges like in this example of Bagel and it's merges by CloudYu (which are not the top ones but still in the top 10 as I write this)

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

On why it works?

Nisten summarized this pretty well in this now famous copypasta tweet and I've confirmed with Maxime that this is his current understanding as well, it's quite unclear why this seems to perform so well, but it of course doesn't stop the "folks who look for AI Waifus" to keep merging.

Following folks like Nathan Lambert from interconnects.ai to start paying attention even though he didn't want to! (Still waiting on your writeup Nathan!)

UPDATE: As of today Monday Jan 29th, Nathan Lambert just released a super comprehensive deep dive into merges, which you can read here 👇👏

YALL + Automated LLM Evaluation

Maxime as also worked on so many models of his own, that he built a convenient little tracking leaderboard to track their performance, which he called YALL, Yet Another LLM Leaderboard and it's on HuggingFace. You can see that NeuralBeagle is the top dog (sorry, I literally could not resist)

It uses the Nous evaluations, and Maxime has created an automation called LLM AutoEval that makes it really simple to run evaluations, which you can run in a Colab super easily.

LLM AutoEval is on Github.

Merge-aology!

Since chatting, Maxime has released a Colab and later a HuggingFace space that takes models names, and shows the genealogy, nay, Merge-aology of the models, which models it was merged from and it's pretty crazy how deep this rabbit hole goes, and crazier even still that these models perform very well after all of these lobotomies!

Try it out here: https://huggingface.co/spaces/mlabonne/model-family-tree

I really hope you enjoy this special deep dive, I definitely learned a BUNCH from this conversation with Maxime, and I'm very happy that he came on!

What A SHOW folks, I almost don't want to write anything in the newsletter to MAKE you listen haha but I will I know many of you don't like listening to be babble.

But if you chose one episode to listen to instead of just skimming the show-notes, make it this one.

We've had 2 deep dives, one into the exciting world of multi-modalilty, we chatted with the creator of Moondream1, Vik and the co-founders of Prophetic, Wes and Eric about their EEG/fMRI multimodal transformer (that's right!) and then we had a DEEP dive into the new Hourglass Diffusion Transformers with Tanishq from MedArc/Stability.

More than 1300 tuned in to the live show 🔥 and I've got some incredible feedback on the fly, which I cherish so if you have friends who don't already know about ThursdAI, why not share this with them as well?

TL;DR of all topics covered:

* Open Source LLMs

* Stability AI releases StableLM 1.6B params (X, Blog, HF)

* InternLM2-Math - SOTA on math LLMs (90% GPT4 perf.) (X, Demo, Github)

* MedArc analysis for best open source use for medical research finds Qwen-72 the best open source doctor (X)

* Big CO LLMs + APIs

* Google teases LUMIERE - incredibly powerful video generation (TTV and ITV) (X, Blog, ArXiv)

* 🤗 HuggingFace announces Google partnership (Announcement)

* OpenAi 2 new embeddings models, tweaks turbo models and cuts costs (My analysis, Announcement)

* Google to add 3 new AI features to Chrome (X, Blog)

* Vision & Video

* Adept Fuyu Heavy - Third in the world MultiModal while being 20x smaller than GPT4V, Gemini Ultra (X, Blog)

* FireLLaVa - First LLaVa model with commercial permissive license from fireworks (X, Blog, HF, DEMO)

* Vikhyatk releases Moondream1 - tiny 1.6B VLM trained on Phi 1 (X, Demo, HF)

* This weeks's buzz 🐝🪄 - What I learned in WandB this week

* New course announcement from Jason Liu & WandB - LLM Engineering: Structured Outputs (Course link)

* Voice & Audio

* Meta W2V-BERT - Speech encoder for low resource languages (announcement)

* 11 labs has dubbing studio (my dubbing test)

* AI Art & Diffusion & 3D

* Instant ID - zero shot face transfer diffusion model (Demo)

* 🔥 Hourglass Diffusion (HDiT) paper - High Resolution Image synthesis - (X, Blog, Paper, Github)

* Tools & Others

* Prophetic announces MORPHEUS-1, their EEG/fMRI multimodal ultrasonic transformer for Lucid Dream induction (Announcement)

* NSF announces NAIRR with partnership from all major government agencies & labs including, OAI, WandB (Blog)

* Runway adds multiple motion brushes for added creativity (X, How to)

Open Source LLMs

Stability releases StableLM 1.6B tiny LLM

Super super fast tiny model, I was able to run this in LMStudio that just released an update supporting it, punches above it's weight specifically on other languages like German/Spanish/French/Italian (beats Phi)

Has a very surprisingly decent MT-Bench score as well

License is not commercial per se, but a specific Stability AI membership

I was able to get above 120tok/sec with this model with LM-Studio and it was quite reasonable and honestly, it’s quite ridiculous how fast we’ve gotten to a point where we have an AI model that can weight less that 1GB and has this level of performance 🤯

Vision & Video & Multimodality

Tiny VLM Moonbeam1 (1.6B) performs really well (Demo)

New friend of the pod Vik Hyatk trained Moonbeam1, a tiny multimodal VLM with LLaVa on top of Phi 1 (not 2 cause.. issues) and while it's not commercially viable, it's really impressive in how fast and how quite good it is. Here's an example featuring two of my dear friends talking about startups, and you can see how impressive this TINY vision enabled model can understand this scene. This is not cherry picked, this is literally the first image I tried with and my first result.

The image features two men sitting in chairs, engaged in a conversation. One man is sitting on the left side of the image, while the other is on the right side. They are both looking at a laptop placed on a table in front of them. The laptop is open and displaying a presentation, possibly related to their discussion.

In the background, there is a TV mounted on the wall, and a cup can be seen placed on a surface nearby. The scene suggests a casual and collaborative environment where the two men are sharing ideas or discussing a topic.

Vik joined us on the pod to talk about why he didn't go with Phi-2, he also mentioned that Phi-1.5 was retroactively also MIT'd, it's license literally says MIT now on HF 👏 Great conversation, tune in for that at around 00:31:35

Adept is teasing FuYu Large - their CHONKY VLM

Adept previously released Persimmon, and then Fuyu VLM (which is a type of persimmon we see you adept) and now tease the release for Fuyu Heavy, a much bigger model that can compete or come close to GPT4V and GeminiUltra on MMMU and MMLU (text) while being 20x smaller approx.

While we don't yet get to play with this, they show some great promise in the benchmarks

⭐️ Performance: Excels at multimodal reasoning and matches/exceeds text-based benchmarks.❗️ Challenges Faced: Dealt with issues related to image data, model stability, and pre-training data scarcity.✅ Evaluations: Outperforms Gemini Pro on MMLU and MMMU benchmarks.AI Summary by Arc Browser (haha see how I cheated here? I sometimes do shortcut summaries using Arc Max, it's dope, try it) https://t.co/BZi6EKhS5R

Fireworks AI releases FireLLaVa - with a commercially available license

 FireLLaVA is the first commercially permissive open-source LLaVA model, a type of multi-modality model called a Vision-Language Model (VLM) that can understand both visual and textual inputs.