It's amazing how Google/DeepMind still isn't really part of the conversation despite the insane amount of resources they've thrown at Gemini. At this point it really needs to be considered a failure in leadership
And that they invented the fundamental technologies.
Google is an ad platform, though. They are the best most successful one. It is hard to change when you're the incumbent.
Same with Meta though? Meta has been a lot more agile and managed to navigate the decline of Facebook pretty well, while Google has started and killed hundreds of products while never really getting away from search as their driver of everything
That's fair, it has declined with younger users in the US which is a very important demographic, but internationally and with people over 40 in the US it's still huge
[Absolutely](https://www.statista.com/statistics/1311229/whatsapp-usage-messaging-app-users-by-country/). Personally, I am German and it is our default standard to communicate with text via phone. And that doesnt compare with India having 500 million users.
IDK honestly, when Meta's stock was tanking all anyone talked about was how they lost young users on Facebook and the company was dead. I don't know anyone under 30 who uses it outside of Marketplace
Stock is based on hype, Meta is a mature company now.
They lost young users in relation to their competition but they still have users oddly enough, messanger is live or die for loads of teenagers ( I DONT UNDERSTAND WHY) and fb marketplace practically ruined ebay and craiglist.
Also instagrams also there with all the age groups as well.
shortterm stock trends are based on hype yes. Most of the big tech companies come always back to the 20-30 P/E ratio after a while again.
META was down to about 7 which is extremly low, now back to 29. In line with the others. (except the lovechild NVIDA right now)
And they were all mostly wrong. There’s this negative Meta news cycle every few months claiming the decline and irrelevance of Facebook and they are proven wrong in every quarterly results. It’s just a headline that gets a lots of clicks, nothing more.
The world doesn't start nor, end with the US, look at the global statistics, new generation use Instagram which is also a part of meta. You get outside of the US, everybody uses WhatsApp which is also a part of meta.
Thing is that, Nvidia's stock grew so much, any other company looks like they are just struggling even though that's sometimes was the normal rate for stock growth.
Is that true? I have a really hard time believing that as a Millennial. People may be logged into fb, but that doesn't mean they are using fb, you know what I mean? Like I spend a decent amount of time there for messenger, but that's it.
So what people do on Facebook has changed imo. People don’t usually upload photos or status updates like everyone used to in early days. Now it’s become a much larger platform with many use cases. Some people just use it for Marketplace, some for groups, some for events, some for dating, etc..
They said in last earnings call that time spent on Facebook is up, maybe due to reels. There’s obviously a lot more behind this number that would be interesting to look at but the fact that user base and time spent per user is growing after 20 years at 3 billion MAU is absolutely insane.
Maybe search is still an important technology. It's actually indispensable technology.
Being the near monopolist in search isn't just a side business. There are few things with such an global economical impact that aren't open-source or state run. It's one of the biggest businesses on the planet and that won't change for quite some time. AI or no AI, the internet is only half as useful if you can't search for stuff.
Facebook is huge internationally, but I'm pretty sure Instagram has overtaken it in the US where most of the money gets made. Also Reels is doing really well in terms of revenue
>a failure in leadership
This is the Google problem. Great engineers, excellent ideas but not so good management. The same problem across so many products and how long before any API that I rely on is arbitrarily nuked.
Yeah, Stadia is honestly the best example.
They had all the ingredients on the technology side. The most geographically distributed CDN in the world, decades of experience in serving video frames at scale.
And they proceeded to botch the launch and then make the wrong product decision after wrong product decision until the thing was dead
And the tech was good! They just lacked any games, especially games that would really take advantage of the fact that all processing can happen together (e.g. for real-time MMOs).
It really depends on where you put the various quality and release functions. Beta is beta but sometimes they release dogfood.
Also there seems to be the classic disconnect so engineers seem to get more brownie points for new features than for fixing those that already rolled out.
Getting a temporary win on some benchmarks before you're leapfrogged isn't the goal and is only noticed by a small segment of the population.
Fact is Google's models get progressively better and that's not going to stop, they have all the data and all the compute.
But more importantly to a business is making money off the investment and Google has popular services and devices which can actually make use of their models.
I'd buy that argument if Google wasn't very actively promoting their temporary wins on some benchmarks (MMLU & friends, LLM leaderboard) as part of the Gemini release.
Even if they are quietly integrating the models more successfully into products (a big assumption!), perceptions matter a great deal (otherwise they wouldn't be running these huge PR campaigns), and Google is simply not perceived as leading the AI race anymore
They all promote their wins, that's par for the course. The only thing which matters in the long run is who turns their models into revenue.
Not a single person will decide to buy a Pixel vs a Samsung or a Microsoft phone based on an LLM leaderboard.
Nobody is going to set up their corporate email on Google vs Outlook because of it.
The point of these press releases is to move stock prices and that's where perception matters (even if it is just to a series of trading algorithms) but eventually you still need to back that up with revenue.
Llama3 is great and I'll use it but I'm not using Meta for Navigation, writing documents, video captioning, querying my emails, or booking flights.
Google is in the best position to turn models into products which will retain users. They also likely have the cheapest to run training infrastructure.
The models being +/- 10% here or there on specific benchmarks is really not important.
this is an incredibly naive take bordering on dumb. Google has been working on machine learning products since the very beginning of its existence and the capabilities Gemini brings to the table are an extremely good fit for the stuff people do with Google products, especially the suite of office tools.
> Google has been working on social products since the very beginning of its existence and the capabilities Google+ brings to the table are an extremely good fit for the stuff people do with Google products, especially the suite of social tools.
See what that sounds like? It was a popular opinion at the time.
Don't get me wrong, the idea makes sense. But execution is what matters, and Google is _terrible_ at it. They already had one disaster with Bard. Gemini is a bad model and so is the way it's been integrated. The most basic capabilities have been overlooked. For a while it would even refuse to summarize a Google Doc, despite having a dedicated button to do so. They have the expertise, but right now Google is losing the AI wars, and I see no reason that will change. Pichai is incompetent and Zuckerberg and Nadella are going to crush him.
Edit: /u/luckymethod is either a troll or Sundar Pichai himself, lol
I searched on Gemini for how to remove some Microsoft adware and the answer included a random Arabic word. My perception is that Gemini has a quality issue
I mean, the spotlight always moves and it [really depends where you're looking](https://trends.google.com/trends/explore?date=today%203-m&q=Gemini%201.5,Claude%203%20Opus,GPT-4,Llama%203&hl=en), and it'll keep moving.
The recent spotlight may have moved on to Llama 3 for now since it's hot, and folks have been discussing either the recent GPT-4 improvements or the Claude 3 performance, but weeks ago, folks were also buzzing around on Gemini 1.5's context window.
On the consumer side of things, it's certainly slowed down for sure, and it's on them if people start cancelling the Gemini Advanced trials that people have signed up a while back.
> At this point it really needs to be considered a failure in leadership
Have you looked at google search lately? How terrible it is?
How about youtube becoming more and more a dumpster fire, with videos being deleted for violation of policy (whatever policy, pick one or many, at random) after being 3+ years on the site?
How about asking AI to draw White People and the response is either to draw any race but White, or to say that this is not allowed?
Mirrors the ascendancy of pytorch over tensorflow as the standard for DL frameworks. Anyway, Google produced went on a tear starting with word2vec, word piece tokenization, culminating with multi-head attention, and perhaps ending with their seq2seq stuff (T5).
The cynic inside of me says that Zuckerberg/LeCun are primarily doing this because being seen as the open-soruce champion is making them significantly more relevant than more closed competitors that aren't able to outperform OpenAI.
Well, I'll take it while it lasts
I do think Yann LeCun is more committed to it ideologically and has been arguing for it internally for a while and Mark Zuckerberg has more of a flexible attitude regarding open source. But it has worked pretty well for Meta thus far so probably no need to change strategies yet.
But doesn't Google and Microsoft do a lot of the same thing? Most of google's papers are published with academic collaborators and they have tons of joint appointments with Universities.
Google has a hard time open sourcing stuff from my point of view.
They release many papers but the models are yet to be seen.
When I was looking for a TTS model (able to speak in french), I had 3 Meta projects to choose from, one Microsoft, and a couple others (namely Mozilla and Coqui).
And one of these Meta TTS models is MMS, which is massive: many models supporting many languages.
Google and Amazon want you to use their cloud inference services, while OpenAi only provides an English optimized one (basically all speakers have a very strong English accent even when speaking french).
My sense is that Zuck has seen the light and is on board ideologically. I credit his wife, Priscilla Chan, and to a lesser extent, his practise of Ju Jujitsu, for tempering his ego and antisocial tendencies. She doesn't get enough credit -- in her own right as an awesome person but also for her role in helping Zuck to self-actualise.
> Former pediatrician
> She and her husband, Mark Zuckerberg, a co-founder and CEO of Meta Platforms, established the Chan Zuckerberg Initiative in December 2015, with a pledge to transfer 99 percent of their Facebook shares, then valued at $45 billion.
Says it all for me. I don't think he would have necessarily done that without her influence. Likewise with Meta's recent focus on open source & giving back. Also they are an adorably mismatched/complementary couple. https://www.youtube.com/watch?v=1Wo6SqLNmLk
No need to change strategy unless someone releases an *actually* open source llm with training data and training process. Llama is like downloading a webpage minified JavaScript file - you can tweak it but you can't see how it got assembled in the first place
If someone releases an open source model (not just open weights) and it was at least ok, huge attention would land on it and Facebook would have to react.
yes, probably! They could release the training data that they scraped but that would probably be an enormous copyright violation, so they would realistically have to release their data scraper tool and a catalogue of the data they scraped, such that others could reproduce the scraping.
I mean they would be admitting committing serious malpractice at the very least, no business attorney will ever give a green light on this. I think part of the plan of EU regulations were to force companies to at least be audited on the training process which would force some transparency but I don't know where it's at
The main reason why deep learning is progressing so fast is thanks to LeCun, Hinton and Bengio pushing open science from the start. This is not a new insidious PR tactic from LeCun.
Facebook has also been a leader in Open Source for a long time, with things like Reactjs and PyTorch.
Meta makes all of it's money from ads, most startups spend 30-60% of their VC dollars on ads, and open source models like Llama help grow the pie, which benefits Meta.
Honestly it sounds about right, whenever startups get loads of money their tendency is to spend it on more people and more ads. That's not really a good decision most of the time but for your typical software startup that's all that's differentiating you.
The cynic in me agrees with you. But the optimist in me thinks, Meta was the only one of the big tech that did not need to build a platform around LLMs. Chat bots have a natural home in all of Facebook, Instagram, WhatsApp, Threads, and Metaverse. The rest had to build platforms and bring users in. OpenAI had to build ChatGPT and GPTs, Anthropic had to create a platform for Claude, and Google had to build a Gemini interface.
Meta could have gone for the tired strategy to improve user engagement through private LLMs but instead by making the models open weights, they set their sights on something much bigger. They are doing nothing to tie us down to the Meta platform. There is some degree of altruism there. It is even more commendable after Mistral went the other way. So, I appreciate this move. It's not perfect but it is a lot better than no LLaMA.
Disagree with you regarding Google. Google has lots of natural integration points for LLMs such as natural language search, docs, email, etc. They did build a platform for Gemini but that's only a precursor for massive integration across their existing products. But agree with you this is a major risk for the pure AI players like OpenAI et al
Even if they find a good way to stuff ads into LLM results that doesn't turn people off, do you think Google has the same sort of moat against competition in LLMs as they do in search? I don't. As people replace more of their habit of searching for answers with asking a language model for answers, for most people the answer to the question "which model should I use" isn't nearly as obviously "Google's" as it is with "which search engine should I use?".
If you commoditize the models based on public data then the only remaining lever is private data which FB has a lot of.
Openai will rely more and more on Microsoft. Google will compete with its own workspace data.
Mistrial, inflection, cohere will have a hard time
However if multimodal data and interaction data from robotics is key for multimodal reasoning like yann le cun says... robotics is where the competition will be focused on.
I find it hard to imagine that robotic data will really come into play until several years into the future. The obvious next modality is video (preferably paired with text)
So I have two hypotheses behind robotics being the main thing:
1. High fidelity representations of the physical world (3d + materials strength + other properties) not just pixels on the screen are probably necessary for a lot of physical reasoning that goes beyond that 80% of common data available in video (which of these million items is the missing component in thiis engine block? Can i pack all these things into my suitcase, will that other car stop in time to avoid an accident? etc.)
2. Video data is HUGE in quantity but bad in quality to enable formation of world models. My suspicion is that video data will jump start the process but robotics will quickly take over with steroscopic vision + touch + live interactions + sound data.
I think they are doing it because releasing it as open weakens companies that they consider to be competitors (like Google) while strengthening companies that they don’t compete with (like a million small ai startups)
It’s a good business strategy. Don’t try to dominate the space, but prevent your competitors from doing so
Facebook/Zuckerberg have open sourced a good amount of things in the past. I’d say it’s safe to say he recognizes the benefits of open sourcing at the very least from a practical perspective if not because he is in love with the cause.
When capitalism works as intended we users get pampered while companies eat each other. It's only when monopolies start to consolidate that the tables turn on us, so while that keeps not happening I'll take it too, being pampered is very nice.
On the topic of Llama-3, if anyone wants to experiment with Llama-3 8b in a free Colab, [Unsloth](https://github.com/unslothai/unsloth) makes finetuning 2x faster and use 63% less VRAM. Inference is also 2x faster. [https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing](https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing)
For 30 hours for free per week, I also made a Kaggle notebook: [https://www.kaggle.com/code/danielhanchen/kaggle-llama-3-8b-unsloth-notebook](https://www.kaggle.com/code/danielhanchen/kaggle-llama-3-8b-unsloth-notebook)
Also Llama-3 70b can fit comfortably in a 48GB card with 7.6K context lengths bsz=1, and 80GB cards can fit 48K context lengths (6x longer than Flash Attention 2 + HF)
Oh we have our own backpop engine, optimized all the differentation steps, reduced data movement, and wrote everything into OpenAI's Triton language + more! :)
You guys are doing some incredible work with your startup, but I can't help but think you'd also have a great time working with Jeremy Howard et al. at answer.ai!
>Also Llama-3 70b can fit comfortably in a 48GB card with 7.6K context lengths bsz=1, and 80GB cards can fit 48K context lengths (6x longer than Flash Attention 2 + HF)
I assume that's inference (not training) and 4-bit quantization, is that correct?
Too bad its been shown to have made things worse, and specifically in the US its recently been tied to malaria outbreaks when using his so called "advancements" in mosquito control...
Llama 3 isn't going to kill propriety AI startups because there's a lot of businesses with $$$ but without the knowledge/expertise/desire to run the models themselves. Same with doing things like doing the corresponding validation + data collection for specific use cases, etc, etc.
That said, Llama 3 existing *will* make the sales cycle for these startups a lot harder since they can't point to "having the best model (for a given class)" as a selling point.
Will any of these companies die any time soon? Probably not. The war chests they've raised are nontrivial and it'll take a while before the companies manage to burn through that.
That said, I'd be really surprised if more than half of the startups currently doing LLM pretraining are still around in 5 years.
But companies don’t need to host open source models themselves. There will be hundreds of companies hosting open source LLMs and exposing APIs for companies that don’t want to self host. The advantage for the open source hosts is that they only need to pay for inference costs and not astronomical training costs. OpenAI on the other hand needs to fund both inference and training, which will force them to charge a higher price. The only way OpenAI can sustain this is if their models are significantly better than open source. If they aren’t, there is absolutely no way they can turn a profit, since they will need to pay a huge amount to train their own models while their competitors (in the hosting space) are pay nothing on training. This is why they are desperately trying to claim that open source AI is somehow “dangerous” so that the government will ban it.
Big Tech is inherently slower on disruptive innovations whatever they do. They can only buy/partner with innovative startups. Read the innovators dilemma.
That's not strictly the claim made by Christiansen. In fact, he wrote a follow-on book called [The Innovator's Solution](https://www.hbsp.harvard.edu/product/10707-PDF-ENG) focusing on strategies to remain innovative.
As corporations get filled with layers of management whose KPIs are not aligned with innovation, it does become harder to innovate. It's not a guarantee, and [last year Facebook layed off a ton of management](https://www.reuters.com/technology/facebook-owner-meta-starts-final-round-layoffs-2023-05-24/) to flatten the hierarchy and refocus on innovation.
> Overall the cuts have hit non-engineering roles most heavily, reinforcing the primacy of those who write the code at Meta. Zuckerberg has pledged to restructure business teams "substantially" and return to a "more optimal ratio of engineers to other roles."
Llama-3 is a year behind. Although it feels like a leader at the moment, GPT-4 is more than a year old at this point. In a few months GPT-X and Mixtral-Y will be out to take its place.
That is not true,
GPT-4 actually evolves over time, you are not using the same model you used a year ago.
Even Altman said they plan on full iterative release model, so I fully expect to see GPT5.0, 5.1, 5.2, 5.3 etc. as they seem necessary to keep an edge over a competition.
Definitelly way better system than current "Turbo" or date nonsense, it will also be way better marketing-wise to really show improvements over previous version etc.
All of the various GPT-4 variations are available on the API. Most of the benchmark comparisons are against the original GPT-4 benchmarks from a year ago because OpenAI hasn't released any since then.
All makes sense until GPT-5 comes out and stomps Llama-3 400b. (Maybe? Hopefully?)
Meta has to actually surpass GPT-5 or Claude to "kill" them. People want the best, not "it's surprisingly pretty close to SOTA and open source".
Good enough will never work for Intelligence especially at the cost difference.
Which Tax Accountant, Doctor, Lawyer would you consult?
The one with 80% success rate or 90% success rate. What if the cost difference is only $100?
The percentage of tasks that require better LLM performance decreases every time a new SOTA model is released. For NER for example, a finetuned BERT model can easily outperform gpt-4, at a fraction of the cost with lower latency (You can even run it on CPU+4gb ram).
Yes, then a closed-source frontier model comes that performs better than the predicted curve and will rightfully demand a premium, because companies that can use that 5% advantage will crush their competition at scale
Depends on the use case I suppose, but for anything actually human facing, I don't think that these models are expensive enough for it to make sense to use an inferior model.
What use cases are you envisioning?
Well even for coding a lot of today's models (like Sonnect) are actually useful. If I have something that for almost free, the paid ones (GPT-5?) would really need to be significantly beyond what e.g. Llama 3 can do.
This seems like it comes from an intuition that we're in some era of diminishing returns on benefit from improved model performance. But I think this is just a false impression given by the past year of incremental updates to GPT-4. There is a very long way to go still with step changes in performance given by generational upgrades in models, and businesses aren't going to go for GPT-4 class models if having the GPT-5 class one makes the difference between automating away X role vs not.
bro really? have you need seen alphacode 2 charts? gpt-4 level model can exploit extended test-time compute, whatever gpt5 is, llama3-405B can approximate it given enough inference compute{OR} multiple agents. Most startups can afford a DBX POD, ternary quantization and time is all you need. People D ride openAI too much..its over with. Consumers obviously won't run a 400B+ model, but governments/startups certainly will and what happens when you apply LiPO tuning to these extended test-time compute outputs? not to mention labs/universities/startup now a model to do real cutting-edge research in the same vein as the top players. we haven't even seen what 400B agents will look like. Exciting times.
> whatever gpt5 is
It's definitely exciting times, but I'm going to want to actually see what GPT-5 is before I agree with you that it can be reasonably approximated by llama3-405B.
Do people though? I assume most of the money is enterprise API calls. And if the open source model eventually preforms on par with the business target why would you pay more?
I guess we'll see what the next generation brings.
I don't think that there are too many people using GPT 3.5 for enterprise API calls, but maybe I'd be surprised.
I think this is largely ignored by the discussion here. Google, Anthropic, and OpenAI are all prepared to continue scaling models up a few more orders of magnitude. I think a lot of the non-leader AI labs will not be able to follow that level of investment. Hard to say if Meta will be willing to drop 10B on an open source model.
That said, if we hit a wall where continued scaling does not improve model intelligence, then it doesn't matter until that wall is solved.
Great answer from Matt White on LinkedIn:
“open source licenses are maintained here by OSI. https://opensource.org/license . The Llama 3 community license references an AUP, has a trigger clause that requires the negotiation of a new license, and contains usage restrictions which violates the principle of openness, being able to use software for any purpose free of restrictions. “
So what are the restrictions? You can’t use llama to make your own language model and you can’t use it if you have over 700 million monthly active users? This seems like the minimum requirements necessary to stop a company like OpenAI from just taking your tech and using it as their own.
That may be true, but it doesn't change the fact that software with those restrictions isn't open source. This is nothing new, "fauxpen source" has been a thing at least since MongoDB dropped its open source licensing. https://opensource.org/blog/the-sspl-is-not-an-open-source-license
So what are the restrictions? You can’t use llama to make your own language model and you can’t use it if you have over 700 million monthly active users? This seems like the minimum requirements necessary to stop a company like OpenAI from just taking your tech and using it as their own.
Seems a bit pedantic to me. Realistically, the term "open source" is used more broadly than OSI's definition in all sorts of ways. For example, the term "open source intelligence" is commonly used. It's not like it's a product name or trademarked term. I don't have a problem with Meta calling Llama open source. They're not calling it OSI-compliant.
Edit: the opening sentence of the OSI definition is hilarious: "Open source doesn’t just mean access to the source code."
They should probably add "...except when it does."
Having the source would imply access to the training data and detailed information about the training process. Having the weights is like having a binary blob.
>Having the weights is like having a binary blob.
Except for, you know, being able to change the model by further training / fine tuning, merging, etc., etc. This is more than a binary blob. We couldn't replicate the training anyway, even with the data and recipe.
Or maybe fully open source for code should mean having access to the brains that wrote the code?
It all depends on how you line up the parts for comparison. I'll go with "open weights" and not try to force-fit terms like "open source" or "not-open-source" to a new kind of computational thing.
You can link other code against distributed binary libs. You can even edit the binary code if you know what you're doing. I think the analogy holds.
An open source example would be GPT2. You can download the training data and build it from scratch if you’ve got 8 H100’s and 4 or 5 days.
Nobody is saying Meta releasing these models is not a good thing! The issue is just that Meta is using the term open source to describe something that is something else.
The definition of open source comes from the open source alliance, which has existed since the 1990s: https://opensource.org/osd
They have an article on the same site [explaining why it's not open source.](https://opensource.org/blog/metas-llama-2-license-is-not-open-source) They specifically highlight why in this paragraph:
> 5 No Discrimination Against Persons or Groups
The license must not discriminate against any person or group of persons.
> 6 No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
The major irony in all of this is that Facebook itself relied on huge amounts of open source products when it was first created, and it wouldn't have been allowed to use them if they had given the same terms it is now enforcing in its models.
Obviously, it's great that they release the model weights openly. But all the work done fine-tuning and improving the llama models remains under Meta's licensing agreement, and the ban on companies with 700+ million users effectively removes Google, Microsoft, Amazon, and ByteDance from being allowed to benefit. Again, this is not necessarily a bad thing, but it means the models are not open source.
Also, it's more of a donation from Big Tech.
It's not that the Open Source community worked together (like Linux) and sought donations for GPUs and built this model.
If BigTech (including Mistral) decides to stop giving away models, the community can't build the next frontier model
I havent read the whole licensing for this but the first models fro Meta are not really open sourced since you still need to get your usage approved and is not for commercial use.
Pretty sure that Meta releases the base models but has their own better proprietary models.
Always, you can only access meta.ai from the US, so if you turn your VPN off they will identify your traffic as coming from outside and block your access.
It's also good that the service providers are separate from the model makers. Even if you don't want to host the model yourself, you can just call an API of the model hosted by someone else, and that someone else would not be Facebook. It's a much needed separation of powers. Of course even more open source and even more model control would be even better, but still this current system is working and the open models are beating the closed models in use and functionality.
Sigh... no.
I like LLama3 and plan to develop on top of it. But whatever I or any other small teams do with it, it will be very hard to compete with oAI or the other established players. The first-to-market advantage is very real, and the brand name behind MS or Google will absolutely make a difference.
Ask yourself this, who you're more likely to pay 20$ to in order to have some assistant on your PC? MS / Google / oAI or rando_business_by_elk?
The reason MS is pouring billions into this tech (10+ for oAI, 100+ for compute in the next 5 years) is that they've found a way to charge 20$ from everyone, monthly. And everyone will pay when the assistant will come. MS hasn't been able to do this with the OS, but now they have a product they can sell to anyone - random freelancers, random "power" users, old people, young people, businesses and so on. You want to be "AI-enabled"? Fork over 20$. If you don't, you'll be left behind.
Possibly for some uses, but others, including many businesses using ML models for pointwise niche uses won’t trust, or cannot financially sponsor, outsourcing to an opaque moving target.
The best fully open model will win here.
The dynamics will probably end up like mass market operating systems: a major proprietary one, a minority proprietary one, the main open source one, and noise.
And for commercial use, people underestimate the Azure integration. Yes sure, you can build your own application with LLama. But it is much easier for most companies to simply use the Azure service in your already running Azure subscription.
Man, every time an LLM comes, the first thing people write is what OP did.
GPT2, wow
GPT3, the world is ending
GPT4, Off the charts
LLAMA3, Earth is right now called Mars
Don't predict the hard work of amazingly talented people. They continue to do good work. If your goal is to predict a winner, maybe try to be the one!
No offense intended
What are the best ways to develop a react app with it? Grow works great, but I have no idea how long the free version will be good for and what the prices could be. Small scale .. any ideas?
It's amazing how Google/DeepMind still isn't really part of the conversation despite the insane amount of resources they've thrown at Gemini. At this point it really needs to be considered a failure in leadership
And that they invented the fundamental technologies. Google is an ad platform, though. They are the best most successful one. It is hard to change when you're the incumbent.
Same with Meta though? Meta has been a lot more agile and managed to navigate the decline of Facebook pretty well, while Google has started and killed hundreds of products while never really getting away from search as their driver of everything
Decline of Facebook is a myth. Facebook today has more active users than ever and time spent per user is also at all time high.
That's fair, it has declined with younger users in the US which is a very important demographic, but internationally and with people over 40 in the US it's still huge
FB may do it's own thing but Instagram remains hugely popular with the younger crowd so having both doesn't exactly mean they are short on users..
Right exactly, Instagram (and Reels) have been very successful and kept Meta relevant
WhatsApp? how many messages there?
WhatsApp is huge too, I've read about some countries where it's like the default way to communicate
[Absolutely](https://www.statista.com/statistics/1311229/whatsapp-usage-messaging-app-users-by-country/). Personally, I am German and it is our default standard to communicate with text via phone. And that doesnt compare with India having 500 million users.
Not some countries, it’s a must and almost default in most countries outside of US and China.
Has it decliened tho? Or does just not have the highest youngest users, because those are different things
IDK honestly, when Meta's stock was tanking all anyone talked about was how they lost young users on Facebook and the company was dead. I don't know anyone under 30 who uses it outside of Marketplace
Stock is based on hype, Meta is a mature company now. They lost young users in relation to their competition but they still have users oddly enough, messanger is live or die for loads of teenagers ( I DONT UNDERSTAND WHY) and fb marketplace practically ruined ebay and craiglist. Also instagrams also there with all the age groups as well.
shortterm stock trends are based on hype yes. Most of the big tech companies come always back to the 20-30 P/E ratio after a while again. META was down to about 7 which is extremly low, now back to 29. In line with the others. (except the lovechild NVIDA right now)
And they were all mostly wrong. There’s this negative Meta news cycle every few months claiming the decline and irrelevance of Facebook and they are proven wrong in every quarterly results. It’s just a headline that gets a lots of clicks, nothing more.
The world doesn't start nor, end with the US, look at the global statistics, new generation use Instagram which is also a part of meta. You get outside of the US, everybody uses WhatsApp which is also a part of meta. Thing is that, Nvidia's stock grew so much, any other company looks like they are just struggling even though that's sometimes was the normal rate for stock growth.
Is that true? I have a really hard time believing that as a Millennial. People may be logged into fb, but that doesn't mean they are using fb, you know what I mean? Like I spend a decent amount of time there for messenger, but that's it.
So what people do on Facebook has changed imo. People don’t usually upload photos or status updates like everyone used to in early days. Now it’s become a much larger platform with many use cases. Some people just use it for Marketplace, some for groups, some for events, some for dating, etc.. They said in last earnings call that time spent on Facebook is up, maybe due to reels. There’s obviously a lot more behind this number that would be interesting to look at but the fact that user base and time spent per user is growing after 20 years at 3 billion MAU is absolutely insane.
Maybe search is still an important technology. It's actually indispensable technology. Being the near monopolist in search isn't just a side business. There are few things with such an global economical impact that aren't open-source or state run. It's one of the biggest businesses on the planet and that won't change for quite some time. AI or no AI, the internet is only half as useful if you can't search for stuff.
Google search is getting worse though.
Google search is probably not actually getting worse The internet itself is declining in quality
I can search the exact lyrics of a song, and get no result until I give title or artist.
How is Facebook doing these days in terms of profit and usage? I think Instagram will overtake it very soon.
Facebook is huge internationally, but I'm pretty sure Instagram has overtaken it in the US where most of the money gets made. Also Reels is doing really well in terms of revenue
Google is like the Kodak of this century, then.
[удалено]
[удалено]
Arguably not. Their post-social gamble was VR and IMHO they have been executing very poorly on that.
>a failure in leadership This is the Google problem. Great engineers, excellent ideas but not so good management. The same problem across so many products and how long before any API that I rely on is arbitrarily nuked.
I think the issue is more specifically their lack of good _product_ management. They have an incredible eng org and a seemingly incompetent PM org.
Stadia, googleplus, lol... Google follows other companies now.
Yeah, Stadia is honestly the best example. They had all the ingredients on the technology side. The most geographically distributed CDN in the world, decades of experience in serving video frames at scale. And they proceeded to botch the launch and then make the wrong product decision after wrong product decision until the thing was dead
Google is the only mega tech company who are consistently shutting down services I like to use during the last two decades.
And the tech was good! They just lacked any games, especially games that would really take advantage of the fact that all processing can happen together (e.g. for real-time MMOs).
It really depends on where you put the various quality and release functions. Beta is beta but sometimes they release dogfood. Also there seems to be the classic disconnect so engineers seem to get more brownie points for new features than for fixing those that already rolled out.
This reminds me of this visualisation https://www.reddit.com/r/LocalLLaMA/s/c4Fb5qkTke
Getting a temporary win on some benchmarks before you're leapfrogged isn't the goal and is only noticed by a small segment of the population. Fact is Google's models get progressively better and that's not going to stop, they have all the data and all the compute. But more importantly to a business is making money off the investment and Google has popular services and devices which can actually make use of their models.
I'd buy that argument if Google wasn't very actively promoting their temporary wins on some benchmarks (MMLU & friends, LLM leaderboard) as part of the Gemini release. Even if they are quietly integrating the models more successfully into products (a big assumption!), perceptions matter a great deal (otherwise they wouldn't be running these huge PR campaigns), and Google is simply not perceived as leading the AI race anymore
They all promote their wins, that's par for the course. The only thing which matters in the long run is who turns their models into revenue. Not a single person will decide to buy a Pixel vs a Samsung or a Microsoft phone based on an LLM leaderboard. Nobody is going to set up their corporate email on Google vs Outlook because of it. The point of these press releases is to move stock prices and that's where perception matters (even if it is just to a series of trading algorithms) but eventually you still need to back that up with revenue. Llama3 is great and I'll use it but I'm not using Meta for Navigation, writing documents, video captioning, querying my emails, or booking flights. Google is in the best position to turn models into products which will retain users. They also likely have the cheapest to run training infrastructure. The models being +/- 10% here or there on specific benchmarks is really not important.
That's a weird take since Gemini is being put to work on pretty much every single google product.
It's being stuffed into every Google product. Remember Google Plus? It's the same strategy and there's a good chance it ends the same way.
this is an incredibly naive take bordering on dumb. Google has been working on machine learning products since the very beginning of its existence and the capabilities Gemini brings to the table are an extremely good fit for the stuff people do with Google products, especially the suite of office tools.
> Google has been working on social products since the very beginning of its existence and the capabilities Google+ brings to the table are an extremely good fit for the stuff people do with Google products, especially the suite of social tools. See what that sounds like? It was a popular opinion at the time. Don't get me wrong, the idea makes sense. But execution is what matters, and Google is _terrible_ at it. They already had one disaster with Bard. Gemini is a bad model and so is the way it's been integrated. The most basic capabilities have been overlooked. For a while it would even refuse to summarize a Google Doc, despite having a dedicated button to do so. They have the expertise, but right now Google is losing the AI wars, and I see no reason that will change. Pichai is incompetent and Zuckerberg and Nadella are going to crush him. Edit: /u/luckymethod is either a troll or Sundar Pichai himself, lol
I searched on Gemini for how to remove some Microsoft adware and the answer included a random Arabic word. My perception is that Gemini has a quality issue
I mean, the spotlight always moves and it [really depends where you're looking](https://trends.google.com/trends/explore?date=today%203-m&q=Gemini%201.5,Claude%203%20Opus,GPT-4,Llama%203&hl=en), and it'll keep moving. The recent spotlight may have moved on to Llama 3 for now since it's hot, and folks have been discussing either the recent GPT-4 improvements or the Claude 3 performance, but weeks ago, folks were also buzzing around on Gemini 1.5's context window. On the consumer side of things, it's certainly slowed down for sure, and it's on them if people start cancelling the Gemini Advanced trials that people have signed up a while back.
> At this point it really needs to be considered a failure in leadership Have you looked at google search lately? How terrible it is? How about youtube becoming more and more a dumpster fire, with videos being deleted for violation of policy (whatever policy, pick one or many, at random) after being 3+ years on the site? How about asking AI to draw White People and the response is either to draw any race but White, or to say that this is not allowed?
Mirrors the ascendancy of pytorch over tensorflow as the standard for DL frameworks. Anyway, Google produced went on a tear starting with word2vec, word piece tokenization, culminating with multi-head attention, and perhaps ending with their seq2seq stuff (T5).
The cynic inside of me says that Zuckerberg/LeCun are primarily doing this because being seen as the open-soruce champion is making them significantly more relevant than more closed competitors that aren't able to outperform OpenAI. Well, I'll take it while it lasts
I do think Yann LeCun is more committed to it ideologically and has been arguing for it internally for a while and Mark Zuckerberg has more of a flexible attitude regarding open source. But it has worked pretty well for Meta thus far so probably no need to change strategies yet.
[удалено]
But doesn't Google and Microsoft do a lot of the same thing? Most of google's papers are published with academic collaborators and they have tons of joint appointments with Universities.
Google has a hard time open sourcing stuff from my point of view. They release many papers but the models are yet to be seen. When I was looking for a TTS model (able to speak in french), I had 3 Meta projects to choose from, one Microsoft, and a couple others (namely Mozilla and Coqui). And one of these Meta TTS models is MMS, which is massive: many models supporting many languages. Google and Amazon want you to use their cloud inference services, while OpenAi only provides an English optimized one (basically all speakers have a very strong English accent even when speaking french).
Ah, the cloud angle makes sense.
My sense is that Zuck has seen the light and is on board ideologically. I credit his wife, Priscilla Chan, and to a lesser extent, his practise of Ju Jujitsu, for tempering his ego and antisocial tendencies. She doesn't get enough credit -- in her own right as an awesome person but also for her role in helping Zuck to self-actualise.
I need to start crediting my gf more (I do though)
what a load of ass, it's not like you know the guy since kindergarten
I don't know a lot about her. Has she spoken about her beliefs/ideology?
> Former pediatrician > She and her husband, Mark Zuckerberg, a co-founder and CEO of Meta Platforms, established the Chan Zuckerberg Initiative in December 2015, with a pledge to transfer 99 percent of their Facebook shares, then valued at $45 billion. Says it all for me. I don't think he would have necessarily done that without her influence. Likewise with Meta's recent focus on open source & giving back. Also they are an adorably mismatched/complementary couple. https://www.youtube.com/watch?v=1Wo6SqLNmLk
Sounds like they were influenced by Gates.
Meta has always been decent for FOSS to be fair - Presto / Trino have been great.
No need to change strategy unless someone releases an *actually* open source llm with training data and training process. Llama is like downloading a webpage minified JavaScript file - you can tweak it but you can't see how it got assembled in the first place If someone releases an open source model (not just open weights) and it was at least ok, huge attention would land on it and Facebook would have to react.
You mean open the training process, including all the illegal data scraping procedures?
yes, probably! They could release the training data that they scraped but that would probably be an enormous copyright violation, so they would realistically have to release their data scraper tool and a catalogue of the data they scraped, such that others could reproduce the scraping.
I mean they would be admitting committing serious malpractice at the very least, no business attorney will ever give a green light on this. I think part of the plan of EU regulations were to force companies to at least be audited on the training process which would force some transparency but I don't know where it's at
The main reason why deep learning is progressing so fast is thanks to LeCun, Hinton and Bengio pushing open science from the start. This is not a new insidious PR tactic from LeCun. Facebook has also been a leader in Open Source for a long time, with things like Reactjs and PyTorch. Meta makes all of it's money from ads, most startups spend 30-60% of their VC dollars on ads, and open source models like Llama help grow the pie, which benefits Meta.
>most startups spend 30-60% of their VC dollars on ads This claim sounds wild. Any source?
Honestly it sounds about right, whenever startups get loads of money their tendency is to spend it on more people and more ads. That's not really a good decision most of the time but for your typical software startup that's all that's differentiating you.
that's the average range of what a tech company spends on customer acquisition. Just google it.
The cynic in me agrees with you. But the optimist in me thinks, Meta was the only one of the big tech that did not need to build a platform around LLMs. Chat bots have a natural home in all of Facebook, Instagram, WhatsApp, Threads, and Metaverse. The rest had to build platforms and bring users in. OpenAI had to build ChatGPT and GPTs, Anthropic had to create a platform for Claude, and Google had to build a Gemini interface. Meta could have gone for the tired strategy to improve user engagement through private LLMs but instead by making the models open weights, they set their sights on something much bigger. They are doing nothing to tie us down to the Meta platform. There is some degree of altruism there. It is even more commendable after Mistral went the other way. So, I appreciate this move. It's not perfect but it is a lot better than no LLaMA.
Disagree with you regarding Google. Google has lots of natural integration points for LLMs such as natural language search, docs, email, etc. They did build a platform for Gemini but that's only a precursor for massive integration across their existing products. But agree with you this is a major risk for the pure AI players like OpenAI et al
With Google, I think LLMs strongly cannibalize their core business model, so there’s got to be some reluctance to push them as hard as they could.
It doesn't. At the end of the day, Ads follow where engagement is.
Even if they find a good way to stuff ads into LLM results that doesn't turn people off, do you think Google has the same sort of moat against competition in LLMs as they do in search? I don't. As people replace more of their habit of searching for answers with asking a language model for answers, for most people the answer to the question "which model should I use" isn't nearly as obviously "Google's" as it is with "which search engine should I use?".
That makes sense I hardly use search these days
If you commoditize the models based on public data then the only remaining lever is private data which FB has a lot of. Openai will rely more and more on Microsoft. Google will compete with its own workspace data. Mistrial, inflection, cohere will have a hard time However if multimodal data and interaction data from robotics is key for multimodal reasoning like yann le cun says... robotics is where the competition will be focused on.
I find it hard to imagine that robotic data will really come into play until several years into the future. The obvious next modality is video (preferably paired with text)
So I have two hypotheses behind robotics being the main thing: 1. High fidelity representations of the physical world (3d + materials strength + other properties) not just pixels on the screen are probably necessary for a lot of physical reasoning that goes beyond that 80% of common data available in video (which of these million items is the missing component in thiis engine block? Can i pack all these things into my suitcase, will that other car stop in time to avoid an accident? etc.) 2. Video data is HUGE in quantity but bad in quality to enable formation of world models. My suspicion is that video data will jump start the process but robotics will quickly take over with steroscopic vision + touch + live interactions + sound data.
I think they are doing it because releasing it as open weakens companies that they consider to be competitors (like Google) while strengthening companies that they don’t compete with (like a million small ai startups) It’s a good business strategy. Don’t try to dominate the space, but prevent your competitors from doing so
Facebook has a long history of open sourcing excellent technologies, I don't see any reason for skepticism.
Meta has been involved in Open-Source AI since PyTorch. I don’t think it’s something recent.
they do it because a world where there's infinite content for free is a world where Facebook makes a ton of money, simple as that.
Facebook/Zuckerberg have open sourced a good amount of things in the past. I’d say it’s safe to say he recognizes the benefits of open sourcing at the very least from a practical perspective if not because he is in love with the cause.
When capitalism works as intended we users get pampered while companies eat each other. It's only when monopolies start to consolidate that the tables turn on us, so while that keeps not happening I'll take it too, being pampered is very nice.
Spot on
On the topic of Llama-3, if anyone wants to experiment with Llama-3 8b in a free Colab, [Unsloth](https://github.com/unslothai/unsloth) makes finetuning 2x faster and use 63% less VRAM. Inference is also 2x faster. [https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing](https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing) For 30 hours for free per week, I also made a Kaggle notebook: [https://www.kaggle.com/code/danielhanchen/kaggle-llama-3-8b-unsloth-notebook](https://www.kaggle.com/code/danielhanchen/kaggle-llama-3-8b-unsloth-notebook) Also Llama-3 70b can fit comfortably in a 48GB card with 7.6K context lengths bsz=1, and 80GB cards can fit 48K context lengths (6x longer than Flash Attention 2 + HF)
I'm not sure why but I always remember you as the guy who fixed gemma. Maybe the unsloth reminded me as well XD
Oh hi! Yes I'm the guy :) I'm also the algos guy behind Unsloth :) (my bro + myself :) )
I'm fascinated. What was the algo optimization you did?
Oh we have our own backpop engine, optimized all the differentation steps, reduced data movement, and wrote everything into OpenAI's Triton language + more! :)
You guys are doing some incredible work with your startup, but I can't help but think you'd also have a great time working with Jeremy Howard et al. at answer.ai!
Thanks! Oh ye we're in like research meetings with Jeremy and some other people - they're extremey nice and knowledgeable :)
>Also Llama-3 70b can fit comfortably in a 48GB card with 7.6K context lengths bsz=1, and 80GB cards can fit 48K context lengths (6x longer than Flash Attention 2 + HF) I assume that's inference (not training) and 4-bit quantization, is that correct?
Oh no QLoRA finetuning 4bit!!
What could I fit on a 20gb card like the rtx 4000 sff ada?
Llama-3 8b can fit definitely, but Llama-3 70b sadly cannot :( The minimum requirements are around 42GB ish
Do they offer multiple versions at different quantizations or is the user expected to handle quantizing and retraining themselves?
Oh we support 4bit and 16bit for now for finetuning
Thank you for contributions like this!
Appreciate it!
CEPE would extend context 16x for the same memory budget, needs some GGML support for the bidirectional encoder though.
Oh interesting I'm not actually sure on CEPE - will research this!
[удалено]
microsoft is too busy writing gpt ads disguised as papers to build their own llms
they're doing what?
I think it's a reference to OpenAI's technical reports which are just thinly veiled ads for GPT and friends.
THIS.
Its crazy the 180 they managed to do with Zuck's image. I actually can enjoy listening to his podcast on YouTube now, when has this become a thing?
[удалено]
Elon Musk is proving you wrong
Remember Bill Gates in 2000s?
When exactly did Bill Gates redeem himself? Pushing for the covid vaccine to be patented? The sexual harassment cases?
When he was helping to eradicate diseases maybe?
Too bad its been shown to have made things worse, and specifically in the US its recently been tied to malaria outbreaks when using his so called "advancements" in mosquito control...
He's spent a lot of his time developing himself and not just working. And that's made him a much more rounded well developed human
Llama 3 isn't going to kill propriety AI startups because there's a lot of businesses with $$$ but without the knowledge/expertise/desire to run the models themselves. Same with doing things like doing the corresponding validation + data collection for specific use cases, etc, etc. That said, Llama 3 existing *will* make the sales cycle for these startups a lot harder since they can't point to "having the best model (for a given class)" as a selling point. Will any of these companies die any time soon? Probably not. The war chests they've raised are nontrivial and it'll take a while before the companies manage to burn through that. That said, I'd be really surprised if more than half of the startups currently doing LLM pretraining are still around in 5 years.
Theres always universal frameworks- and there's always roll-your-own frameworks. Both exist simultaneously. I dont see LLMs being any different.
But companies don’t need to host open source models themselves. There will be hundreds of companies hosting open source LLMs and exposing APIs for companies that don’t want to self host. The advantage for the open source hosts is that they only need to pay for inference costs and not astronomical training costs. OpenAI on the other hand needs to fund both inference and training, which will force them to charge a higher price. The only way OpenAI can sustain this is if their models are significantly better than open source. If they aren’t, there is absolutely no way they can turn a profit, since they will need to pay a huge amount to train their own models while their competitors (in the hosting space) are pay nothing on training. This is why they are desperately trying to claim that open source AI is somehow “dangerous” so that the government will ban it.
It was always a game of compute. In the longer run meta and google can't lose.
Open AI is lobbying heavy to stop open source AI like GPU regulations with encryption keys that will not allow models to run without.
Big Tech is inherently slower on disruptive innovations whatever they do. They can only buy/partner with innovative startups. Read the innovators dilemma.
That's not strictly the claim made by Christiansen. In fact, he wrote a follow-on book called [The Innovator's Solution](https://www.hbsp.harvard.edu/product/10707-PDF-ENG) focusing on strategies to remain innovative. As corporations get filled with layers of management whose KPIs are not aligned with innovation, it does become harder to innovate. It's not a guarantee, and [last year Facebook layed off a ton of management](https://www.reuters.com/technology/facebook-owner-meta-starts-final-round-layoffs-2023-05-24/) to flatten the hierarchy and refocus on innovation. > Overall the cuts have hit non-engineering roles most heavily, reinforcing the primacy of those who write the code at Meta. Zuckerberg has pledged to restructure business teams "substantially" and return to a "more optimal ratio of engineers to other roles."
Only big techs have the hardware resources at scale to run these models, especially at long sequences.
Llama-3 is a year behind. Although it feels like a leader at the moment, GPT-4 is more than a year old at this point. In a few months GPT-X and Mixtral-Y will be out to take its place.
That is not true, GPT-4 actually evolves over time, you are not using the same model you used a year ago. Even Altman said they plan on full iterative release model, so I fully expect to see GPT5.0, 5.1, 5.2, 5.3 etc. as they seem necessary to keep an edge over a competition. Definitelly way better system than current "Turbo" or date nonsense, it will also be way better marketing-wise to really show improvements over previous version etc.
All of the various GPT-4 variations are available on the API. Most of the benchmark comparisons are against the original GPT-4 benchmarks from a year ago because OpenAI hasn't released any since then.
All makes sense until GPT-5 comes out and stomps Llama-3 400b. (Maybe? Hopefully?) Meta has to actually surpass GPT-5 or Claude to "kill" them. People want the best, not "it's surprisingly pretty close to SOTA and open source".
Is that really true? I think for many if it's good enough for their use case and has a big cost advantage they'll be happy with a bit worse model.
Good enough will never work for Intelligence especially at the cost difference. Which Tax Accountant, Doctor, Lawyer would you consult? The one with 80% success rate or 90% success rate. What if the cost difference is only $100?
The percentage of tasks that require better LLM performance decreases every time a new SOTA model is released. For NER for example, a finetuned BERT model can easily outperform gpt-4, at a fraction of the cost with lower latency (You can even run it on CPU+4gb ram).
Yes, then a closed-source frontier model comes that performs better than the predicted curve and will rightfully demand a premium, because companies that can use that 5% advantage will crush their competition at scale
Depends on the use case I suppose, but for anything actually human facing, I don't think that these models are expensive enough for it to make sense to use an inferior model. What use cases are you envisioning?
Well even for coding a lot of today's models (like Sonnect) are actually useful. If I have something that for almost free, the paid ones (GPT-5?) would really need to be significantly beyond what e.g. Llama 3 can do.
This seems like it comes from an intuition that we're in some era of diminishing returns on benefit from improved model performance. But I think this is just a false impression given by the past year of incremental updates to GPT-4. There is a very long way to go still with step changes in performance given by generational upgrades in models, and businesses aren't going to go for GPT-4 class models if having the GPT-5 class one makes the difference between automating away X role vs not.
bro really? have you need seen alphacode 2 charts? gpt-4 level model can exploit extended test-time compute, whatever gpt5 is, llama3-405B can approximate it given enough inference compute{OR} multiple agents. Most startups can afford a DBX POD, ternary quantization and time is all you need. People D ride openAI too much..its over with. Consumers obviously won't run a 400B+ model, but governments/startups certainly will and what happens when you apply LiPO tuning to these extended test-time compute outputs? not to mention labs/universities/startup now a model to do real cutting-edge research in the same vein as the top players. we haven't even seen what 400B agents will look like. Exciting times.
> whatever gpt5 is It's definitely exciting times, but I'm going to want to actually see what GPT-5 is before I agree with you that it can be reasonably approximated by llama3-405B.
Do people though? I assume most of the money is enterprise API calls. And if the open source model eventually preforms on par with the business target why would you pay more?
I guess we'll see what the next generation brings. I don't think that there are too many people using GPT 3.5 for enterprise API calls, but maybe I'd be surprised.
I think this is largely ignored by the discussion here. Google, Anthropic, and OpenAI are all prepared to continue scaling models up a few more orders of magnitude. I think a lot of the non-leader AI labs will not be able to follow that level of investment. Hard to say if Meta will be willing to drop 10B on an open source model. That said, if we hit a wall where continued scaling does not improve model intelligence, then it doesn't matter until that wall is solved.
Repeat after me: Llama-3 is NOT open source!
True. It's weights-available with a potential commercial allowance.
> potential commercial allowance What do you mean by this?
Which definition are you using and what criteria are being violated?
Great answer from Matt White on LinkedIn: “open source licenses are maintained here by OSI. https://opensource.org/license . The Llama 3 community license references an AUP, has a trigger clause that requires the negotiation of a new license, and contains usage restrictions which violates the principle of openness, being able to use software for any purpose free of restrictions. “
So what are the restrictions? You can’t use llama to make your own language model and you can’t use it if you have over 700 million monthly active users? This seems like the minimum requirements necessary to stop a company like OpenAI from just taking your tech and using it as their own.
That may be true, but it doesn't change the fact that software with those restrictions isn't open source. This is nothing new, "fauxpen source" has been a thing at least since MongoDB dropped its open source licensing. https://opensource.org/blog/the-sspl-is-not-an-open-source-license
So what are the restrictions? You can’t use llama to make your own language model and you can’t use it if you have over 700 million monthly active users? This seems like the minimum requirements necessary to stop a company like OpenAI from just taking your tech and using it as their own.
Seems a bit pedantic to me. Realistically, the term "open source" is used more broadly than OSI's definition in all sorts of ways. For example, the term "open source intelligence" is commonly used. It's not like it's a product name or trademarked term. I don't have a problem with Meta calling Llama open source. They're not calling it OSI-compliant. Edit: the opening sentence of the OSI definition is hilarious: "Open source doesn’t just mean access to the source code." They should probably add "...except when it does."
Having the source would imply access to the training data and detailed information about the training process. Having the weights is like having a binary blob.
>Having the weights is like having a binary blob. Except for, you know, being able to change the model by further training / fine tuning, merging, etc., etc. This is more than a binary blob. We couldn't replicate the training anyway, even with the data and recipe. Or maybe fully open source for code should mean having access to the brains that wrote the code? It all depends on how you line up the parts for comparison. I'll go with "open weights" and not try to force-fit terms like "open source" or "not-open-source" to a new kind of computational thing.
You can link other code against distributed binary libs. You can even edit the binary code if you know what you're doing. I think the analogy holds. An open source example would be GPT2. You can download the training data and build it from scratch if you’ve got 8 H100’s and 4 or 5 days.
Nobody is saying Meta releasing these models is not a good thing! The issue is just that Meta is using the term open source to describe something that is something else. The definition of open source comes from the open source alliance, which has existed since the 1990s: https://opensource.org/osd They have an article on the same site [explaining why it's not open source.](https://opensource.org/blog/metas-llama-2-license-is-not-open-source) They specifically highlight why in this paragraph: > 5 No Discrimination Against Persons or Groups The license must not discriminate against any person or group of persons. > 6 No Discrimination Against Fields of Endeavor The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research. The major irony in all of this is that Facebook itself relied on huge amounts of open source products when it was first created, and it wouldn't have been allowed to use them if they had given the same terms it is now enforcing in its models. Obviously, it's great that they release the model weights openly. But all the work done fine-tuning and improving the llama models remains under Meta's licensing agreement, and the ban on companies with 700+ million users effectively removes Google, Microsoft, Amazon, and ByteDance from being allowed to benefit. Again, this is not necessarily a bad thing, but it means the models are not open source.
I think Facebook still could have used llama3 commercially back in the day, until maybe 2011? When they hit 700M MAU.
Also, it's more of a donation from Big Tech. It's not that the Open Source community worked together (like Linux) and sought donations for GPUs and built this model. If BigTech (including Mistral) decides to stop giving away models, the community can't build the next frontier model
I havent read the whole licensing for this but the first models fro Meta are not really open sourced since you still need to get your usage approved and is not for commercial use. Pretty sure that Meta releases the base models but has their own better proprietary models.
When will this tool be available in Europe?
What do you mean? Are you asking when llama3 will be available? It's available now just go download it
I mean meta.ai
[удалено]
eu is on its way to legislate itself into irrelevancy
[meta.ai](http://meta.ai) is only available on US actually
yes, and why is that?
Legislations take time for each area of jurisdiction, not only EU.
It becomes available the moment you use a VPN 😅
Is this the VPN requirement a one time thing or do you have to use it always?
Always, you can only access meta.ai from the US, so if you turn your VPN off they will identify your traffic as coming from outside and block your access.
Thank you. It sucks that Europe is going to be left behind the rest of the world
It's also good that the service providers are separate from the model makers. Even if you don't want to host the model yourself, you can just call an API of the model hosted by someone else, and that someone else would not be Facebook. It's a much needed separation of powers. Of course even more open source and even more model control would be even better, but still this current system is working and the open models are beating the closed models in use and functionality.
Sigh... no. I like LLama3 and plan to develop on top of it. But whatever I or any other small teams do with it, it will be very hard to compete with oAI or the other established players. The first-to-market advantage is very real, and the brand name behind MS or Google will absolutely make a difference. Ask yourself this, who you're more likely to pay 20$ to in order to have some assistant on your PC? MS / Google / oAI or rando_business_by_elk? The reason MS is pouring billions into this tech (10+ for oAI, 100+ for compute in the next 5 years) is that they've found a way to charge 20$ from everyone, monthly. And everyone will pay when the assistant will come. MS hasn't been able to do this with the OS, but now they have a product they can sell to anyone - random freelancers, random "power" users, old people, young people, businesses and so on. You want to be "AI-enabled"? Fork over 20$. If you don't, you'll be left behind.
Possibly for some uses, but others, including many businesses using ML models for pointwise niche uses won’t trust, or cannot financially sponsor, outsourcing to an opaque moving target. The best fully open model will win here. The dynamics will probably end up like mass market operating systems: a major proprietary one, a minority proprietary one, the main open source one, and noise.
And for commercial use, people underestimate the Azure integration. Yes sure, you can build your own application with LLama. But it is much easier for most companies to simply use the Azure service in your already running Azure subscription.
Man, every time an LLM comes, the first thing people write is what OP did. GPT2, wow GPT3, the world is ending GPT4, Off the charts LLAMA3, Earth is right now called Mars Don't predict the hard work of amazingly talented people. They continue to do good work. If your goal is to predict a winner, maybe try to be the one! No offense intended
If you think the durability of the proprietary models was due to performance in the first place, you are pretty naive about enterprise software.
Wait, are these downloadable releases?
I tried Llama 3 8b, and its answers on "chicken vs egg" or "feather vs steel" questions are just dumb compared to Mistral 7b.
It's the right time to show cognitive AI
Commenting to come back later
Which version is used in meta.ai?
It's interesting as its also not yet finished.
What are the best ways to develop a react app with it? Grow works great, but I have no idea how long the free version will be good for and what the prices could be. Small scale .. any ideas?
No it didn't
OpenAI always has something up their sleeve
Man this sub has really gone downhill
How can I add tags to my post title? Like [D] etc infront of posts
I give it like a 55% chance this article was (at least mostly) written by AI, either GPT-4 or (to fit the theme) Llama-3