MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch.
So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices (and this is ofc. 0.1 or smth like that so maybe we'll get something working fine in next few years).
Also it'll be cool to compare Tensorflow vs PyTorch on metal device :)
MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch. So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices
Also, even for ops that do not crash, it often returns garbage. I built PyTorch from git this morning:
Many bugs will probably be squashed over the coming weeks, so I am still very exited about MPS support. I have done some preliminary benchmarks with a spaCy transformer model and the speedup was 2.55x on an M1 Pro. Which is quite nice because transformers were already very fast on M1 Macs, thanks to the AMX units (PyTorch links against Accelerate, so the AMX units are used for matrix multiplication). If they can squeeze more performance out of M1 GPUs in the future, it will be a very nice speedup.
Weeks :?
Looking at every other project from Apple that try to integrate with some opensource lib it'll be months/years if they didn't drop support for it before that :P
Did you get only 2.55x on BERT/other transformer vs CPU version?
I think in this case the PyTorch team also involved. They have already fixed some annoying bugs in the last two days (like matrix multiplication often failing because of buffer size mismatches).
As for BERT inference performance, I am not sure what kind of speedups you are expecting. The M1 Pro gives me 2.6 TFLOPs in single precision matrix multiplication of 768x768 matrices. M1 Pro GPU performance is supposed to be 5.3 TFLOPS (not sure, I haven’t benchmarked it).
What ML are people actually wanting to do on M1 laptops?
I was under the impression that outside toy problems, you can't really get decent results unless you have a TPU or fleet of GPU's for training. If you just train on a laptop you're shooting yourself in the foot compared to someone who gets the same job done 10x faster in the cloud (usually with free credits).
Obviously inference is still possible on a laptop for many models... but not many 'hacker' tasks involve only inference.
I’m not using an M1 machine yet, but I think is extremely useful to be able to do some small scale experiments for testing and debugging purposes locally before starting a full-scale training job in the cloud.
Further, in general, I’m really charmed by the potential of having unified memory; the idea you can test some batch training iterations with a batch size that fills almost all 128MB is a unique capability!
Last, a large amount of unified memory allows to do inference and “prompt engineering” with very large models, locally. E.g. Using GPT-J like models (6B parameters).
What ML are people actually wanting to do on M1 laptops?
(1) development - quickly train smaller networks locally to validate the code. Then offload to a big GPU machine. (2) Smaller convolutional networks and RNNs may be below SOTA, but can still provide a good performance-accuracy trade-off. You could already train these networks pretty fast on an M1 thanks to AMX, but his makes training such networks even faster.
>I was under the impression that outside toy problems, you can't really get decent results unless you have a TPU or fleet of GPU's for training.
This is most certainly not the case. It's impossible to achieve SOTA on most benchmarks but there's a lot more ML and data analysis done on smaller datasets where you don't particularly need a fleet of GPUs.
The whole Diffusion text to image thing which is hugely popular right now doesn't involve any training. Same for text generation where you download pre-trained models. I'm using a laptop with a 3080 and getting much faster results than I typically get from a paid Colab account (not Pro mind you).
It handles (most) of the installation issues and gives you a GUI. You can however bypass the GUI. Every time you run a command the console shows you what the command line looks like so copy/paste into a terminal and away you go.
There's plenty of training related stuff that 10 TFlops is sufficient for. Fine tuning models for example is a great use case for a high ram yet relatively low TFlops environment.
I don't know how much you've done with running ml jobs on cloud boxes but it can be very annoying to work with them when you are actively iterating. Especially if you are doing a lot of work outside the cloud box environment. Even turn key consumer products like Colab can fall over if you accidentally press "Back" in your browser... I once lost many hours of work because I inadvertently swiped left :)
Apple will continue improving their performance in this space and once jobs that run for hours can be run overnight... I suspect people will often choose to run overnight simply for the ergonomics benefit.
Totally, performance for actual useful training etc. is at least a couple of hardware generations away. This is super useful however for people like me who like to use an m1 Mac as their daily driver and don’t always want to have to fire up a remote rig just to sketch a little. Also being able to do some local inference can be great for when building a web GUI, etc.
So absolutely not mature, but having PyTorch working better also on macs is only positive imho.
Can someone explain to me why M1 Ultra doesn’t scale ratably compared to RTX? It seems it should perform at about 1/2 of 3090. Is it because Apple locked down what you can do with the GPU?
How so? NVIDIA GPUs have their own APIs, AMD does, Direct3D is its own. The Apple platform acceleration APIs are Metal. The MS acceleration APIs are all generally Direct3D, etc.
The fact the you want Apple to support other platforms doesn't mean that they're wrong for not doing so. They same arguments to require Apple to support your preferred API also applies to MS, NVIDIA, AMD, etc and the same arguments to not do that apply them all as well.
That other companies use shitty proprietary code to lock out competition doesn't make Apples attempt to do the same any better. It compounds the error and wasted effort. In fact Apple's is the worst of the lot you mention because at least you can use alternatives with the others.
Apple’s is the result of their experience promoting OpenGL and OpenCL. They found GL and CL both remained resolutely in the past, and people also seemed resolutely opposed to using them.
So GL and CL had to make a compelling argument for why apple should continue to invest in the technology, and even now they still aren’t able to show a single compelling argument for them being better than any alternative.
Personally I would have rather a single open API, but the open API did not appear to be super interested in competing with the other platforms. The end result is the open platforms turn into a pile of expense, licensing, and engineering for behind the curve technology and API.
They don't have to develop it. They just have to not prevent it from existing. If theirs is genuinely better, it will get used. But Apple is too scared to openly compete so they lock down and artificially limit competition.
Anyone can implement the opencl and OpenGL APIs on top of metal. Apple isn’t preventing that.
But native OpenGL requires apple writing OpenGL implementations for their hardware. It requires apple spending engineering time implementing misfeatures from OpenGL and opencl in order to support APIs that are only every considered the backup API.
What about Vulkan? They’d need to implement the same crummy APIs and semantics (that is a bunch of kernel and similar work) for an API that NVIDIA and AMD were both uninterested in.
And again, anyone could have made a Vulkan emulation layer on metal.
On the contrary, I believe MacBooks have continued to eat general market share since they switched to Apple Silicon, and they are by far the best built UNIX-based laptop you can get, and have by far the best battery life. There’s a reason so many devs (and scientists, like myself) use them.
They're the fourth largest laptop vendor, making 9% market share, up from 7% the year prior [1]
Apple is #2 world wide for phones at 18% [2]
Apple is #1 for phones in the US with 51% share [3]
Apple is #2 for phones in the EU at 23% share [4]
Apple's laptop seem to wipe the floor in power usage, while not getting anyway more in gaming land than any prior generation [5]
Apple's revenue is the 3rd highest in the world which also doesn't seem particularly irrelevant.
You can hate Apple all you want, but going "I don't like apple, therefore the company is irrelevant" is a dumbass approach. Hell what will android get without finding out what's in iOS?
Apple aggressively pushed openCL when it was new, and NVIDIA and AMD aggressively pushed their platform specific CUDA APIs, so all anyone did was complain about apple using openCL when no vendors supported it.
Apple had (runtime and ABI) stable openCL built into macOS and iOS by default and on desktop devs kept requiring you install NVIDIAs or AMDs CUDA systems, and on iOS everyone found that openCL was just as clunky to use as openGL.
Apple made proprietary APIs that were more usable and better interacted with the hardware, and everyone just used those instead, until Apple deprecated openCL and openGL as they merely represent a pile of engineering work and support that is essentially unused.
https://news.ycombinator.com/item?id=31424048
Accelerated PyTorch Training on M1 Mac (pytorch.org)
436 points by tgymnich 2 days ago 143 comments