Deploying machine learning models on Kubernetes

  Переглядів 12,885

mildlyoverfitted

mildlyoverfitted

День тому

In this video, we will go through a simple end to end example how to deploy a ML model on Kubernetes. We will use an pretrained Transformer model on the task of masked language modelling (fill-mask) and turn it into a REST API. Then we will containerize our service and finally deploy it on a Kubernetes cluster.
Code from the video:
github.com/jankrepl/mildlyove...
00:00 Intro
00:22 3 step procedure diagram
01:42 Existing framework overview
02:09 Creating an API
09:25 Containerization
13:53 Containerization - custom platform
15:47 Preparing a minikube K8s cluster
17:43 K8s: Deployment and service
21:31 K8s: 2 cool features - self-healing and load balancing
26:00 Outro
If you have any video suggestions or you just wanna chat feel free to join the discord server: / discord
Twitter: / moverfitted
Credits logo animation
Title: Conjungation · Author: Uncle Milk · Source: / unclemilk · License: creativecommons.org/licenses/... · Download (9MB): auboutdufil.com/?id=600

КОМЕНТАРІ: 45
@ludwigstumpp
@ludwigstumpp Рік тому
Always a pleasure to watch someone as talented as you! Keep it up :)
@mildlyoverfitted
@mildlyoverfitted Рік тому
Wow, much appreciated:) Thanks:)
@abdjanshvamdjsj
@abdjanshvamdjsj 9 місяців тому
Brooooo this was so good.
@mildlyoverfitted
@mildlyoverfitted 9 місяців тому
Glad you liked it!
@JoseMiguel_____
@JoseMiguel_____ Рік тому
You're great. Thanks for sharing this in such a nice way.
@mildlyoverfitted
@mildlyoverfitted Рік тому
My pleasure!
@davidyates4857
@davidyates4857 10 місяців тому
Great video very informative.
@mildlyoverfitted
@mildlyoverfitted 10 місяців тому
Glad you liked it!
@thinkman2137
@thinkman2137 5 місяців тому
Thank you for detail tutorial!
@thinkman2137
@thinkman2137 5 місяців тому
But torchserve now has kubernetes intergration
@mildlyoverfitted
@mildlyoverfitted 5 місяців тому
I will definitely look into it:) Thank you for pointing it out!!
@maksim3285
@maksim3285 9 місяців тому
Thank you, it helped me a lot .
@mildlyoverfitted
@mildlyoverfitted 9 місяців тому
Happy to hear that!
@fizipcfx
@fizipcfx Рік тому
he is back 🎉
@kwang-jebaeg2460
@kwang-jebaeg2460 Рік тому
OH !!!!! Glad to meet you again !!!!
@mildlyoverfitted
@mildlyoverfitted Рік тому
Glad you are here:))
@aditya_01
@aditya_01 4 місяці тому
great video thanks a lot really liked the explanation !!!.
@mildlyoverfitted
@mildlyoverfitted 4 місяці тому
Glad it was helpful!
@vishalgoklani
@vishalgoklani Рік тому
Welcome back, we missed you!
@mildlyoverfitted
@mildlyoverfitted Рік тому
Hehe, thank you! Nice to hear that:)
@alivecoding4995
@alivecoding4995 Рік тому
I agree!
@johanngerberding5956
@johanngerberding5956 Рік тому
very cool video!
@mildlyoverfitted
@mildlyoverfitted Рік тому
Thank you! Cheers!
@user-cp1pe2tx7h
@user-cp1pe2tx7h Рік тому
Great!
@lauraennature
@lauraennature Рік тому
New video 🤩
@user-ds5sh9uj7o
@user-ds5sh9uj7o Рік тому
Would appreciate a video using VScode to include docker contain files, k8s file and Fast API
@nehetnehet8109
@nehetnehet8109 10 місяців тому
Great
@nehetnehet8109
@nehetnehet8109 Рік тому
Realy goood
@evab.7980
@evab.7980 Рік тому
👏👏👏
@unaibox1350
@unaibox1350 Рік тому
Amazing video. In min 5:25 how did you do to open the second bash in the console? I was searching for a long time and I can't find anything. Thanks and regards!
@mildlyoverfitted
@mildlyoverfitted Рік тому
Thank you! You need to install a tool called tmux. One of its features is that you can have multiple panes on a single screen.
@unaibox1350
@unaibox1350 Рік тому
@@mildlyoverfitted Thank you! Will dig in it now
@davidpratr
@davidpratr 2 місяці тому
really nice video. Would you see any benefit of using the deployment in a single node with M1 chip? I'd say somehow yes because an inference might not be taking all the CPU of the M1 chip, but how about scaling the model in terms of RAM? one of those models might take 4-7GB of RAM which makes up to 21GB of RAM only for 3 pods. What's you opinion on that?
@mildlyoverfitted
@mildlyoverfitted 2 місяці тому
Glad you liked the video! Honestly, I filmed the video on my M1 using minikube mostly because of convenience. But on real projects I have always worked with K8s clusters that had multiple nodes. So I cannot really advocate for the single node setup other than for learning purposes.
@davidpratr
@davidpratr 2 місяці тому
@@mildlyoverfittedgot it. So, very likely more petitions could be resolved at the same time but with a very limited scalability and probably with performance loss. By the way, what are those fancy combos with the terminal? is it tmux?
@mildlyoverfitted
@mildlyoverfitted 2 місяці тому
@@davidpratr interesting:) yes, it is tmux:)
@unaibox1350
@unaibox1350 Рік тому
I am having a problem in the min 18:00 the model load is being killed all the time. I tried to "minikube config set memory 4096" but still having the same problem. Any idea? I've been looking for a solution for 3 hours and there is no way
@mildlyoverfitted
@mildlyoverfitted 11 місяців тому
Hm, I haven't had that problem myself. However, yeh, it might be related to the lack of memory.
@zhijunchen1248
@zhijunchen1248 9 місяців тому
Hi, I would like to use GPU to accelerate this demo, can you give me some tips? Thank you
@mildlyoverfitted
@mildlyoverfitted 9 місяців тому
So if you wanna use minikube this seems to be the solution. minikube.sigs.k8s.io/docs/handbook/addons/nvidia/
@zhijunchen1248
@zhijunchen1248 9 місяців тому
@@mildlyoverfitted thankyou, i use the "--device" flag of transformers-cli to enable GPU. And I found that serving app takes up almost gpu memory and no compute power. Whatever, thankyou for your video!
@alivecoding4995
@alivecoding4995 Рік тому
What terminal application is this, with the different panels?
@mildlyoverfitted
@mildlyoverfitted Рік тому
tmux
@kwang-jebaeg2460
@kwang-jebaeg2460 Рік тому
Look forward to show your face alot :))
Machine Learning on Kubernetes | Salman Iqbal
25:45
Kubernetes Community Days UK
Переглядів 2,4 тис.
Creating an Ubuntu WordPress Server - EP3: Installing PHP
25:33
Vasiliy Lomachenko vs George Kambosos | INTERNATIONAL LIVE STREAM
3:10:05
Top Rank Boxing
Переглядів 1,1 млн
Building the Ultimate Workout Tracker with React Native & MongoDB
3:59:34
notJust․dev
Переглядів 257 тис.
Do NOT Learn Kubernetes Without Knowing These Concepts...
13:01
Travis Media
Переглядів 202 тис.
Deploying ML Models in Production: An Overview
14:27
Valerio Velardo - The Sound of AI
Переглядів 37 тис.
Running Generative AI & LLM on a Kubernetes Cluster | Cloud Institute
30:32
Cloud Institute
Переглядів 4,2 тис.
Kubernetes 101: Deploying Your First Application!
40:47
DevOps Directive
Переглядів 42 тис.
How to Deploy ML Models in Production with BentoML
34:15
Valerio Velardo - The Sound of AI
Переглядів 15 тис.
Docker for Machine Learning
13:50
Aladdin Persson
Переглядів 7 тис.
Портативная PS 5 🎮 #ps5 #expressly
0:22
ExpresSLY Shorts
Переглядів 296 тис.
Why spend $10.000 on a flashlight when these are $200🗿
0:12
NIGHTOPERATOR
Переглядів 17 млн
M4 iPad Pro Impressions: Well This is Awkward
12:51
Marques Brownlee
Переглядів 5 млн
Apple Event - May 7
38:22
Apple
Переглядів 6 млн
Fiber kablo
0:15
Elektrik-Elektronik
Переглядів 1,7 млн