What I Wish I Had Known Before Scaling Uber to 1000 Services • Matt Ranney • GOTO 2016

День тому

This presentation was recorded at GOTO Chicago 2016. #gotocon #gotochgo
gotochgo.com
Matt Ranney - Chief Systems Architect at Uber, Co-founder of Voxer
ABSTRACT
To Keep up with Uber's growth, we've embraced microservices in a big way. This has led to an explosion of new services, crossing over 1,000 production services in early March 2016. Along the way we've learned a lot, and if we had to do it all over again [...]
Download slides and read the full abstract here:
gotocon.com/chicago-2016/pres...
RECOMMENDED BOOKS
Sam Newman • Monolith to Microservices • amzn.to/2Nml96E
Sam Newman • Building Microservices • amzn.to/3dMPbOs
Ronnie Mitra & Irakli Nadareishvili • Microservices: Up and Running• amzn.to/3c4HmmL Mitra, Nadareishvili, McLarty & Amundsen • Microservice Architecture • amzn.to/3fVNAb0
Chris Richardson • Microservices Patterns • amzn.to/2SOnQ7h
Adam Bellemare • Building Event-Driven Microservices • amzn.to/3yoa7TZ
Dave Farley • Continuous Delivery Pipelines • amzn.to/3hjiE51
/ gotochgo
/ gotoconference
#Microservices #ScalingSoftware #scaling
Looking for a unique learning experience?
Attend the next GOTO Conference near you! Get your ticket at gotocon.com
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
ukposts.info...

КОМЕНТАРІ: 123

@OgnyanDimitrov 4 роки тому

Plus one for admitting a lot of hard truths in such a diplomatic and polite tone. Scaling from 200 to 2000 for a year sounds like a monstrosity to me! - Whoa! Great job! I want to greet the one who was in charge of this situation!

@RobertWierzbowski 7 років тому

On rationalizing "using your baby" vs switching to a better supported service, I like the quote on writing by William Faulkner: "You must kill your darlings". We get attached to complex, clever, difficult code, but the best code is dull, simple, unclever, or non-existant.

@SenorQuichotte 6 років тому

you've probably never wrote a killer perl one-liner

@joshpetit 11 місяців тому

@@SenorQuichotte ahaha no one would doubt my 200 character awk one liners are a piece of work.

@_stokyo_ 2 роки тому

The takeaways for me were: Own your uptime Use a common profiling shape that works for all used programming languages Configure distributed tracing Use a common metric dashboard Consistent logging Performance IS important even if it's a small subset of a call chain Stop logging everything

@cupofkoa 2 роки тому

Honesty is incredibly valuable.

@dac514 7 років тому

Step 1) Hire two thousand developers. Step 2) ??? Step 3) Profit.

@webmastergrumpy 7 років тому

Great story, great presentation.

@leozilla 7 років тому

one of the best presentations about the danger of (micro)services

@Mike-rt2vp 5 років тому

@@qm3ster ((((:-))))

@csvegso 3 роки тому

I fully agree with all the statements. We faced all of these issues. One by one. In the painful ways.

@revmusical 5 років тому

Great content , pretty much captures every pitfall of using the microservices

@jenesuispluslameme 7 років тому

Great talk from experience with microservices - thanks

@ChuckJHardy 7 років тому

Great talk, well done.

@MargaretLeber 3 роки тому

"Immutability changes everything."

@PratyushAgarwal 3 роки тому

One of the key takeaways (that I didn’t see in other comments) was the point regarding tradeoffs. Everything that is being decided (small or big) has tradeoffs. Thus, decisions need to be taken deliberately instead of just accepting it as the way things are headed.

@richardhp77 2 роки тому

Yeah that was cool. Make your trade offs tailored to your organisation so the bad stuff doesn't have such a big impact

@globetrotter373 Рік тому

Architecture 101

@web3tel 6 років тому

Great presentation

@fnjordy 7 років тому

Multiple languages for services are advantageous as it encourages a better separation of the transaction. The challenge is an environment with bias on one language that leads to 2nd tier broken support for the other languages as no one wishes to fix the dominant platform when it errs. Much like RFCs vs. an implementation of a protocol being the reference.

@KangoV 2 роки тому

Very nice talk. Eclipse allows cross cutting changes. IntelliJ does not. I really miss workspaces.

@dardanbekteshi3177 2 роки тому

Just realized we have the exact same problems like Uber, only thing that differs is we don't have Uber as a product 😂

@transparentpolitics 7 років тому

Good talk. Politics is an "obvious" (to me) of teams owning their *own* uptime -- not uber's uptime as a whole. Suggestion: I have this same problem : you were not looking at the audience and engaging with your eyes. Lift up your head and not down at the monitor all the time.

@vanhoangtran1270 5 років тому

thanks for sharing

6 років тому

ge looks so happy!

@rickardoberg 7 років тому

I think a majority of the issues brought up could be minimized by using EventSourcing rather than REST as the internal distributed system pattern. A service is responsible for writes, and letting other services consume those through e.g. feeds (using Atom, this would still be RESTful btw). Those dependent services can then project those events into whatever database they need locally, and do not depend on uptime or scalability of the other service. This also minimizes need and impact of migrations, as all databases are just projections of events, and can be rebuilt by simply deleting them (we do this in production all the time, in a rolling fashion). Blue/green deployments is also an option (and VASTLY simpler using EventSourcing) to constantly do major upgrades to services without downtime.

@MartinMortensen 5 років тому

I don't even know where to start. But eventaorucing and microservices is by no means less complexity.

@richardhp77 2 роки тому

@@MartinMortensen event sourcing brings it own demons, but in this particular use-case, based on what uber are trying to achieve with rapid development of decouple features, you could make the argument that event sourcing would be a better trade off than distributed micro services

@baptistepattyn7555 2 роки тому

@@richardhp77 Event sourcing does not exclude microservices.... on the contrary

@AlexKavanagh29x 7 років тому

Politics - succinctly - is the process of deciding who gets what, when and how. (Harold Lasswell) .. whenever there are scarce resources (including people), and people fight over them, politics is what happens.

@thunderloong 7 років тому

Politics is what happens when people agree not to kill each other over resources. Scarcity has no validity. Distribution has no validity.

@richardwalker3760 7 років тому

Strongly disagree. *Economics* is the process of deciding who gets what, when and how. Politics only matters when the government tries to control the economy.

@anis.lounis 7 років тому

Well said!

@MattMcCormick1 6 років тому

This is a much better definition. I've heard the Company > Team > Self before and always wonder a) where is the end customer in that equation because they are what really matters and b) there are plenty of cases where the self's decision could go against the company but it would be the right one (ie. illegal behaviour at the company)

@tauruscolvin 5 років тому

Insightful talk -- thank you. Regarding Matt's question: 'How can we safely insert the "ride with a puppy" step into the Uber architecture?'. Perhaps a dab of functional programming? That is, we assert that each microservice should be a composition of (pure) functions -- aka "pipe". These "micro-pipes", would give us the *synchronous* composability we want while preserving *asynchronicity* in the space between the services achieved through event sourcing. Google seems to call this pattern "functional microservices".

@ArchimedesTrajano 5 років тому

43:29 I go with the notion that politics specifically "playing politics" is the notion of utilizing people to achieve a goal but making it so that the people that are important to you are satisfied. Bad politics reduces the amount of people being satisfied, good politics conversely increase the number of people satisfied.

@TheBookDoctor 7 років тому

Great talk. Interesting how many of the problems you discuss really do boil down to the "politics" category of human problems: emotional sunk-costs arguments for sticking with the known-but-broken vs. taking the cost of switching to the new-but-not-broken. Or as Douglas Adams said, "People are a problem."

@mennovanlavieren3885 3 роки тому

29:37 Ah, ORM induced performance problems.

@rftluong 7 років тому

Great talk ... the number of ads in it are ridiculous though.

@MarshallSorenson 7 років тому

tldr; like, we thought micro-services meant nano-services, so like, that's bad.

@vincentmax4571 5 років тому

Cool stuff

@Skorps1811 4 роки тому

That's a COVID-like growth of microservices there

@HoldOffHunger 7 років тому

"Don't bring out the sticks. Carrots are the only way to go." Nailed it.

@RayBrooks0 7 років тому

Matt, such a great presentation! Also, you have tape on your face.

@omarmoataz 4 роки тому

I watched 21 minutes of this talk and I still can't figure out if I learned anything.

@_stokyo_ 2 роки тому

Can someone explain what cross language context propagation means? He talks about passing context down to intermediary calls but I don't grasp what the alternative is?

@HaithamElGhareeb 7 років тому

One of the most interesting Microservices sessions I ever saw at all. Thanks a lot.

@eugeneleroux6140 6 років тому

Lots to think about, thanks.

@jabberl. 4 роки тому

node, go, java, python - "Wow, that's like a lot of languages!" LMAO

@reespozzi4334 2 роки тому

It is. Less the better

@bigdlamz 3 роки тому

The is what happens when you embrace microservices with all its buzzwordy glory. Some decisions need to be centralized eg decomposition & language to use.

@Pjblabla2 Рік тому

Very informative It was unusual to see someone with the chief systems architect role in modern IT organisation It'll be good to get an understanding of different roles within Uber's IT organisation and how it is structured E.g. are these just his observations in retrospect or does he have the authority to influence future decisions based on these insights

@Silmarieni1 7 років тому

Isn"t a bunch of criticisms of microservices in fact a failure to choose the right size for a microservice ?

@TheNiters 5 років тому

10x growth in engineers in 1 year is probably too fast... but might be hard to avoid

@pkeshava 7 років тому

Pardon my ignorance. What is WIWIK?

@dn5426 7 років тому

"What I Wish I Knew"

@boltonkim7090 4 роки тому

My thoughts after watching this video: 1. Are engineering teams at Uber comprised only of junior engineers and fresh grads? or 2. Maybe Uber doesn't have much of an engineering management culture? or 3. Maybe Uber's product organization wants things moving so fast that the engineering organization has to throw common sense away?

@Divinii 4 роки тому

10:55 Wow! *"By having multiple languages, it can fragment the culture."*

@rentsy3444 4 роки тому

don't do it

@Divinii 4 роки тому

@@rentsy3444 Thanks!

@tsingylu 4 роки тому

I can feel his frustration

@n8style 5 років тому

"It's so hard to get a count of how many services are running because it's changing so rapidly" Wait what, you don't have an automatic count you do it manually?!?!

@waleedjubeh5955 Рік тому

When I hear "Buying new computers (scaling) is cheaper than hiring software engineers." I have shocked about this fact. Bom!

@chashdeveloper 7 років тому

He is an excellent presenter. I have attended one of his talk in 2015

@saidimuapale 7 років тому

Subtext of this talk: the Uber Engineering mega-team is fairly dysfunctional.

@JulienCROUZET 7 років тому

Plot twist : After reading a comment on UKposts from a random guy with no real experience of high traffic they decided to change everything because I thinked that they're "fairly dysfunctional"

@jerrygreenest 3 роки тому

Double Plot Twist: After they noticed another guy used the word “thinked” in his arguments, they reconsidered what the else random guy is saying, and indeed fired half of people

@-Jason-L Рік тому

Approaching microservices like RPC is why you will have so many issues. It's a mistake made by folks rooted in older distributed system approaches.

@tux1968 7 років тому

@Beebop121 7 років тому

I really dodged a bullet staying away from Silicon Valley.

@charlessharpe2121 7 років тому

Why not use an event-driven, non-blocking I/O to speed up REPO synchronization, aka nodejs? Is the company focus dependent upon centralized or decentralized philosophies?

@christopherstamp9716 7 років тому

Charles B. Sharpe it's not that easy. They obviously use node js but node js is single threaded and can run into similar scaling problems as any language. The complication is concurrency/caching and cpu metrics/resource management. Very complicated environment

@tscan89 7 років тому

Seems like they need a new microservice to coordinate all their microservices. :)

@sahild6584 7 років тому

Tim Scanlin nice sarcasm

@Calphool222 6 років тому

Actually, Netflix has a presentation where they talk about that. Even when you're doing a microservice based architecture, you end up having some services that are far more critical than others. Aspects of monolithic systems begin to re-emerge. In monoliths we addressed these "critical components" with various high availability tricks. Netflix took a slightly different approach. They built Hystrix to provide systemic fallback and short-circuiting of cascade failures, which are an emergent property of a microservice based architecture.

@hariveturi4193 4 роки тому

@@Calphool222 I happened to stumble onto this talk after the one Netflix one :) I know it has been a long time since this comment but still. Really loved the netflix talk. Wish they would've done QA later.

@minyakonga8897 3 роки тому

best have so many defs, like best libs, best cuz i know it, or best ....

@beedeeuniko 4 роки тому

the frequency of him saying "like", "kinda", "sort of", "totally" and "you know" is - like - staggering, you know?

@Skorps1811 4 роки тому

is like kinda sort of totally staggering, you know?

@georgesmith3022 4 роки тому

So somebody discovered distributed systems again, called them microservices, but nobody seems to know how to make them. Is it another fad that will go away?

@electroman7396 4 роки тому

Uber does no profit and just pumped up and hyped. Why is everyone taking it as an example?

@nickbarton3191 4 роки тому

I'm surprised that he's not a jibbering wreck of shredded nerves.

@DodaGarcia Рік тому

He kind of is though

@minyakonga8897 3 роки тому

everything is a tradeoff, everything is a choice

@ahmedbadia1485 4 роки тому

this guy talks like Richard from silcion valley, guess the show was accurate after all

@billmoney1 7 років тому

sup

@szym1 7 років тому

like like like like like like I can't bear it over 13:25

@dieterplaetinck4431 7 років тому

me too. i gave up on this presentation. I couldn't stand the "street" talk. like like you know like

@jake8217 Рік тому

All of these sounds like Uber problems than microservices problems.

@dotanoob466 4 роки тому

1. break laws 2. have a toxic culture 3. get a-holes to invest in an app made by ppl who end up doing all the work but getting no credit.

@kobac8207 6 років тому

Thanks for this presentation and for a completely unexpected reason. I thought that guys doing architecture at Uber actually know to do architecture. Who the fuck scales to 1k microservices in a year??? Do they even have a clue how and when to separate a microservice? Reason for a new microservice is a new bounded context. How in the hell did you manage to find 1k bounded context in the app that routes the vehicles for taxi rides? Uber is a distributed ball of mud. This guy is an antipattern himself.

@olegzandrvondenmanoresoftw596 5 років тому

How many "likes" can you say in a minute lulz

@m424t 7 років тому

This was a very disappointing video. Unfortunately it seems like Uber architecture was designed without much thought in a way junior devs would approach it. Given the very basic nature of problems they're having, I am surprised that Matt hasn't mentioned problems with transaction support and REST API versioning.

@mensaswede4028 6 років тому

So Uber is a great business idea, but as usual it was poorly implemented. But it succeed in spite of its poor implementation because it was a great business idea. The world is not a meritocracy.

@Teslawaverunner 3 роки тому

Interesting how people have one personality during structured presentations and another when asked questions: the way the presenter dealt with the questions at the end was rude and condescending and probably gives an insight into the culture present at Uber

@anujpandey8753 2 роки тому

too high level and too abstract. Touching so many things at a time and not diving into details. This presentation could have been better.

@nelsonc5339 6 років тому

Wow...! Are the *investors* watching...?! this is _like_ a *Technical Debt Time Bomb* of a "stack"! Uber is going to *implode* so spectacularly... 😧

@DavinStewart 5 років тому

Like, please stop saying like, like all the time.

@cantwaittowatch 4 роки тому

rambling so many things..yikes!

@hoopssheaffer 7 років тому

This guy doesn't seem like a very good engineer.

@stoneshou 6 років тому

previous CTO of Voxer

@Calphool222 6 років тому

Huh? He's pulling back the curtain and describing the difficult problems Uber is facing, most of which were emergent properties of being engineering purists. The ignorance of Conway's Law and a hundred other systems thinking phenomena in engineering circles is epidemic. Code slinging is an important baseline skill that everyone needs to have, but it's *not* what it takes to lead armies of technical people, and having intentionally ignorant HR managers is *not* a solution.

@tj8870 6 років тому

It's the complete opposite. This is how a great engineer thinks. Seeing the trade-off between all the different types of methodology, technology and architectures. It is impossible to design a fully working system at a large scale. Every layer you add increase the complexity of the project exponential not matter what architecture you use or how great your starting UML was. Trade-offs are constantly added to the ones you were already facing. On top of that you are building off the previous architecture. Any flaw in the system or outdated methods or technologies only get propagated with growth. Small scale architecture is ridiculously simple relative to large scale architecture. Personally, I often find that maintaining an MVC project with 1 or 2 teammates is more time consuming and produces worse results then just coding it by myself and they are handling 2000 coders... it is crazy...

@MrHics 7 років тому

regular ads? downvoted

@suckerformimi 4 роки тому

This conference is crap

@SC1240 3 роки тому

This dude can't present well at all. Decent info though.

@ralfoeldi 6 років тому

After 15 minutes... why isn't this guy sooo embarrassed that he crawls under a rock and hides? THIS guy is responsible for anything? Sounds like a junior engineer after having completed his first project (which might be close to the mark).