Instead of using five masters like GFS, which typically ran on five machines dedicated to the purpose, Colossus uses a variable number of so called curators to handle all the metadata work (think of directory entries and inodes in Unix, separate from the data blocks). They're just regular Borg tasks, running along those from other services like YouTube, Gmail, Reader, etc. The more storage added to the cluster, the more traffic it gets and the busier the curators become. For very small clusters five or fewer curators might suffice, but when you hit hundreds of petabytes, you need many, many more of them. Borg requires identities, including the system account that runs the Colossus service, to have quota, so there needs to be a way to grant it an amount of CPU cores and memory somewhat proportional to the storage cluster size. (Fun things happen when the cell shrinks instead of growing.) Monitoring the entire stack also requires resources of its own, proportional to cluster size, so there were additional grants at higher priority just for those jobs.
I've always wondered how Google does this internally.
For all the cloud companies, "quota" is literally dollars. Bandwidth, memory, cpu, these are things they sell. So they are measured in money. The existing corporate budgetary process already manages dollars (indeed, that's its main function). If you want more of one of these things, you'll be displacing some customer who stood ready to pay for them (i.e. on the spot market), so you need to replace the revenue they were providing. Very straightforward.
What does google measure all these things in? Quotons? Where do quotons come from? When google becomes more profitable, does the Central Bank of Quoton print more quotons each month than they had previously? Does Google have a Chief Quoton Officer?
Totally curious about this. Google is the only company operating on this scale that doesn't sell raw bandwidth/cpu/ram as their product, nor do they buy these things from underlying cloud vendors (like e.g. Cloudflare does). This makes a lot of things simpler internally and probably avoids a lot of internal conflicts, but it does mean that they have to have a parallel internal accounting system for this stuff.
Is there an official exchange rate between the two accounting systems? Are different teams allowed to sell each other quotons, in exchange for budget dollars? Has the IRS ever audited Google's Quoton books?
Okay that last question was not serious. But the rest were.
This is what I was wondering too. I suppose it's useful because it separates internal demand from external prices, so internal teams can be compared with a common currency, and planning can happen on that basis, and then totally different teams can care about how to purchase compute/storage/etc.
But how do they decide when to expand a datacenter or build a new one?
The demand is priced in quotons but the costs are in dollars.
Somebody somewhere at google must be maintaining a quoton/dollar exchange rate. It's weird that it's hidden from the rest of the company.
> so internal teams can be compared with a common currency
What's wrong with comparing them using dollars (or euros or whatever) as the currency? That's what Google's revenues are measured in terms of. If the price of electricity doubles (like it has in the EU) a team with euro-constant revenues could swing from being highly profitable to a massive loss... and nobody would notice because quotons conceal the cost of electricity.
What I think the GP means is previously, GFS ran on their own machines and didn't need to participate in the Google-wide quota system for CPU/RAM.
However Colossus _does_ run on the general Google compute infrastructure, and so needed a way to get CPU/RAM quota where none existed (because it was all used by existing, non-storage users).
I've no idea what this is, but it sounds fascinating. Was this a way to auto-allocate more resource based on rules?