Stuff The Internet Says On Scalability For November 30th, 2018 - High Scalability -

 highscalability.com  11/30/2018 16:59:42 
  • Quotable Quotes:
    • @william_r_kerr: "The number of new international students enrolling at American institutions fell by 6.6% during the 2017-18 academic year, on top of a 3.3% decline the year before."
    • @codinghorror: “Apple’s silicon is now so well ahead that we’re not really expecting Android vendors to catch up any time soon.” 
    • @emileifrem: Adobe used 125 MongoDB servers to run their activity feed. It was replaced by 48 Cassandra servers. Now it runs on 3 (THREE!) servers of Neo4j. With more data (yet smaller disk footprint), higher load and more functionality. The scalability of a native graph database.
    • @swardley: First up at #AWSreInvent is McKinsey revisiting their 2009 report on Cloud Computing with their talk "We were so wrong, you'd have been a muppet to follow our advice" ... damn, daydreaming again. X : What did McKinsey do? Me : Clearing the Air on Cloud, 2009. Basically an argument that cloud was for startups but proper enterprises would be better off investing in their data centres and adopting a cloud model would be a money-losing mistake 
    • @AstroKatie: Hold your hand up 12 inches from your face: you’re seeing your hand as it was a nanosecond ago. Everything you look at is, to one degree or another, in the past. The farther away in space, the more ancient in time.
    • Ms. Hou: I used to think the machines are geniuses. Now I know we’re the reason for their genius.
    • @lightbend: How to scale? Here you go: With over 125 million players, and supporting over 8 million *concurrent gaming sessions*, we are really happy to learn that #Fortnite is running #Akka under the hood!
    • @mattklein123: The Firecracker announcement from @awscloud is super, super cool. I love seeing real innovation in the OS/VMM space, and a willingness to toss away legacy in order to vastly simplify the problem space. Kudos on making this OSS as well.
    • @hichaelmart: So my hot take is that Firecracker is gVisor done right: at the VM level, and more importantly, NOT using a garbage collected language 🔥
    • @profgalloway: More Americans have Prime than go to church or own a landline. #thefourbook
    • @mjpt777: SBE 1.10.0 is out. Java codecs see ~25% performance bump with this release and more if using G1 GC (those GC write barriers can hurt!).
    • @mattbeane: AI platform finds accident predictors in Waze data, gives drivers real-time feedback, 90% of drivers slow down, accidents down nearly 20%. Seems like benefit might extend to non-Waze users too, but they don’t say...
    • @anliguori: No BIOS in Firecracker.  We are expressly not trying to support legacy OSes.  The device model ends up being extremely simple.  No live migration either because serverless workloads are short lived.  We really want to explore this space deeply and not get tied down by legacy.
    • David Gerard: The price peak [for bitcoin] was December 2017 — but the hash rate was six times that by August 2018, at one-third of the price. Mining 1 BTC cost way less than 1 BTC during the bubble. So, to compete, miners built out big. The hash rate changes approximately every two weeks. But capital expenditure — building and deploying single-purpose mining hardware — has a rather longer lead time.
    • @michael_at_work: AWS has done so much for the world, and there’s so much money in OSS, it only makes sense that OSS give a little something back to AWS to help support them going forward, especially now with AWS in a financial crisis. (Oh wait, got my acronyms mixed up. This, but the opposite.)
    • @kellabyte: Your DB loses writes & pages every night. Your nerd pride is too tight to admit it just keeps compacting gigabytes. You just like to be a hero fixing things in the limelight. Your boss asks if everything is alright and you give them the green light and return to fortnite #opslife
    • James Hamilton: Faster than a Pi with more memory than a Raspberry Pi but, yes, it’ll [Graviton] run much of the same software.
    • @bcantril: Am waiting for the year that reInvent goes full Red Wedding, locking the doors and announcing that every attendee's product or service is now a forthcoming AWS offering. Or maybe that was this year?
    • @rakyll: Great that AWS went with @envoyproxy. Fragmentation in sidecars and load balancers is a huge issue and multiples work tremendously. Great that cloud providers and community seems to be agreeing on something, and we can just focus our energy on Envoy. To give an example, how many load balancers do we maintain to provide similar telemetry collection at load balancer level? Too many. The stack is highly fragmented, any simple feature requires months of planning and execution.
    • Mikhail Davidov: This sounds like a pretty big ‘if’ but there lies the crux of the T2: it does too much. The T2 is a fully featured and bulky platform. It shares many common components and drivers from the iOS and macOS platforms that frankly, just should not be there.
    • Chris Benner: The returns to capital are significantly outpacing the returns to labor.
    • @tjholowaychuk: $10/TB for queries, might as well use BigQuery at $5/TB unless it has a much faster lower-bound for latency, will have to see about that
    • @abooghost: Hard to believe a decade plus spent hiring psychologists to design monetary feedback loops in free to play games with the explicit purpose of getting children addicted could've gotten a bunch of children addicted
    • @drbarnard: “Apple can and does dramatically shape the App Store economy… I’d love to see Apple wield that power to shape the App Store in ways that will sustain and encourage meaningful development instead of continuing to allow the deck to be stacked against it.”
    • @ajaynairthinks: And love how casually Jeff threw in “Lambda processes trillions of executions for hundreds of thousands of active customers every month” - and growing :) Wanna hear more about the isolation, density and other cool things it enables #AWS Lambda to do? Checkout SRV409 !
    • @mckelveyf: Yeesh...airlines in UK used an algorithm to NOT assign families near each other when randomly picking seats, nudging families to pay for pre-assigned seats. Algorithms might hide the decision, but it's someone special to cook something like up
    • Karl Rupp: Clearly, transistor counts still follow the exponential growth line. AMD's Epyc processors with 19 billion transistors contribute the highest (publicly disclosed) transistor counts in processors to-date. For comparison: NVIDIA's GP100 Pascal GPU consists of 15 billion transistors, so these numbers are consistent. With the upcoming introduction of 10nm process nodes it is reasonable to assume that we will stay on the exponential growth curve for transistor counts for the next few years. These additional transistors will provide more cores, while the gains we see for SpecINT to measure single-threaded performance are primarily due to compilers employing auto-vectorization and auto-parallelization. Thus, if you want to benefit from future processors over what you have now, make sure to have parallel workloads. 🙂
    • David Rosenthal: Until May 27, 2020 there is a constant flow of 75BTC/hour coming in to the market. For the price to average $4K over the next 18 months, speculators have to inject on average $300K/hour in real money into the market. That is almost $3.9B. Right now its hard to see why they would do that. If they don't, the price crash will continue, making it even harder to see why they would pump money in. This is starting to look like a death spiral.@tanepiper: I maintain the most popular library for Bitly for node. I don't use it myself anymore and no one from Bitly has ever offered money to continue to maintain it, or offered to take it over. I fix bugs when I have time to or accept PRs. Beyond that I have more important things.
    • @rbranson: Did some basic benchmarking of a1.4xlarge vs c5.4xlarge using Phoronix C-Ray benchmark. Render took 108s on c5, 112s on a1. c5 is $0.68/hr, a1 is $0.408/hr. Looks like it really is quite the bargain. #reinvent
    • devonkim: I view the democratization of infrastructure similar to democracy- the best part of it is that anyone can do it, and the worst part of it is that anyone can do it. On the flipside of specialists getting involved, I also see an awful lot of bad / inappropriate networks and security layouts in cloud environments created by traditional infrastructure engineers because they carried too many principles from managing physical networks.
    • @Obdurodon: Good analogy 1/2 from HN today: You don't need an architect to design a garden shed, but you do for a house or an office building. SW devs who only build garden sheds should STFU about building houses.
    • @ajassy: New A1 instances are 1st to be powered by custom #AWS Graviton processors, based on @Arm architecture. Excited to pass cost savings back to customers, reducing them by up to 45% for scale-out workloads (like microservices & web servers): https://cnb.cx/2ShHEKx . #reInvent
    • TehnixI've seen a lot of people complain about [Amazon Timestream] pricing, so I thought I'd share a little why we are excited about this: We have approximately 280 devices out, monitoring production lines, sending aggregated data every 5 seconds, via MQTT to AWS IoT. The average messages published that we see is around ~2 million a day (equipment is often turned off, when not producing). The packet size is very small, and highly compressable, each below 1KB, but let's just make it 1KB. We then currently funnel this data into Lambda, which processes it, and puts it into DynamoDB and handles rollups. The costs of that whole thing is approximately $20 a day (IoT, DynamoDB, Lambda and X-Ray), with Lambda+DynamoDB making up $17 of that cost. Finally, our users look at this data, live, on dashboards, usually looking at the last 8 hours of data for a specific device. Let's throw around that there will be 10,000 queries each day, looking at the data of the day (2GB/day / 280devices = 0.007142857 GB/device/day)...From these (very) quick calculations, this means we could lower our cost from ~$20/day to ~$4.5/day. And that's not even taking into account that it removes our need to create/maintain our own custom solution.
    • @jessfraz: It seems like a lot of people are confused with the difference between container runtimes and firecracker. Firecracker is a virtual machine manager. Qemu is also a virtual machine manager. Katacontainers uses Qemu. I’d love to see them switch to Firecracker. :)
    • John McClean: The real tension isn’t between OO and FP, it’s between mutual imperative code that by passes where possible compiler checks to maximise runtime dynamism (as if all we needed was to migrate to Ruby) and constrained functional code that enables the compiler to find problems in our code before we do.
    • @HishamElfar: 2019 Prediction: Amazon will launch a nationwide grocery delivery service in April 2019 as they have a monopoly on food storage and are the only game left in town.

      2021: The UK formally changes its name to Amazon UK. All citizens are judged on their Prime Membership.

    • @aniccia: This study found ebikes expand bike ridership in age and gender (see Oslo graphs). Women were ~2x as likely as men to ebike v bike, and those 35+ age were ~50% more likely to ebike v bike as a 25-34 yr old. Ave speeds tail off >27 kph for both, perhaps due to safety & road net.
    • @xaprb: Something that isn’t a “Law,” but has held true through my entire career: when you instrument and measure something, you always learn surprising things. The code is doing WHAT? That query usually runs in microseconds but sometimes hours? We have a server nobody knew about? etc. This isn’t just database-monitoring-related. It’s true of any system, like… application code; off the shelf software; company finances; my personal health and vital signs—whatever.
    • AlexDeBrie: One issue with DynamoDB with serverless is that you had to determine your capacity ahead of time. No more. Like AWS Lambda, you can now pay per-request. This is great for coupling the cost to the value you're provided your users.
    • @bodil: (And if you don't want it bad enough to maintain it, maybe it's time to move on to something with a development model that suits you better. There are lots of them out there. Rust's model is particularly nice. Python too, if you still prefer a dictator in charge.)
    • @chamath: This is an amazing company we’ve backed since inception. DroneSeed has created a fleet of FAA approved aerial drones that conduct reforestation. The technology they’ve built to do this is amazing...and they’re replanting our forests at the same time.
    • @stu: The pace of innovation is getting faster and faster AWS is a nearly $27 billion revenue run rate business growing 46% y/y There is no compression algorithm for experience containers and serverless computing are completely rethinking the basic unit of compute @ajassy #reInvent
    • Andy Jassy: I would’ve made sure that we hired more salespeople and more professional services faster. We launched AWS in 2006 with two salespeople. At this point, we have a very large field [organization], with a lot of account managers and a lot of solutions architects and a lot of professional services folks, but we have just insatiable demand.
    • tnolet: Not dissing the general message. Startups can be brutal, but still. Dude is 22 and claims having seen more hardship than others. Try dealing with raising kids, health issues, sick family members, dying friends, mid life (all of which will probably happen as you near the age 40) and a startup will look like a pleasant distraction from the actual hard stuff in life. I'd say enjoy being 22, don’t take yourself to seriously and reflection will come with age.
    • Joe Emison: Here is a much better serverless architecture, which I will call a Serviceful Serverless application. In this architecture, all aspects of the application that do not need to be unique or differentiated from standard functionality (e.g., user management and authentication) are handled by a managed service (e.g., AWS Cognito, Auth0, Google Firebase Auth). One immediate reaction that some practitioners have to the above table is disbelief and skepticism--disbelief that any such reductions in code count are possible, and skepticism that, if it is possible, that the architecture is flexible and extensible enough for continued development as new feature requests come up. There is only so much that I can do in this article to convince you that these architectures do deliver extensible, extremely low-line-of-code-count applications; the best way for you to understand is to try them.
    • Ankur Sethi: Are Frameworks Evil? I’m not advocating that frameworks are evil, or that you should write all your applications in VanillaJS. That would waste far too much productive developer time, and the result will probably end up performing badly at runtime. Frameworks exist for a reason. But it’s important to consider your audience. If you’re building for resource constrained devices — which you certainly are if your product targets a country like India — you could consider using a lighter framework such as Riot or Preact. Your users will thank you.
    • njs: The popular concurrency primitives – go statements, thread spawning functions, callbacks, futures, promises, ... they're all variants on goto, in theory and in practice. And not even the modern domesticated goto, but the old-testament fire-and-brimstone goto, that could leap across function boundaries. These primitives are dangerous even if we don't use them directly, because they undermine our ability to reason about control flow and compose complex systems out of abstract modular parts, and they interfere with useful language features like automatic resource cleanup and error propagation. Therefore, like goto, they have no place in a modern high-level language. Nurseries provide a safe and convenient alternative that preserves the full power of your language, enables powerful new features (as demonstrated by Trio's cancellation scopes and control-C handling), and can produce dramatic improvements in readability, productivity, and correctness.
  • « Go back