No connection to the Akiflow team. I found the tool on my own and really like it. It’s snappy and does what it claims to do. I don’t really need their Calendly-lite implementation but everything else is very well put together.
Astounding to see racism and ignorance represented as virtue signaling so blatantly!
Are you aware that the majority of STEM seats in Indian colleges are set aside for either affirmative action for ‘lower’ castes (itself an arbitrary criteria) and women?
The average man in India is competing for approx 10% of college seats!
And saying an Indian STEM degree is worth millions in lifetime earnings reveals even more ignorance - outside of a handful of colleges, the India higher education system is considered woefully inadequate, underfunded, pumping out graduates who only succeed out of personal initiative.
Every major corporation in India essentially has a retraining program to actually teach what’s necessary for work. The worker training issue is one of the top challenges faced by businesses there.
Perhaps it’s time you described your own ethnicity and background etc. Let’s learn a little more about where your biases flow from.
If you actually grew up in India in the time period of the OP, or knew anything about it other than nonsense in woke
publications, you would feel ashamed for your tone and envy and render a sincere apology to the OP.
> Are you aware that the majority of STEM seats in Indian colleges are set aside for either affirmative action for ‘lower’ castes (itself an arbitrary criteria) and women?
And these seats are never filled and go to upper castes anyway?
My class was supposed to be 50% low caste with 25% Dalits in a class of 60. There were literally 2 Dalits in my class. And of late, the OBC categories are being crowded in by upper castes through political action.
A whole 50% of my class was actually "payment seats" where they paid double the fees to get seats with low entrance scores. Basically, a 50% quota for the privileged.
Curious non-Indian here, how did you know that there were exactly two Dalits in your class? Is this a thing people talk about? Were they open about it? Or do you just recognize them on sight?
I am not a Hindu, so my caste radar is fairly weak. I cant tell caste by last name or by religious practices.
Having said that, People ask folks how much they scored in their entrance exam. Then it becomes obvious who were Dalits are - they had the lowest scores. The 2 of them were also fairly underprivileged, so you could guess from their less trendy and not as new clothes too.
50% were payment seats with about the same entrance rank as OBC students. So it was hard to tell the OBCs apart from the payment seat students. India actually has "affirmative action" for the rich in all private universities, which most of Indian H1Bs have graduated from. My parents couldn't afford these seats. The H1B upper castes wont tell you that, and will pretend that rich Dalits are taking over all the seats. No such thing is happening. I have never met a rich Dalit in my life. I am sure they exist, but I have never met them.
Their academic performance improves with time, and some of them are engineering managers at Microsoft now.
> how do you know that some are engineering managers at Microsoft, now?
I am connected to those folks on LinkedIn.
> Also, is "caste radar" a word you made up, or is that a generally-used concept?
I made it up. But I have read content online, where Dalits feel a huge pressure when they are asked which temple they attend, or how they celebrate certain festivals. I don't know what these ritual differences are. Many folks pick up caste from last names, I can only pick up a small subset of them. What I meant to say was that - I can't tell the caste of a person as easily as Hindus. The Dalits I know, I surmised their caste from their entrance rank and many of them didn't hide it.
I merely quoted some stats, stats are not racist. I read through your rant but didn't see an explanation of why only upper caste Indians are emigrating to usa. whats that about? How do only 30% of Indians make up 90+% indians in USA.
Surely a stem degree and an opportunity to get a post graduate stem degree in a blazing hot tech market is worth more than $30.
Do you really think GP and a venezuelan asylee have same amount of privilege because they both came over with $30 in their pockets. Yet that was the logic GP used to justify his lack of privilege.
Read it over again Magoo: the subject is a specific kind of degree and how it opens doors for a specific industry in the Bay Area. A specific industry known for being insanely competitive and requiring almost comical qualifications to get in. Put the pieces together.
People with degrees, especially in stem are small (in compare to all ppl) group of people who are willing to put multi year, intense effort studying non trivial things
So yea, just those traits make them more likely to earn more
Go ahead and give degree to somebody without skills in good paying jobs and see whether that person will be milions ahead too
A degree gets you in the door at all and starts you off with a higher salary (I would say significantly so) than someone with no comparable education. Those two things are a huge advantage, both in terms of money and in terms of the odds that you will be able to start making it at all.
@AtlasLion you are right real world performance matters. We test extensively with actual workloads, and the speed up holds there too. For example: lots of real world BI queries are repeated over smallish data sets of 10 to 50 GB. We test that size factor and pattern all the time.
Databricks broke the record by 2x) and is 10x more cost effective, in an audited benchmark. Snowflake should participate in the official, audited benchmark. Customers win when businesses are open and transparent…
Databricks and snowflake should pay an independent third party to re-run these. In-house benchmarks by either company don't count with results this different.
Databricks didn't run the Snowflake comparison in-house. From their article it says: "These results were corroborated by research from Barcelona Supercomputing Center, which frequently runs TPC-DS on popular data warehouses. Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance."
There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did.
The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete data), sequential query test, and concurrent query test.
Consider the following analogy: Professional athletes compete in the Olympics, and there are official judges and a lot of stringent rules and checks to ensure fairness. That's the real arena. That's what we (Databricks) have done with the official TPC-DS world record. For example, in data warehouse systems, data loading, ordering and updates can affect performance substantially, so it’s most useful to compare both systems on the official benchmark.
But what’s really interesting to me is that even the Snowflake self-reported numbers ($267) are still more expensive than the Databricks’ numbers ($143 on spot, and $242 on demand). This is despite Databricks cost being calculated on our enterprise tier, while Snowflake used their cheapest tier without any enterprise features (e.g. disaster recovery).
Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me.
Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size.
Snowflake is the opposite: you can't even specify partitions! There is only clustering.
For a business, running snowflake is easy because engineers don't have to babysit it, and we like it because now we're free to work on more interesting problems. Everybody wins.
Unless those problems are DB optimization. Then snowflake can actually get in your way.
Totally. Simplicity is critical. That’s why we built Databricks SQL not based on Spark.
As a matter of fact, we took the extreme approach of not allowing customers (or ourselves) to set any of the known knobs. We want to force ourselves to build the best the system to run well out of the box and yet still beats data warehouses in price perf. The official result involved no tuning. It was partitioned by date, loaded data in, provisioned a Databricks SQL endpoint and that’s it. No additional knobs or settings. (As a matter of fact, Snowflakes own sample TPC-DS dataset has more tuning than the ones we did. They clustered by multiple columns specifically to optimize for the exact set of queries.)
>That’s why we built Databricks SQL not based on Spark.
Wait... really? The sales folks I've been talking to didn't mention this. I assumed that when I ran SQL inside my Python, it was decomposed into Spark SQL with weird join problems (and other nuances I'm not fully familiar with).
Not that THAT would have changed my mind. But it would have changed the calculus of "who uses this tool at my company" and "who do I get on board with this thing"
Edit:
To add, I've been a customer of Snowflake for years. I've been evaluating Databricks for 2 months, and put the POC on hold.
Credit to you for these amazing benchmark scores via an official process. You've certainly proved to naysayers such as Stonebreaker that lakes and warehouses can be combined in a performant manner!
Shame on your for quoting a fake non-official score for Snowflake in your blog post with crude suggestions to make it seem you're showing an apples-to-apples comparison.
I run a BI org in an F500 company that uses both Databricks & Snowflake on AWS. I can tell you that such dishonest shenanigans take away much from your truly noteworthy technical achievements and make me not want to buy your stuff for lack of integrity. Not very long ago, Azure+GigaOM did a similar blog post with fake numbers on AWS Redshift and it resulted in my department and a bunch of large F500 enterprises that I know moving away from Synapse for lack of integrity.
On many occasions, I've felt that Databricks product management and sales teams lack integrity (especially the folks from Uber & VMW) and such moves only amplify this impression. Your sales guys use arm-twisting tactics to meet quotas and your PM execs. are clueless about your technology and industry. My suggestion is to overhaul some of these teams and cull the rot - it is taking away from the great work your engineers and Berkley research teams are doing.
Snowflake claims the snowflake result from Databricks was not audited. It’s not that Databricks numbers were artificially good but rather Snowflake’s number was unreasonably bad.
Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.
It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw.
TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right.
Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.
The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you.
But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
This is my point also, and I'm being downvoted for it.
If two people are in disagreement about the same facts, then one of them is either misinformed or lying. It's that simple.
If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process.
I'm personally unimpressed with the TPC process in general. I remember one "benchmark" that showed the performance of a 2RU server breaking some record, and it was a minor footnote that it was using a disk array with 7,500 drives in it -- dedicated to that one server for the duration of the test. That's an absurd setup that will never exist at any customer, ever.
I ran that same software myself on literally the exact same server, and it couldn't even begin to approach the posted TPC numbers on typical storage. It was at least two orders of magnitude slower.
The rub was that its inefficient usage of storage was the main problem, and the vendor was pulling a smoke & mirrors trick to hide this deficiency of their product. The TPC numbers were an outright fraud in this case, at least in my mind.
So to me, TPC looks like a staged show where the auditors are more like the referees in a WWE wrestling competition.
That doesn't allow end users any configuration, but this doesn't apply to the company itself which can apply settings from the background on behalf of end users.
Hey Will I’m sorry it didn’t work out. We’ve actually adjusted really well to COVID (in no small part thanks to our amazing employees).
We care a lot about our hiring experience. I’d be happy to chat about this over email. Feel free to drop me a message at Bilal dot Aslam at databricks dot com.
