My £4 a month server can handle 4.2M requests a day

• Protobuf

  171 63,657 10.0 C++

  Protocol Buffers - Google's data interchange format

  

There are many binary encoding protocols. A popular one is [protobufs](https://developers.google.com/protocol-buffers), which are used by gRPC.

• django-query-profiler

  3 129 0.0 Python

  Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

I agree that this stuff can definitely be handled better.

https://github.com/django-query-profiler/django-query-profil... has a neat option for detecting likely N+1 queries. I usually use the Django Debug Toolbar for this.

Django's ".only()" method lets you specify just the columns you want to retrieve - with the downside that any additional property access can trigger another SQL query. I thought I'd seen code somewhere that can turn those into errors but I'm failing to dig it up again now.

I've used the assertNumQueries() assertion in tests to guard against future changes that accidentally increase the number of queries being made without me intending that.

The points you raise are valid, but there are various levels of mitigations for them. Always room for improvement though!

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• django-query-profil

  2 - -

I agree that this stuff can definitely be handled better.

https://github.com/django-query-profiler/django-query-profil... has a neat option for detecting likely N+1 queries. I usually use the Django Debug Toolbar for this.

Django's ".only()" method lets you specify just the columns you want to retrieve - with the downside that any additional property access can trigger another SQL query. I thought I'd seen code somewhere that can turn those into errors but I'm failing to dig it up again now.

I've used the assertNumQueries() assertion in tests to guard against future changes that accidentally increase the number of queries being made without me intending that.

The points you raise are valid, but there are various levels of mitigations for them. Always room for improvement though!

• azure-docs

  89 9,947 10.0 Markdown

  Open source documentation of Microsoft Azure

  

U-SQL on Azure Data Lake Analytics[0] and ECL from HPCC Systems[1] are two I have fairly extensive experience with. There may be other such platforms, but those and Ballista are the three on my radar.

[0] https://github.com/MicrosoftDocs/azure-docs/blob/master/arti...

[1] https://wiki.hpccsystems.com/pages/viewpage.action?pageId=28...

• FrameworkBenchmarks

  366 7,384 9.8 Java

  Source for the TechEmpower Framework Benchmarks project

  

The best/fastest web servers today manage >600k rps on a single server [0].

django is on that list, near the bottom, at 15k. Clearly his request response is more complicated than a simple fortune.

There are a few factors I've experience here. Interpreted languages have encouraged development patterns that are slow. Ease of allocating memory has tended to promote its overuse. Coding emphasis has been heavily weighted on developer productivity and correctness of code over lean and fast.

I find that poor web server configurations are pretty common. Smaller shops tend to use off the shelf frameworks rather than roll their own systems. Various framework "production" setups often don't include any caching at all. Static files are compressed and then sent on every request instead of preserving a pool of pre-compressed common pages/assets/responses. It's like the framework creators just assume there's going to be a CDN in front of the system, and so they don't even try to make the system fast.

The latest crop of web devs have very little experience setting up production systems correctly. Companies seem more interested in AWS skills than profiling. Then you have architectural pits like microservices. Since there is so little emphasis on individual system performance it seems that it has become or is becoming a lost skill.

Then there is so much money being thrown at successful SaaS that it just doesn't matter that their infrastructure costs are potentially 50x what they actually need. It seems that the only people squeezing performance out of software are the poor blokes who are scrimping by on shoestrings with no VC money in sight.

[0] https://www.techempower.com/benchmarks/

• cryptogalaxy

  22 111 0.0 Go

  Discontinued Get any cryptocurrencies ticker and trade data in real time from multiple exchanges and then save it in multiple storage systems.

Suggest a related project