How SerpApi sped up data extraction from HTML from 3s to 800ms (or How to profile and optimize Ruby code and C extension)

• flamescope

  3 2,989 0.0 Python

  FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.

  

I searched over the web how to profile C extensions for Ruby and C code in general, and found out Brendan Gregg’s tutorial on Linux perf. That was my first usage of Linux perf profiler. I’ve also tried gperftools and pprof, because seen its usage. And flamescope, because it was made by Brendan Gregg. There are many similar tools and it was hard to figure out what to use during two weeks or so.

• flamegraph

  47 4,307 7.4 Rust

  Easy flamegraphs for Rust projects and everything else, without Perl or pipes <3 (by flamegraph-rs)

  

flamescope shows the same as flamegraph. Both of these tools use the same tools to generate chart probably.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• perf_data_converter

  1 279 0.0 C++

  Tool to convert Linux perf files to the profile.proto format used by pprof

  

I’ve installed perf_data_converter to be able to use perf.data report with pprof.

• nokogiri-rust

  1 5 0.0 HTML

  Discontinued Ruby FFI wrapper around scraper crate to be used instead of Nokogiri. Status: proof of concept.

As of an experiment, I’ve made an FFI wrapper around the Rust scraper crate. at_css.text calls of proof of concept are 60 times faster than Nokogiri ones.

• bcc

  71 19,499 9.2 C

  BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

  

Haven’t tried to use bcc for tracing and profiling.

• lexbor

  10 894 8.3 C

  Lexbor is development of an open source HTML Renderer library. https://lexbor.com

  

I’m glad to have the opportunity to contribute to an open-source project that is used by thousands of people. Hopefully, we will speed up Nokogiri (or XML parser it uses) to match the performance of html5ever or lexbor at some point in the future. 800 ms to extract data from HTML is still too much.

• rbspy

  11 2,466 8.6 Rust

  Sampling CPU profiler for Ruby

  

c function is not very helpful to find the performance problem, so we dug deeper.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• Nokogiri

  20 6,109 9.4 C

  Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.

  

It worked because CFLAGS are passed here and there in ext/nokogiri/extconf.rb.

• linux

  983 170,949 10.0 C

  Linux kernel source tree

Haven’t read the entire documentation about perf.

• oga

  1 - -

Julien Khaleghy also tried Oga gem instead of Nokogiri. It was about six times faster than Nokogiri.

• ruby-ll

  1 - -

But some tests were failing with LL::ParserError from ruby-ll that is used in Oga.

Suggest a related project