How SerpApi sped up data extraction from HTML from 3s to 800ms (or How to profile and optimize Ruby code and C extension)

• flamescope

  0 2,938 0.0 Python

  FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.

  

  I searched over the web how to profile C extensions for Ruby and C code in general, and found out Brendan Gregg’s tutorial on Linux perf. That was my first usage of Linux perf profiler. I’ve also tried gperftools and pprof, because seen its usage. And flamescope, because it was made by Brendan Gregg. There are many similar tools and it was hard to figure out what to use during two weeks or so.

• flamegraph

  11 3,906 6.2 Rust

  Easy flamegraphs for Rust projects and everything else, without Perl or pipes <3 (by flamegraph-rs)

  

  flamescope shows the same as flamegraph. Both of these tools use the same tools to generate chart probably.

• InfluxDB

  www.influxdata.com

  sponsored

  Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

  

• perf_data_converter

  0 264 0.0 C++

  Tool to convert Linux perf files to the profile.proto format used by pprof

  

  I’ve installed perf_data_converter to be able to use perf.data report with pprof.

• nokogiri-rust

  0 5 0.0 HTML

  Ruby FFI wrapper around scraper crate to be used instead of Nokogiri. Status: proof of concept.

  As of an experiment, I’ve made an FFI wrapper around the Rust scraper crate. at_css.text calls of proof of concept are 60 times faster than Nokogiri ones.

• bcc

  19 18,422 7.5 C

  BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

  

  Haven’t tried to use bcc for tracing and profiling.

• lexbor

  1 822 0.0 C

  Lexbor is development of an open source HTML Renderer library. https://lexbor.com

  

  I’m glad to have the opportunity to contribute to an open-source project that is used by thousands of people. Hopefully, we will speed up Nokogiri (or XML parser it uses) to match the performance of html5ever or lexbor at some point in the future. 800 ms to extract data from HTML is still too much.

• rbspy

  4 2,425 0.0 Rust

  Sampling CPU profiler for Ruby

  

  c function is not very helpful to find the performance problem, so we dug deeper.

• Onboard AI

  getonboard.dev

  sponsored

  Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.

  

• Nokogiri

  5 6,080 9.2 C

  Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.

  

  It worked because CFLAGS are passed here and there in ext/nokogiri/extconf.rb.

• linux

  203 161,582 10.0 C

  Linux kernel source tree

  Haven’t read the entire documentation about perf.

• oga

  0 - -

  Julien Khaleghy also tried Oga gem instead of Nokogiri. It was about six times faster than Nokogiri.

• ruby-ll

  0 - -

  But some tests were failing with LL::ParserError from ruby-ll that is used in Oga.

Suggest a related project