We optimize the memory of the Rails service (real case)

For Ruby and Ruby on Rails Junior +, Middle Developers

We often write code without delving into how many resources it will take to execute it. And it could be ok. But, firstly, every normal developer is trying to get better and swinging his skills. Secondly, this can easily turn into a real bug with absolutely not obvious roots, and the question still needs to be addressed.

Mysterious bug

, CRM Kubernetes. Excel , . Sidekiq - , ( exception-, ). ‘in_progress’, Sidekiq , .

, , , , , . , CRM, , .

""

"" — . , - , . benchmark-memory, :

def benchmark_memory(&block)
  Benchmark.memory do |x|
    x.report('') do
      yield
      GC.start(full_mark: true, immediate_sweep: true) #   Garbage Collector',     
    end
  end
end

Garbage Collector . «» . GC Ruby , . , , . , , 100 GC:

 100    

50.966M memsize (     1.445M retained)
850.405k objects (    24.514k retained)
50.000  strings (    50.000  retained)

  100    

80.554M memsize (   920.363k retained)
 1.303M objects (    17.565k retained)
50.000  strings (    50.000  retained)

, Garbage Collector’, :

benchmark_memory { puts 1 }

Calculating -------------------------------------
1
                       464.000  memsize (     0.000  retained)
                         2.000  objects (     0.000  retained)
                         0.000  strings (     0.000  retained)

, 464 .

:

 100 :

1.482M memsize (     1.480M retained)
24.814k objects (    24.793k retained)
50.000  strings (    50.000  retained)

 1000 :

12.811M memsize (    12.809M retained)
226.587k objects (   226.556k retained)
50.000  strings (    50.000  retained)

100 , 1000 — . — , . , , , ( ). , , .

, , , . Garbage Collector’ 100 — 50 ( ), 500 1000. 38_000 — "" 18,5 ! (38 * 500 = 19_000 =~ 18,5 ).

?

1:

, — , xlsx csv. «», , . , «» , 40% :

 100   xlsx:

1.482M memsize (     1.480M retained)
24.814k objects (    24.793k retained)
50.000  strings (    50.000  retained)

 100   csv:

852.195k memsize (   849.968k retained)
12.034k objects (    12.007k retained)
50.000  strings (    50.000  retained)

2:

:

, , . ReportWorker’ . . , :

  def export_collection(objects_ids)
    Application.where(id: objects_ids.uniq).order(created_at: :desc)
  end

, , ? ActiveRecord find_each:

 def to_csv_export(objects_ids)
    CSV.open(file_path, 'w') do |csv|
      csv << export_headers #    

      export_collection(objects_ids).find_each do |application| #     ,     
        begin
          csv << export_row(application.decorate) #     
        rescue StandardError => e
          errors << "#{application.id}: #{e}"
        end
      end
    end
  end

, :

 100 :

166.688k memsize (   165.445k retained)
1.729k objects (     1.715k retained)
50.000  strings (    50.000  retained)

 1000 :

167.510k memsize (   166.227k retained)
1.734k objects (     1.719k retained)
50.000  strings (    50.000  retained)

, , , , .

:

, , — , 18,5 , 200 . , , , , . , , . . , , .

: 170 ,