Comments on: Ruby on Rails: Import CSV data into database http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#utm_source=feed&utm_medium=feed&utm_campaign=feed Just another WordPress site Fri, 18 Mar 2011 22:33:24 +0000 hourly 1 http://wordpress.org/?v= By: Ashish http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-93 Ashish Fri, 18 Mar 2011 22:33:24 +0000 http://goodbadtech.com/?p=43#comment-93 I hope this thread/message board is still active... I've implemented FasterCSV into my project for a long time now. Actually now I've moved to importing XLS files and parsing the data and importing into MySQL. Problem I'm running into now, after the database has become significantly large (>200,000 records) is that when parsing and importing data now...rails is painfully slow. A file of about a 1000 records which previously used to take minutes...now takes at least 30. I'm having major memory leaks and performance problems. Any possible explanations or remedies? I've gone through the code and noticed that there are several connection strings to the database which are constantly connecting and quitting..I know this is probably a big issue and I'm in the process of fixing it. However, any other pointers on where I can look to increase performance and decrease memory issues? I hope this thread/message board is still active…

I’ve implemented FasterCSV into my project for a long time now. Actually now I’ve moved to importing XLS files and parsing the data and importing into MySQL. Problem I’m running into now, after the database has become significantly large (>200,000 records) is that when parsing and importing data now…rails is painfully slow. A file of about a 1000 records which previously used to take minutes…now takes at least 30. I’m having major memory leaks and performance problems.
Any possible explanations or remedies? I’ve gone through the code and noticed that there are several connection strings to the database which are constantly connecting and quitting..I know this is probably a big issue and I’m in the process of fixing it. However, any other pointers on where I can look to increase performance and decrease memory issues?

]]> By: Darren http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-47 Darren Tue, 23 Mar 2010 23:41:17 +0000 http://goodbadtech.com/?p=43#comment-47 Forgot to mention in my last comment but if you push this to a background job you should do some (rescue FasterCSV::MalformedCSVError) action. Or even better do it if not tossing to background job so you don't get app errors if users upload a file that isn't parsed perfectly. Forgot to mention in my last comment but if you push this to a background job you should do some (rescue FasterCSV::MalformedCSVError) action. Or even better do it if not tossing to background job so you don’t get app errors if users upload a file that isn’t parsed perfectly.

]]> By: /good/bad/tech http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-46 /good/bad/tech Tue, 23 Mar 2010 21:42:04 +0000 http://goodbadtech.com/?p=43#comment-46 Thanks for the feedback. I'm glad it helped you out. Thanks for the feedback. I’m glad it helped you out.

]]> By: Greg http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-45 Greg Fri, 19 Mar 2010 12:50:49 +0000 http://goodbadtech.com/?p=43#comment-45 Just implemented for a project.  Works great.  Many thanks for the work.   Just implemented for a project.  Works great.  Many thanks for the work.
 

]]> By: Darren http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-44 Darren Wed, 24 Feb 2010 20:20:16 +0000 http://goodbadtech.com/?p=43#comment-44 Err I meant model Err I meant model

]]> By: Darren http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-43 Darren Wed, 24 Feb 2010 09:34:35 +0000 http://goodbadtech.com/?p=43#comment-43 In regards to moving the logic into the controller: http://gist.github.com/313281 This is just a quick example and I'm not sure it would work perfect. My setup is quite a lot different, but a good  starting point. Also error handling could be done a few ways. For me first I'm using delayed_job and after a csv file is uploaded I past off the load to delayed_job b/c we have imports of 100,000 + lines. As load_csv method is looping over rows we collect errors, if any, into an array and send out the results in an email. In regards to moving the logic into the controller: http://gist.github.com/313281
This is just a quick example and I’m not sure it would work perfect. My setup is quite a lot different, but a good  starting point.
Also error handling could be done a few ways. For me first I’m using delayed_job and after a csv file is uploaded I past off the load to delayed_job b/c we have imports of 100,000 + lines. As load_csv method is looping over rows we collect errors, if any, into an array and send out the results in an email.

]]> By: /good/bad/tech http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-42 /good/bad/tech Tue, 23 Feb 2010 19:31:59 +0000 http://goodbadtech.com/?p=43#comment-42 Hi John, I'm glad the post was helpful and that you already tracked down answer for the question you had. Regards, Spencer Hi John,

I’m glad the post was helpful and that you already tracked down answer for the question you had.

Regards,
Spencer

]]> By: John McLeod http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-41 John McLeod Wed, 17 Feb 2010 18:56:47 +0000 http://goodbadtech.com/?p=43#comment-41 Found it. My routes were messed up. map.import_proc '/import/proc/id', :controller => 'imports', :action => 'proc_csv' should be... map.import_proc '/import/proc/:id', :controller => 'imports', :action => 'proc_csv' John Found it.
My routes were messed up.
map.import_proc ‘/import/proc/id’, :controller => ‘imports’, :action => ‘proc_csv’
should be…
map.import_proc ‘/import/proc/:id’, :controller => ‘imports’, :action => ‘proc_csv’
John

]]> By: John McLeod http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-40 John McLeod Wed, 17 Feb 2010 18:41:56 +0000 http://goodbadtech.com/?p=43#comment-40 I can replace Line 11 with... <%= link_to "process", :controller => 'imports', :action => 'proc_csv', :id => @import.id %> and everything works fine. John I can replace Line 11 with…
<%= link_to “process”, :controller => ‘imports’, :action => ‘proc_csv’, :id => @import.id %>
and everything works fine.
John

]]> By: John McLeod http://goodbadtech.com/2009/05/13/ruby-on-rails-import-csv-data-into-database/#comment-39 John McLeod Wed, 17 Feb 2010 16:25:27 +0000 http://goodbadtech.com/?p=43#comment-39 Hello, Thanks for the tutorial, it helped extremely. I do have a little problem. The import is created but the proc_csv does not. I get an ... "<code>undefined method `to_sym' for nil:NilClass" error on the Imports/show view - Line 11.</code> <code>8: 9: <% else %> 10: 11: <%= link_to "process", import_proc_path(@import.id) %> 12: 13: <% end %> Thank you for any help with this. John </code> Hello,
Thanks for the tutorial, it helped extremely.
I do have a little problem.
The import is created but the proc_csv does not.
I get an …
“undefined method `to_sym' for nil:NilClass" error on the Imports/show view - Line 11.
8:
9: <% else %>
10:
11: <%= link_to "process", import_proc_path(@import.id) %>
12:
13: <% end %>

Thank you for any help with this.

John

]]>