![[image]](http://mowser.com/img?url=http%3A%2F%2Fwww.rubyonrails.org%2Fimages%2Frails.png)
What's new in Edge Rails: EXPLAIN
There are some new features related to EXPLAIN in the forthcoming Ruby on Rails 3.2 we'd like to share:
As of this writing they are available for the adapters sqlite3, mysql2, and postgresql.
Running EXPLAIN Manually
You can now run EXPLAIN on the SQL generated by a relation this way:
User.where(:id => 1).joins(:posts).explain
The result of that method call is a string that carefully imitates the output of database shells. For example, under MySQL you get something similar to
EXPLAIN for: SELECT `users`.* FROM `users` INNER JOIN `posts` ON `posts`.`user_id` = `users`.`id` WHERE `users`.`id` = 1
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------------+
| 1 | SIMPLE | users | const | PRIMARY | PRIMARY | 4 | const | 1 | |
| 1 | SIMPLE | posts | ALL | NULL | NULL | NULL | NULL | 1 | Using where |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------------+
2 rows in set (0.00 sec)
and under PostgreSQL the same call yields something like
EXPLAIN for: SELECT "users".* FROM "users" INNER JOIN "posts" ON "posts"."user_id" = "users"."id" WHERE "users"."id" = 1
QUERY PLAN
------------------------------------------------------------------------------
Nested Loop Left Join (cost=0.00..37.24 rows=8 width=0)
Join Filter: (posts.user_id = users.id)
-> Index Scan using users_pkey on users (cost=0.00..8.27 rows=1 width=4)
Index Cond: (id = 1)
-> Seq Scan on posts (cost=0.00..28.88 rows=8 width=4)
Filter: (posts.user_id = 1)
(6 rows)
Please note that explain runs the query or queries and asks the database for their respective query plan afterwards. This is because due to eager loading a relation may trigger several queries to fetch the records and their associations, and in such cases some queries need the result of the previous ones.
If the relation triggers several queries the method still returns a single string with all the query plans. This is an output meant for human consumption so we preferred to present everything as a string in a format which is familiar right away rather than a structure.
Automatic EXPLAIN For Slow Queries
Rails 3.2 has the ability to help in detecting slow queries.
New applications get
config.active_record.auto_explain_threshold_in_seconds = 0.5
in config/environments/development.rb. Active Record monitors queries and if they take more than that threshold their query plan will be logged using warn.
That works for anything running find_by_sql (which is almost everything, since most of Active Record ends up calling that method). In the particular case of relations, the threshold is compared against the total time needed to fetch the records, not against the time taken by each individual query. Because conceptually we are concerned with the cost of the call
User.where(:id => 1).joins(:posts).explain
rather than the cost of the different queries that call may trigger due to the implementation.
By default the threshold is nil in the test and production environments, which means the feature is disabled.
The value of that parameter is nil also if the threshold is not set, so existing applications will need to add it by hand if they migrate to 3.2 to be able to enable automatic EXPLAIN.
Silencing Automatic EXPLAIN
With automatic EXPLAIN enabled, it could still be the case that some queries are just slow and you know they have to be. For example, a heavyweight report in the backoffice.
The macro silence_auto_explain allows you to avoid having EXPLAIN run repeatedly in those areas of code:
ActiveRecord::Base.silence_auto_explain do
# no automatic EXPLAIN here
end
Interpreting Query Plans
The interpretation of the query plans is another topic, these are some pointers:
Happy debugging!
jQuery: New Default
In Rails 3.1 jQuery is going to be the default JavaScript library. Also, RJS has been extracted out. This post explains what that means for new applications, and what to look for while upgrading existing applications.
New Applications
Starting with Rails 3.1
rails new my_app
generates an application with jQuery.
The -j option of the generator lets you choose Prototype and Scriptaculous instead:
rails new my_app -j prototype
Such an application does not have RJS yet available though. From now on prototype-rails is needed for RJS, see below.
Upgrading Applications Using No RJS
Existing applications using no RJS should remove any references to ActionView::Base.debug_rjs in the project. Typically that means deleting
config.action_view.debug_rjs = true
from config/environments/development.rb. Other than that, upgrading should work out of the box as far as these changes is concerned.
Upgrading Applications Using RJS
Existing application using RJS should work out of the box with prototype-rails.
prototype-rails
RJS has been extracted to prototype-rails.
Applications using RJS have to add this line to their Gemfile while working against Rails master before the 3.1 release:
gem 'prototype-rails', :git => 'git://github.com/rails/prototype-rails.git'
prototype-rails is gonna be a gem when Rails 3.1 is out.
prototype-rails is the one who provides now the RJS template handler; the configuration flag ActionView::Base.debug_rjs; the ability to pass a block to link_to_function and button_to_function; the :update option of render, both in controllers and views; and the modules ActionView::Helpers::PrototypeHelper and ActionView::Helpers::ScriptaculousHelper.
Extraction has taken backwards compatibility into account as much as possible. With that goal in mind, everything is put back where it was before, either by reopening classes or modules, or via alias_method_chain when the refactor involved cutting some part of a method out, as happened with render :update.
There's an exception: ActionView::Helpers::PrototypeHelper and ActionView::Helpers::ScriptaculousHelper are no longer ancestors of ActionView::Helpers. They are now injected into ActionView::Base and ActionView::TestCase directly.
When you include a module M into a class C, the class stores a proxy to M in its ancestor chain. After inclusion, if you add methods to M instances of C respond to them. But if you include another module N into M, the ancestor chain of C is not updated. Instances of C won't respond to N's methods. Although a complete dynamic method dispatch would treat both ways of augmenting M equal, that's the way it works as of today.
By the time prototype-rails is loaded ActionView::Helpers has already been included into ActionView::Base and ActionView::TestCase, so we need to inject them directly where they are needed. Standard usage just works, but please take into account that change in case you reopened or used ActionView::Helpers assuming those ancestors.
Before release, prototype-rails is going to provide also a way to bring Prototype and Scriptaculous to any application.
One Last Detail: The xhr Test Helper
The xml_http_request/xhr test helper is a simple convenience method that sets the X_REQUESTED_WITH header to "XMLHttpRequest". If the test request has no Accept header, a fixed value of
[Mime::JS, Mime::HTML, Mime::XML, 'text/xml', Mime::ALL].join(', ')
serves as default.
The Accept header in Prototype calls corresponds to that value, but in jQuery it depends on the dataType attribute. Also, in other JavaScript libraries the Accept header may behave differently. What to do with the default?
This method has been left as is. In particular, if your application uses jQuery the default Accept header in the test will be different from the Accept header sent by jQuery. That's some vinegar: Rails encourages you to base interfaces on explicit format parameters. So, for example, best practice is to provide a /users.json endpoint, rather than a /users endpoint with a JSON representation chosen depending on the Accept header.
If you desperately need a matching Accept header in tests you are still able to pass it. And you can also write your own convenience test helpers easily, since a XHR request is just one whose X_REQUESTED_WITH header matches /XMLHttpRequest/i.
Try It Out!
If your application uses RJS it would be really helpful that you test it against master with prototype-rails and report any issues you may find. Thanks!
Why HTTP Streaming?
Rails 3.1 is going to support HTTP streaming, aka chunked responses, this post explains what's all about.
What Is HTTP Streaming?
Ordinary dynamic HTTP responses need a Content-Length header. Their timeline look like this:
HTTP request -> dynamic content generation -> HTTP response
Those are three serial steps because normally you need to generate the content in order to be able to know its size, and thus fill the Content-Length header of the response.
HTTP provides an alternative to this schema to be able to flush data as it is produced, known as chunked transfer encoding. That's what we are referring to as streaming in recent commits.
Streamed responses have no Content-Length header. Rather, they have a Transfer-Encoding header with a value of "chunked", and a body consisting of a series of chunks you write to the socket preceded by their individual sizes. Modulus details.
This is an example taken from Wikipedia:
HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked
25
This is the data in the first chunk
1C
and this is the second one
3
con
8
sequence
0
Point is, you are able to flush chunks to the socket as soon as you have them, no need to wait for the whole thing to be generated.
When Do Web Browsers Fetch Assets?
Web broswers parse documents as their content is received. When they find an asset referenced, think an image, stylesheet, or script, a request to fetch them is fired. That happens in parallel while the document is being received and processed, no matter whether the content comes chunked or not.
Browsers have limits on the number of concurrent requests they are allowed to do, a global one (typically +30), and another per domain (nowadays typically 4 or 6), but within those limits, requests for getting assets happen as the content is parsed.
Modern clients do not even block on JavaScript files as old ones did, they implement scanners that look ahead for asset nodes and request them. For example, this is the preload scanner of WebKit.
Trivia: While investigating this I discovered by accident that if the MIME type is unclear, for example "text/html" without an explicit charset, then web browsers buffer 1 KB of data firing no asset requests to be able to peek at the content and do an educated guess.
So What's The Benefit Of Streaming?
Streaming doesn't cut latency, neither it cuts the time a dynamic response needs to be generated. But since the application sends content right away instead of waiting for the whole response to be rendered, the client is able to request assets sooner. In particular, if you flush the head of an HTML document CSS and JavaScript files are going to be fetched in parallel, while the server works on generating content. The consequence is that pages load faster.
Followup
Streaming is still being polished for Rails 3.1, expect another post in the future covering its practical aspects in Ruby on Rails applications.
Thanks
Tony Gentilcore provided his insider's guidance into this, thank you very much Tony! Also, thanks a lot to the Browserscope project for their really useful tables.
What's New in Edge Rails
So, Edge Rails is still chugging right along. There are new and interesting fixes, changes, and refactors going on all of the time. So, lets take a look at just a few that've gone in since the last post (it's been a while, I know, I'm sorry!).
ActionView and Helpers
XSS escaping is now enabled by default. This means that if you want to explicitly output HTML to your views, you'll probably have to mark it as html_safe! before sending it through.
<%= 'my <a href="http://www.rubyonrails.org">safe</a> string'.html_safe! %>
Many of the built-in helpers have been updated for this change and if you see an issues with the Rails helpers being incorrectly sanitized, you should create a new ticket.
distance_of_time_in_words has gained 'over', 'about', and 'almost' keywords, thanks to Jay Pignata and John Trupiano. This provides you with an improved level of granularity when approximating the amount time passed. So, instead of just "2 years ago", it can now also report "almost 2 years ago," "about 2 years ago," and "over 2 years ago," depending on the proximity to being exactly 2 years old.
assert_equal "almost 2 years", distance_of_time_in_words(from, to + 2.years - 3.months + 1.day)
assert_equal "about 2 years", distance_of_time_in_words(from, to + 2.years + 3.months - 1.day)
assert_equal "over 2 years", distance_of_time_in_words(from, to + 2.years + 3.months + 1.day)
assert_equal "over 2 years", distance_of_time_in_words(from, to + 2.years + 9.months - 1.day)
assert_equal "almost 3 years", distance_of_time_in_words(from, to + 2.years + 9.months + 1.day)
The HTML form helper, fields_for - generally used for nesting additional model forms - now allows for explicit collections to be used, thanks to Andrew France. So, instead of just including all of your blog.posts, you should have it only display your published blog.posts, for example. Or:
<% form_for @person, :url => { :action => "update" } do |person_form| %>
...
<% person_form.fields_for :projects, @active_projects do |project_fields| %>
Name: <%= project_fields.text_field :name %>
<% end %>
<% end %>
API Change for content_tag_for: The third argument - being the optional CSS prefix - will now also affect the generated CSS class. This prefix will now be appended to the generated element's CLASS attribute.
<%= content_tag_for(:li, @post, :published) %>
# => <li id="published_post_123" class="published_post">...</li>
ActiveResource and ActiveRecord
Taryn East has added update_attribute(s) methods to ActiveResource. These methods act very similarly to the ActiveRecord methods we already know and love.
Building or creating an object through a has_one association that contains conditionals will now automatically append those conditions to the newly created object, thanks to Luciano Panaro.
class Blog
has_author :commit_author, :class_name => 'Author', :conditions => {:name => "Luciano Panaro"}
end
@blog.build_commit_author
# => #<Author name: "Luciano Panaro" ... >
Pratik Naik added a new option to ActiveRecord's accepts_nested_attributes_for to :limit the number of records that are allowed to be processed. Also, while we're covering accepts_nested_attributes_for, José Valim as renamed the _delete option to _destroy to better follow what is actually occurring. A deprecation warning has been added to _delete, for the time being.
Jacob Burkhart updated the new autosave option in Rails 2.3 to allow for an :autosave => false, which will disallow saving of associated objects, even when they are new_record?s.
Some Internals
Previously, CDATA elements could be ignored when converting from XML to a Hash, so now, thanks to John Pignata, Hash#from_xml will now properly parse and include CDATA elements values.
Josh Peek has relocated global exception handling into ActionDispatch::Rescue. So, this is now being handled at the Rack middleware level.
And finally, Yehuda Katz and Carl Lerche began work on a Rails::Application object to better encapsulate some of the application start up and configuration details. Also, a good bit of initialization has now gone on to move into this new object.
Remember, if you prefer to have a shorter audio summary of some of this content and more, you should check out the Ruby5 podcast over at Envy Labs; it's released every Tuesday and Friday with the latest news in the Ruby and Rails community.
Photo: Clock Tower by Brian Taylor
What's New in Edge Rails: The Security Edition
It's been a bit over two weeks since the last WNiER ("winner"?) post and in the time since our last visit, Ruby on Rails 2.3.4 was released to fix some reported security issues. It is important that you try to upgrade your applications as soon as possible, or even just apply the provided patches if a full upgrade isn't easily accomplished in your situation.
Along with this release, you're also going to see several bug fixes and enhancements to the Rails framework, coming from many contributors, that have been discussed here over the previous weeks and even a few that are mentioned just below.
Security updates
Michael Koziarski posted fixes (here and here) for cleaning and verifying multibyte (unicode) strings. The problem was reported by Brian Mastenbrook and Manfred Stienstra provided input for the fix. These changes should disallow malformed unicode strings from getting past the HTML escaping logic provided by the form helpers.
Coda Hale reported and also added a patch to Rails, fixing a timing attack vulnerability in ActiveSupport::MessageVerifier. Although not likely to be exploited in the wild, the vulnerability may allow an attacker to forge the signatures which encode your application's cookie store. If successfully broken, an attacker could modify their session objects without altering your application to the change.
There have been some issues reported around the Rails 2.3.4 release, specifically with regard to Ruby 1.9 support. While they have not all yet been fully substantiated, this certainly underscores the importance of having proper test coverage and both a staging and production environment for your applications.
Down to the metal
Yehuda Katz and Carl Lerche put in quite a bit of work around ActionController::Metal and Rack's Middleware, recently. ActionController::Metal now acts as a Rack middleware and at the same time, there is a new ActionController::Middleware class that operates as normal Rack middleware.
And, if that wasn't enough, Yehuda went on to add ActiveModel::Lint. ActiveModel::Lint allows you to determine whether or not an object is compliant with the ActiveModel API, via:
ActiveModel::Compliance.test(object)
The output is similar to a Test::Unit output and will indicate with which portions of the ActiveModel API the given object is - or more importantly is not - compliant.
If Metal is your thing, you may want to take a look at Yehuda Katz's recent blog post, How to Build Sinatra on Rails 3.
Pour some sugar on me
Quite a few changes, small and large, occurred around ActiveRecord and friends. Most of these cleaned up some existing functionality, either making it easier to use, perform more closely to what would be expected, or even adding some new features that will soon feel like old friends.
Taryn East added a little ActiveRecord-like love to ActiveResource. In this patch, ActiveResource received the familiar first, last, and all shortcut methods for wrapping the basic find method.
Proc and symbol support was added to the validates_numericality_of ActiveRecord validation, by Kane.
For those of you who use the :anchor option when generating URLs, you may notice that after this patch by Jeffrey Hardy, Rails will now execute the to_param method on the object provided as an :anchor.
@post = Post.first
@comment = Comment.first
post_url(@post, :anchor => @comment) # => http://www.example.com/posts/1#comment-1
Well, something similar to that, anyway. :) This updates the :anchor options to follow a similar functionality as the other options provided when generating URLs.
José Valim cleaned up some bits in the Rails scaffold. The generated new and edit views will now reference a new _form partial. This is a much DRYer way to go about it, and more closely follows what would likely happen if you were to code it yourself. Also, while he was there, he removed a bit of inline CSS (specifically, a green flash message), in favor of a CSS class and updating the default scaffold stylesheet.
And, probably the most interesting change in this group is the addition of a new ActivRecord#previous_changes method, by Scott Barr. previous_changes allows you to see what changed before the last save in your local ActiveRecord object instance. This is particularly useful when calling after_save methods which might need to know what exactly had changed. I'll let him give you a code sample:
person = Person.find_by_name('bob')
person.name = 'robert'
person.changes # => {'name' => ['bob, 'robert']}
person.save
person.changes # => {}
person.previous_changes # => {'name' => ['bob, 'robert']}
person.reload
person.previous_changes # => {}
Okay, let's do it your way
While a lot of us prefer US English, we (begrudgingly) recognize that we aren't always the center of the universe. As such, there are some more localization updates to report in Edge Rails:
Sven Fuchs added localization support to the ActiveRecord::RecordInvalid exception's error message. Then, Akira Matsuda followed Sven with support for localizing the SELECT tag helper's prompt text (the default being, "Please select").
Finally, this is certainly a welcome addition and potentially a major player in localization support within Rails: Antonio Tapiador del Dujo added a patch which allows Rails plugins to define and maintain their own locale files. All that is necessary for the plugin developer to do is to provide a config/locales/ directory within their plugin and then create their own .rb or .yml files (i.e. en.yml). That means that plugins can now be much more responsible for their own localization support and do not have to modify the application's locale files after installation.
Food for thought
Finally, just a small note that the default, preferred table collation for MySQL has been changed. Previously, Rails defaulted to utf8_general_ci when either the database or the table creation script did not dictate otherwise. Now, that has been changed to utf8_unicode_ci. Certainly worth a note with so many Rails applications using MySQL in their back-end.
Update: Set the attribution of previous_changes to Scott Barr. Sorry, Scott!
Photo: Security at the Hoover Dam by Alex E. Proimos
