Hello everyone. In this presentation, I will discuss the history of keyword arguments, their current implementation, how keyword argument separation was handled, and future improvements to keyword arguments.

My name is Jeremy Evans. One of my main focuses since becoming a Ruby committer last year has been implementing keyword argument separation.

Back when I started Ruby, around the time of Ruby 1.8, Ruby didn’t have keyword arguments.

The book I used to learn Ruby programming was the first version of Programming Ruby. This book was released in December 2000 and covers Ruby 1.6. The book specifically mentions keyword arguments.

It states that Ruby 1.6 does not have keyword arguments (although they are scheduled to be implemented in Ruby 1.8). Looks like the schedule slipped by about 10 years, since Ruby 1.8 was released in 2003, and Ruby did not support keyword arguments until 2013, when Ruby 2.0 was released.

Shortly after that quote, Programming Ruby talks about how you can use hashes to achieve the same effect as keyword arguments. If you have a method like this, and you want to add keyword arguments to it,

You can pass a hash of values. It does mention this is slightly clunky.

Mostly because these braces could be mistaken for a block. However, it also mentions that the braces are optional if the hash is the last argument.

Resulting in this code. So while older versions of Ruby didn’t support keyword arguments, they provided something that handled pretty much the same need. This same syntax on the caller side can work when using keyword arguments in Ruby 2.7 and Ruby 3.

Before keyword arguments were supported, and hashes are used instead, you would list the hash as a normal parameter. This is the example given in Programming Ruby.

However, this makes the keywords a required parameter. In general, it is much more common for such hashes to be optional parameters.

So the typical case for older Ruby code would be to use an empty hash as the default value of the parameter. This does allocate a new hash every time the method is called without an options hash, which can slow things down.

So high performance code generally would setup a frozen hash constant,

and use that constant as the default value of the hash parameter. This way calling the method without an options hash can avoid allocations.|Let’s assume that this was our existing method in our Ruby 1.8 codebase, and Ruby 2.0 was released and we want to change the option hash to keyword arguments.

In the method definition, we would replace the optional hash argument with separate keyword arguments.

The keyword arguments must have default values, because when keyword arguments were introduced in Ruby 2.0, all keyword arguments were optional. Required keyword arguments were not introduced until Ruby 2.1.

This change requires changing callers of the method. In the Programming Ruby example, string keys are used, and keyword arguments in Ruby 2.0 only supported symbol keys.

So specifying keyword arguments when calling the method requires changing using symbols as keys.

It would probably be fairly common to switch to the simplified hash syntax introduced in Ruby 1.9 when making this change.

If you wanted to support arbitrary keyword arguments, you could add a double splat argument to the method definition.

The double splat syntax could be used when calling the method if you have a hash you want to treat as keywords. In Ruby versions before 2.7, using the double splat operator is optional in most cases, since hash arguments are implicitly converted to keyword arguments.

Ruby 2.0 tried to make the introduction of keyword arguments as backwards compatible as possible. If you were using an option hash approach.

You could switch the option hash to a keyword splat, and everything would continue to work. Ruby would automatically treat the hash as keywords.

You could even use explicit keywords with a hash argument, and this would also work, so long as all keys used in the hash were valid keywords.

Since it was already valid syntax to omit braces for a final hash argument, and the same syntax is used for keyword arguments, backwards compatibility was kept for that. In general, keyword arguments and final positional hashes were considered interchangable.|In most cases, this approach worked fine and did what users wanted it to do. Unfortunately, there were a couple of cases where the approach did not do what you wanted to do.

Mixing optional arguments and keyword arguments generally resulted in undesired behavior if the optional argument could be a hash.

This is because the passed hash argument is treated as keywords instead of as the optional argument.

Similarly, if the method accepted an argument splat as well as keywords.

The passed hash argument is treated as keywords instead of included in the splat arrray.

In both cases, the way to work around the problem would be to stick an empty hash as the last argument.

The empty hash would then be treated as keywords.

Allowing the preceding hash to be treated as an optional argument or member of the splat array.

The keyword argument issues didn’t just affect literal hashes, but any object that is implicitly converted to a hash. Let’s say you passed an instance of YAML::DBM. This class is included in the standard library. Most people reasonably expect this will get passed as the optional argument. However, that is not the case. It gets passed as keyword arguments, even though it is not a hash. Why is that?

Examining the source code of the YAML::DBM class, near the bottom, we see that it defines the to_hash method. This is what causes the problem. In Ruby 2, all methods that accept keyword arguments and are passed more than the number of mandatory arguments will see if the final argument responds to to_hash. If the final argument responds to to_hash, it will call to_hash on the argument, take the resulting hash, and treat that hash as keywords. This caused substantial problems for libraries that defined the to_hash method in any of their classes.

One of the other strange behaviors of keyword arguments was how it handles hashes differently depending on their contents. Let’s use a slightly different example with a method taking an optional argument, explicit keyword, and keyword splat. The method will return the argument values so we can how it handles arguments.

If we pass no arguments

we get nil, nil, and the empty hash.

If we pass an empty hash,

we still get nil, nil, and the empty hash. This is because the empty hash is treated as keywords.

If we pass a hash that has k and y symbol keys,

The hash is treated as keywords, with k being set to the explicit keyword argument, and the remaining entries assigned to the keyword splat.

If we instead pass a hash that has k and y string keys,

The hash is treated as a positional argument. This is because it contains no symbol keys, and Ruby then determines it should not be treated as keywords, since between Ruby 2.0 and 2.6, keyword argument keys could only be symbols. Ruby 2.7 allows non-symbol keys in keyword splats, but because Ruby 2.6 and below and Ruby 3.0 will treat this case as passing a positional argument, Ruby 2.7 does as well.

Finally, if we instead pass a hash that has a k symbol key and a y string key,

Ruby will take such a hash and will split it into two hashes, one containing only symbol keys and another containing all other keys. The hash of symbol keys will be treated as keywords. The other hash will be treated as a positional argument. So in this example, the optional positional argument is set to hash containing the y string key, and the hash with the k symbol key is used to set the keyword argument.

That finishes my discussion of the past. Now let me discuss the present, and by that I mean Ruby 2.7, and how it changed compared to Ruby 2.6 in regards to keyword arguments.

In November 2017, Matz announced at the RubyConf and RubyWorld conferences that Ruby 3.0 will have real keyword arguments, or keyword arguments that are separated from positional arguments.

The discussion of how to implement keyword argument separation occurred in feature 14183 in Ruby’s bug tracker. This is one of the largest discussions in the bug tracker, with over 100 comments in a 2 year period.

The original proposed behavior was for full keyword argument separation, where keyword arguments would never be treated as positional arguments, and positional arguments would never be treated as keyword arguments.

So if we have this code,

where method foo takes a regular argument with a default value being an empty hash,

and method bar takes keyword arguments,

calling the foo method with a hash argument would be fine.

And calling the bar method with keyword arguments would be fine.

However, calling the foo method would keywords would be an error,

and calling the bar method with a hash would be an error.

My main issue with this approach was that it broke backwards compatibility for this case, where you are calling a method with keyword arguments where the method accepts an options hash. My libraries tend to use this pattern extensively.

My libraries tend to use both keyword arguments and hash arguments when calling the methods that accept options hashes. I didn’t want to change these methods to accept keyword splats, because those are substantially worse for performance due to the hash allocations. Also, in many cases the methods that support option hashes also support non-Symbols keys in the hash, and in some cases they support non-hash objects in addition to hash objects. Using keywords would not allow me to handle either of those cases.

I was well aware of the problems with keyword arguments when also used with optional or rest arguments, which were all caused by positional argument to keyword conversion, and I agreed that we should fix these cases.

The first proof of concept patch related to keyword argument separation was posted by Endoh-san last March. This implemented full keyword argument separation. While I only had a little experience with the internals of Ruby back then, I was fairly sure I would not be able to convince Ruby developers to keep backwards compatibility for methods that did not accept keyword arguments unless I had a working proposal with updated tests.

So over the course of the next week, I learned enough about the internals to modify Endoh-san’s patch to implement my proposal. I probably spent equal amounts of time reading existing code, and using trial and error to figure out if my understanding was correct. gdb was both my best friend and my worst enemy. Eventually, I was able to get my proposal working, being backwards compatible for methods without keyword arguments, and fixing the issues for methods with keyword arguments.

When I began my work, I had a goal that all code that will break in Ruby 3.0 due to the keyword argument separation changes will issue deprecation warnings in Ruby 2.7. However, there were a couple of cases where we actually changed the behavior in Ruby 2.7.

One of the changes in Endoh-san’s patch was to allow non-Symbol keys in keyword hashes, if the method accepts arbitrary keywords. This change was made so that more methods that currently accept option hashes can switch to accepting keywords.

In this code were are taking a hash with a string key,

and we are double splatting the key when calling the method.

This results in a TypeError in Ruby 2.6.

But Ruby 2.7 accepts this without an error.

Shortly before modifying Endoh-san’s patch, I realized that in order to fix the delegation issues with keyword arguments, I would need to make splatted empty keyword hashes not pass positional arguments when calling a method that does not accept keyword arguments.

This change was made so that you could write simple delegation methods in Ruby 3. Here is an example method named bar that accepts arbitrary arguments.

And here is a method named foo that delegates all arguments and keyword arguments it receives to bar.

The problem with foo in older versions of ruby is that it does not delegate arguments correctly. If you call bar directly with an argument

You get the expected values returned in both Ruby 2.6 and 2.7.

However, if you call foo with an argument

You get two arguments passed in Ruby 2.6, one being an empty hash, even though an empty hash was not provided when calling foo.

With the changes in Ruby 2.7, you get the expected result for calling foo, exactly the same as if you had called bar directly.

This is why before Ruby 2.7, delegating methods would always use regular arguments and not keyword arguments. This example still works fine in Ruby 2.7 and will work in 3.0 because bar does not accept keyword arguments.

However, it would not work if you change bar to accept keyword arguments.

If you call foo with keywords

Ruby 2.6 is fine, since the keywords are passed as a hash to foo and the hash will be implicitly converted back to keywords when foo calls bar.

Ruby 2.7 results in the same behavior, but issues a deprecation warning, because in Ruby 3.0, the hash will no longer be implicitly converted to keywords when calling bar.

In Ruby 3.0, the hash remains as a positional argument when calling bar.

Delegation turned out to be a much tricker issue than I originally expected. I showed a few slides ago this way to do delegation, delegating both arguments and keyword arguments, and said that it would work correctly in Ruby 2.7. There are actually two problems with this approach.

The first is, if you have a hash, and you provide it as a positional argument to the foo method

Ruby 2.6, 2.7, and 3.0 all return the same result.

However, Ruby 2.7 warns. This has to be a bug, right? Ruby should only be warning in cases where the behavior is going to change, it shouldn’t warn in the case where the behavior is the same. It turns out, this is not a bug.

The warning here comes from this call to foo. In this call to foo, the positional hash argument is converted to keywords, and that is what is causing the wanrning.

Inside foo, the hash is keyword splatted in the call to bar. Because bar does not accept keywords, the splatted hash is turned back into a regular hash argument. That doesn’t warn because Ruby 3 will keeps compatibility with older versions of Ruby for that type of call. With the initial proposal for full keyword separation, this type of code would actually emit two warnings, not just one. It would warn once in the call to foo and again in the call to bar.

The other problem with this approach I have already covered as a positive in terms of Ruby 2.7 behavior. This problem is that you can’t use this approach with older versions of Ruby because you end up passing an empty hash to the target method if calling the delegating method with no keywords where the last positional argument is not a hash.

What I thought we needed was a delegation approach that would be both backwards compatible with older versions of Ruby, and would not issue a warning in cases where the behavior wouldn’t change between Ruby 2.6 and 3.0. Unfortunately, I wasn’t able to come up with a good approach.

I was only able to come up with a passable hack. I originally called the hack pass_keywords,

but it is now known as ruby2_keywords.

The basic idea with ruby2_keywords is that you can keep your existing delegation code that worked in previous Ruby versions.

However, you could use ruby2_keywords to flag the method to pass keywords through the method.

Because the ruby2_keywords method may not be defined in previous Ruby versions, you would need to check whether you could use ruby2_keywords.

If you called the foo method with keywords

They would be converted to a hash and stored as the last element of the splat array in the method. However, the hash would have a special flag.

When an array of args is used in a splat call to another method, if the last element is a hash that has that special flag

Then the hash will be treated as keywords instead of as a positional argument.

This results in the same behavior on Ruby 2.6, Ruby 2.7, and Ruby 3.0. It doesn’t cause any warnings on Ruby 2.7.

One of the reasons for full separation of keywords and positional arguments is that it is always possible to add keywords later without breaking any code. This is called safe keyword extension, and is something we gave up by default when we chose the more compatible approach. However, another change added in 2.7 was the ability to add safe keyword extension to a method.

We can implement safe keyword extension by adding syntax to indicate that it is forbidden to pass keywords to the method.

So if we have a method named foo that takes an arbitrary number of arguments.

You could call it with a hash

and you could call it with keywords to get the same result.

If you add keywords to the method later,

You actually break both of these calls in Ruby 2.7, resulting in an ArgumentError.

In the positional hash case, you also get a warning, since Ruby 2.7 will helpfully convert the positional hash to keywords and warn, before raising an error because the keywords are not valid.|In Ruby 3.0, passing a positional hash will work correctly, since the hash will be treated as a positional argument

but passing keywords will not work.

To avoid this issue, for newly defined methods where you don’t need to worry about backwards compatibility, you can use the star-star-nil syntax when defining the method. This syntax was not valid in Ruby 2.6, but in Ruby 2.7 it is valid and means the method forbids keywords.|It changes the behavior of the method so that that method acts like it accepts explicit keywords, but no explicit keywords are defined.

This will make using keywords with this method raise an ArgumentError.

However, using a positional hash argument will work correctly, even in 2.7.

Ruby 2.7 does not automatically convert this hash to keyword arguments, because we don’t need to worry about backwards compatibility with Ruby 2.6, as the star-star-nil syntax is not valid in Ruby 2.6.

One of the big issues with keyword argument separation was how to handle methods defined in C, both those implemented by core classes and those defined in C extensions. In Endoh-san’s original patch, methods defined in C did not implement keyword argument separation.

In order to support keyword argument separation for C methods, there needed to be a way to expose to such methods whether the method was called with a positional hash or keywords, while keeping the API for calling the C functions the same.

In order to give methods defined in C the ability to check if keywords were passed to the method, I added a function called rb_keyword_given_p to Ruby’s public C-API. This is function is callable from methods defined in C to check whether keywords were passed when calling the method.

Most methods defined in C that accept a variable number of arguments use rb_scan_args in order to parse the arguments. Previously, rb_scan_args treated a hash argument and keyword arguments the same.

Here’s an example of using rb_scan_args.

The first two arguments are argc and argv, usually passed directly from the caller’s arguments.

The third argument is a format string that indicates how elements of argv should be assigned.

Remaining arguments are generally local variables that the elements of argv should be assigned to.

In this case, the first character of the format string is 1, indicating the method takes one mandatory argument.

The second character is 1, indicating one optional argument.

The third character is a star, indicating a rest argument, which combine all remaining arguments into an array.

The last character is a colon, indicating keyword arguments. Up to Ruby 2.6, a final hash argument would always be treated as keywords, mirroring the behavior of methods defined in Ruby. In Ruby 2.7, this modifier uses rb_keyword_given_p to determine whether or not to emit a warning message, again to mirror behavior of methods defined in Ruby.

Adding rb_keyword_given_p and fixing rb_scan_args handled most issues when calling C methods with keyword arguments.

However, there is another side to this coin, which is that C methods can call Ruby methods.

One way to do this is to use rb_funcallv. This function takes the method receiver, an id of the method to call, the number of arguments to call the method with, and a C array of the arguments. One issue with this API is that it does not offer the ability to specify whether you are passing keyword arguments when calling the method.

To add the ability to pass keyword arguments when calling Ruby methods from C, I added a rb_funcallv_kw method, that accepted an additional argument for whether the call passes keywords.

There are actually many C functions that are used to call Ruby methods, and all of these lacked the ability to specify whether keywords were passed when calling.

In all cases, the fix was to add a kw variant of the method that accepts an additional flag for whether keywords are passed. These methods are defined in Ruby 2.7 and later versions, and do not exist in Ruby 2.6 and earlier versions. So C extensions that want to be backwards compatible with earlier versions need to implement these using a macro if they are not defined. Ruby’s extension documentation provides example macros for all of these.

With those changes, both methods defined in C and methods defined in Ruby could handle keyword arguments correctly in Ruby 2.7. Unfortunately, it became apparent that there were numerous special cases that still didn’t handle keyword arguments correctly.

Here is a partial list of special cases that needed to be fixed to handle keyword arguments correctly. Unfortunately, due to limited time, I can’t go over details for these.

That ends the discussion of the present status of keyword arguments as of Ruby 2.7. Let’s finish up with a short discussion about the future of keyword arguments in Ruby.

In early January, the compatibility code and deprecation warnings added in Ruby 2.7 in related to keyword arguments were removed, and positional hash arguments are now never treated as keyword arguments. Keyword arguments are now never split into a positional hash and a keyword hash. Calling a method with an empty keyword splat no longer passes an empty positional hash argument. So while I’m discussing the future, if you have been using the master branch after the release of 2.7, you’ve already experienced these changes.

One idea I had after the release of Ruby 2.7 was an approach to optimize keyword arguments.

Here’s an example of calling a method that takes an explicit keyword with a keyword splat. This allocates a hash on the caller side.

Here’s a slightly modified method accepts arbitrary keywords. This allocates 2 hashes, one on the caller side and one on the callee side.

In both examples, I think it should be possible to avoid the hash allocation on the caller side. We can add a flag for whether the keyword argument passed during the method call is mutable. In a case like this, the flag would not be set, indicating the keyword hash is not mutable.

The callee could take the hash directly and access the :bar member of the hash to set the bar keyword. So this would not allocate a hash.

In the case where the method accepts arbitrary keywords, the caller would pass the hash directly and the callee would duplicate the hash, so this code would allocate one hash instead of two hashes.

Here’s a more advanced case with multiple hash splats when calling. In this case, the caller side has to combine these hashes into a single hash. This case would set the flag, indicating the keyword hash is mutable.

If the method accepts arbitrary keywords, because the mutable flag was set, the callee would not need to duplicate the hash, it could use the hash allocated on the caller side directly, resulting in a single hash allocation instead of two hash allocations.

I was able to implement this approach and it was committed in February. So calls to methods that accept keywords will not allocate more than one hash.

And calls to a method that accepts explicit keywords using a single keyword splat would not allocate any hashes. This significantly improves the performance of most simple methods that use keywords.

I hope you had fun learning about the past, present, and future of keyword arguments.