Good morning everyone. I am excited and honored to be opening the final day at the inaugural Isle of Ruby.|Today I will be speaking about my experiences running the programming unit for a small government department for the last 15 years, using Ruby as the main programming language for the last 12.

My name is Jeremy Evans, and I have been using Ruby since 2004. I am the maintainer of numerous ruby libraries,

the most popular of which is Sequel, the database toolkit for Ruby. I also maintain a web toolkit for Ruby named Roda, as well as an authentication framework named Rodauth that builds on top of Sequel and Roda.

Now, I did not start out wanting to maintain these and a bunch of other Ruby libraries. The reason I maintain these libraries is because they form the core of many personal projects as well as the core of the applications I am responsible for at work.

Our equivalent at the national level here in the United Kingdom would be the National Audit Office.

Before I go any further, please be advised that all opinions in this presentation, and especially the following section, are my personal opinions and not the opinions of my department.

I would now like to talk about software development as it is typically done at other departments in my state, though I will simplify it substantially in the interests of time, and exaggerate slightly for comedic effect. I am not sure how similar it is in other states or countries, but I am guessing there is some overlap.

Government software development resolves around large projects. In most cases, there is an existing system in place, but it has various warts, such as running on a mainframe. So executives propose building a modern system, and throwing away the previous system.

The government will start by developing a long request for proposal detailing what they think the requirements are for the system. The request for proposal will be prepared by program managers and analysts without the involvement of the internal developers who maintain the current system.|Then the government will solicit proposals from companies to build the system. After removing proposals that are not considered acceptable, the government will follow standard government purchasing regulations and award the contract to build the system to the company with the lowest bid.

Payment on the contracts will be based on feature checklists or deliverables. There will be little consideration for how easy the software is to use, and no consideration for how easy the software will be to maintain.

The contractor will build the system using their own developers who are incentivized to complete development as quickly as possible. The contractors will use C# if using a Microsoft stack or Java otherwise. Near the end of the contract, the contractor will attempt to train government staff on how to maintain the system.

There will be minimal if any unit or model testing done during development. There will be decent integration or acceptance testing, but the testing will be performed manually by government employees or contractors using checklists.

In some cases, the contractor will base their solution on an existing expensive enterprise resource planning system such as a SAP or Peoplesoft, and then the contractor will heavily customize the installation to meet the system requirements.|This will allow the contractor to claim that they are just using common off the shelf software in their proposal, when in reality so much custom work will be done that most of the benefits of using common of the shelf software would not apply.

There will be a large amount of oversight of the project. The project managers will write quarterly progress reports, and send them to executive management, the Department of Technology, and an external independent project oversight company.|When the project runs into problems during development, the stakeholders overseeing the project will be aware, but as they are not experts in software development, other than approving the expenditure of more money or lengthening the project schedule, there will not be much they can do to fix the problems.

As you might be able to guess, I am not a fan of this approach for building software. I will now share the alternative approach to software development that we use in our department.|Now, this is not a fair comparison, as our department is small and not similar to other departments.

First, we try as much as possible to avoid building large systems. If stakeholders request a large system, we discuss the situation with them and try to convince them to build a smaller system initially that only does what is most important. In some cases, we are successful, in some cases we are not, but we always try to reduce the initial scope if a large system is requested. We can always add more features later, after the system is working, that is what we tell people.|The idea that you should start with a working simple system before expanding the complexity, instead of trying to build a complex system from the start, has been around for decades.

John Gall wrote a book called Systemantics back in 1975, which discusses how systems work and how they fail.

A couple sentences in that book eventually became referred to as Gall’s Law.

Gall’s Law states that a complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.| Note that Gall was not referring to software development, he was discussing systems in general, be they computer based, mechanical, or manual.

We have Gall’s Law in mind when we build systems in our department. In general, we try to add them as subsystems or new features to our existing working systems. We aim to complete the development of almost all projects in under a month, with the majority being completed in under a week.|When I say completed, I mean having all features implemented with complete automated tests. Systems may take longer to go into production due to delays in getting approval from the stakeholders that requested the system.

A typical project for us would be to take a paper form based process with one or two levels of review and integrate it into our intranet site. There would be an online form for employees to complete, forms for the reviewers to approve or deny the request, emails for notifying the next reviewer at each step, and reports for the employee, reviewers, and management to see the status of requests.

We build all of these systems internally using government staff, with no use of contractors. The developer responsible for building the system will meet directly with all interested stakeholders to determine requirements.

Often, stakeholders will request we add features to the system that are not strictly needed, or will request a specific feature that they think will meet their needs, instead of just letting us know what their needs actually are.|It is the developer’s job to discuss the system requirements with the stakeholders, figure out what features are important, what the underlying needs actually are, and then build the simplest system that will meet the needs of the stakeholders.

The developer responsible for building the system knows that they will be responsible for maintaining the system when it is in production, which encourages them to design the system in a way that will make maintenance easiest.

The developer also knows that if users have trouble using the system, they will be the one to listen and respond to the users’ complaints, and it will be their job to modify the system to make it easier to use. This encourages the developer to design the system to be easy to use, so they do not have to make such modifications in the future.

As the title of this presentation suggests, all of our internal development is done in Ruby. Ruby has been our primary development language for new projects since mid-2005, and all of our existing internal systems were converted to Ruby by 2009. I will talk a little bit later about the stack we use and how it has evolved over time.

We do not have a large amount of oversight during our development process.

The only external oversight is from external security assessments and penetration tests, which are each performed every 3 years.

In terms of internal oversight, the developer will work on the system until they think it is ready. If they run into problems during development and need help, then they will talk to me and we will pair program to fix the problem. After the developer thinks the system is complete, then they will request a code review from me. I will review all of the code, provide a list of requested changes, and that process will repeat until all issues have been addressed.

After a clean code review, the developer will notify the stakeholders, who will use a development version of the system and see if it meets their needs. The stakeholders may request changes at that point, in which case we will discuss the changes with them and agree on how to handle the situation, similar to the initial discussion with them regarding requirements.

Before any of our systems go into production, we require automated testing at the model level and web level for all parts of the system. We perform coverage testing on a regular basis for all of our applications, and our line coverage is between 93 and 100 percent depending on the system.

Since we prefer to build small systems instead of large systems, we generally have multiple outstanding requests to build systems at any point in time. So one of the issues we deal with is how to prioritize the requests.

One of the things we consider is the size of the request. We try to build smaller systems before larger ones. Hopefully this encourages stakeholders to be more willing to accept a smaller system.

Another thing we consider is how often the system will be used. If this system automates a manual paper form-based process, we will ask: how many forms are submitted per month? Depending on the volume, we may give the system high priority, or even tell the requester that due to the low volume, it does not make sense to automate the process.

We also try to consider how important the system is to the organization. If the current process is paper form-based, we will ask: what the consequences are of losing a form? If there are legal issues, we may want to automate a system with low volume just to ensure we can closely track progress to make sure we are following the law.

Finally, we consider who is requesting the system. As much as we like to be egalitarian, if the requester is an executive and they want us to give the system priority, then the system will get priority.

So, what do we use Ruby for? I mentioned that Ruby is the sole language we use for software development, so this is basically the same question as: what software development do we do?

As you may expect, the primary software development we do is web applications.

Our largest application is our intranet site, called The Hub. The majority of our development is adding new features to The Hub.|The Hub has some pretty standard intranet features. It contains a lot of information on internal processes, our comprehensive manual, and our employee directory.

Each employee has a profile page showing information about the employee, such as their division and position, audits they have worked on, audits they are currently working on, and awards they have received.

The employee’s profile page also has a link to a map of the floor that they work on, with their desk location highlighted. This makes it easy for new staff to navigate the office, and for existing staff to easily find new staff.

Most of the new development centers around automating existing processes. When I started, almost all processes were manual, paper-based processes. Now, most of the common processes are automated via online forms.|For example, submitting requests to take time off, attend training, get reimbursed for overtime, modify facilities, purchase supplies and equipment, and many others are automated. Records are kept so employees can see the status of all previous requests.

Most of the processes have custom review/approval workflows based on different requirements. Some processes will only require supervisory review. In other cases, there will be between two and five levels of review, with the number of levels dependent on the specifics of the request and the employee’s position in the department.

Another web application that we develop is our recruiting system. Most of the employees that we hire are entry level auditors directly out of college or graduate school, and most of recruiting system is designed to handle the recruiting process for these applicants.

The recruiting system allows prospective employees to apply to take our online exam. After applying, our human resources staff review the application, and if they approve it, the applicant is notified and can take our online exam.

The online exam is timed and has 75 muliple choice questions. You have to get about 80% of the questions correct to rank highly enough to advance. The exam is difficult and only about 30% of applicants rank highly enough to advance.

Assuming the applicant scores highly enough, they are notified they can advance further and can take our online writing assessment. When the applicant begins the writing assessment, they are given a prompt and an upload form, and they have two hours to upload a writing sample similar in style to our audit reports. This writing assessment is graded by our in house editor, and only about one third of applicants score highly enough to advance.

After that, there is a phone interview and then an in-house interview, and the system handles the information related to those. The system also handles all internal workflows related to processing these applications, and has extensive reporting capabilities. There are also smaller subsystems of the recruiting system related to recruiting for other, more advanced auditing positions.

The last major web application we develop is our recommendations system. In addition to reporting our audit findings, one of our primary functions is to make recommendations to improve government. The other departments we audit are required to respond to our recommendations on a regular basis on their progress implementing our recommendations, until the recommendations have been fully implemented.

The recommendations system allows our staff to add recommendations, and allows other government departments to respond to the recommendations online through our website. Each response goes through four levels of review, and after being fully reviewed, the department’s response and our assessment of their response is posted on our website. This holds other departments accountable, and their implementations of our recommendations are often considered when the Legislature reviews the department’s budget.

As I mentioned, we use Ruby for all development, so I would like to talk a bit about some other systems we develop.

For employees to login to any of our web applications, we want them to use the same Windows username and password that they use to login to their computer, so they do not have to remember a separate password.

However, from a security perspective, we do not want our web servers talking directly to our domain controllers. So all of our web applications authenticate using SSL to connect to a custom authentication proxy written in Ruby. The web applications submit the username and password provided by the user. This proxy first checks that the user is in the list of allowed usernames, then connects to Windows using LDAP over SSL to authenticate the password.

We also use Ruby to implement 2-factor authentication for our VPN. We use OpenVPN, which does not have native support for 2-factor authentication. In order to allow 2-factor authentication with the VPN certificate as the first factor and the Windows password as a second factor, after OpenVPN authenticates the certificate, we have OpenVPN run a statically compiled C program which uses a Unix socket to connect to a server written in Ruby. That Ruby server connects to Windows using LDAP over SSL to authenticate the password for the username specified in the VPN certificate.

We use Ruby to download financial information from the state’s mainframe on a daily basis for usage by our financial audit team. This used to require a gem that supported FTP over SSL, but thankfully after ruby 2.4 was released, we were able to switch to using net/ftp from the standard library.

We develop custom programs to assist auditors with their audit work. Most recently, we used ruby with capybara, selenium, and headless chrome to download almost 3000 PDFs from a separate government website written in ASP.NET which required Javascript. We then used ruby, pdftk, and mupdf to extract a specific page from each PDF and combine those extracted pages into a single PDF so that the auditors could easily go through the results and include the data in their audit work papers.

We have many reporting programs that use Sequel to extract data from internal Microsoft SQL Server databases and create reports from them.

Basically, Ruby is our programming toolbox, and pretty much everything we need to build we can easily build using Ruby.

So how did we start using Ruby? When I was first given the task of maintaining our websites back in 2003, they were developed as static pages using Net Objects Fusion.

While I had no previous professional programming experience, I did have some exposure to PHP, and I decided to use that.

In late 2004, I heard about Rails, and tried it out. I saw that it was a great improvement over the spaghetti PHP I was using. After a few months of using it in personal projects and a few more testing it at work, I switched our intranet site to Rails in mid-2005.

In 2008, I learned about Sinatra, and was drawn to Sinatra’s much simpler approach to web development. We started using Sinatra for all new development, and the initial versions of our recruiting and recommendations systems were written in Sinatra.

On the ORM side, when I first starting using Rails, I used ActiveRecord, which was a huge time saver compared to manually writing all SQL queries as I was doing in PHP.

After being exposed to Sequel in 2008, I saw the benefits of Sequel’s method chaining approach to building queries. I converted all ActiveRecord usage to Sequel that year, and we have been using Sequel exclusively since.

In the lower levels in the stack, we’ve always used PostgreSQL as the database, starting with version 7.1.

The operating system has always been OpenBSD, since we were already successfully using OpenBSD for our firewall and had experience with it, and security and simplicity are more important than performance in our environment.

For the web server, we originally used Apache and Webrick, then Lighttpd and SCGI, then Nginx and Mongrel, and finally Nginx and Unicorn, which we have been using successfully for many years.

On the testing side, we use minitest, minitest-hooks, rack-test, and capybara for testing.

All of our web applications are explicitly designed around a simple web 1.0 experience, though we do use HTML5 features such as date inputs and required inputs.

We avoid using javascript as much as possible, only using it when there is no other way and we absolutely must have dynamic behavior on the web page. Most commonly, this is for input forms that can accept an arbitrary number of line items.

We use so little javascript that we manually test the javascript that we do use when we modify the related code.

All of our applications run on the same stack described here, and use the same library versions. This makes it much easier to switch between applications during development.

Whenever we upgrade a library, we run one command, which runs the tests for all of our applications using the new library version, and we can see if anything breaks.

This command can also be used to test on multiple versions of Ruby, and all of our applications pass their test suite using Ruby 2.1 to 2.4.

After upgrading libraries, we may decide to use new features added to the libraries. This is generally done starting with our simplest application, and then applied to our remaining applications.

A couple of examples of this are when we switched to using frozen Sequel datasets, databases, and models, and when we starting using refrigerator to freeze all core classes during runtime.

We used the same strategy when we rolled out automatic escaping in templates and when we switched our parameter access to use Roda’s typecast params plugin.

Obviously when you choose Ruby without Rails, you are going to limit the number of developers you can hire that already have experience with your stack. The good news is that in our experience, a Roda and Sequel based web stack is easy for new programmers to learn.

You should not extrapolate from a sample size of one, but our current developer had no professional programming experience and had never programmed in Ruby before we hired her. She was able to quickly become productive and implement new features using Sequel and Roda. If we were using Rails, I think it would have taken her significantly longer to become productive.|For some companies, it may be easy to find developers that already know Rails. However, we attempted to recruit developers at many different experience levels and specifically highlighted that we use Ruby. We did not even have a single person apply with any Ruby or Rails experience, so Rails’ popularity advantage would not really matter to us.

As I am the department’s Information Security Officer, one specific focus area for me is security. As you would expect, we try to protect against the common vulnerabilities in web applications.

We attempt to mitigate cross site scripting using Roda’s support for automatically escaping output in templates.

We protect against cross site request forgery using rack csrf.

We protect against SQL Injection by using Sequel to construct all queries. We actually do not have any raw SQL usage in any of our web applications.

We use a restrictive Content-Security-Policy in all of our applications, to mitigate possible damage in case an attacker is able to exploit a cross site scripting vulnerability.

However, we go a lot further than that in terms of defense in depth security planning. A lot of our security planning starts by assuming there is a SQL injection or remote code execution vulnerability, and implementing features that would make the vulnerability more difficult to exploit and limit the damage it could do.

We run all of our applications on our own hardware, in isolated subnets, with strict ingress, egress, and loopback firewall rules that limit the types of connections that can be made to and from the servers.

All applications run with separate operating system users with reduced privileges, and with separate database users per application.

For applications with public facing components, such as the public parts of our recommendations and recruiting systems, those run as separate operating system users with even fewer privileges, and separate database users that are only granted the minimum access needed to perform the public facing functions.|So if there was a vulnerability in our public recruiting system, a user may be able to exploit it, but they would not be able to change their exam or writing assessment scores, since the database user does not have the privileges to do that. Likewise, a vulnerability in the public recommendations system could only be used to add responses, not to update existing responses.

To mitigate arbitrary file access and remote code execution vulnerabilities, we run all of our applications chrooted. Our applications are started as root, and after they are loaded, but before accepting connections, they chroot to the working directory of the application, then drop privileges to the application’s operating system user.

To make it more difficult to execute blind return oriented programming attacks that are based on exploiting consistent memory layouts, we have each Unicorn worker process exec after forking, so that all Unicorn worker processes have unique memory layouts.

Finally, to make exploitation more difficult and to reduce the kernel attack surface for privilege escalation, we limit the set of allowed kernel system calls for the application to the minimum the application needs to function.

Combined, these security controls make it more difficult to successfully exploit our applications and make it more difficult to use a successful exploit in order to attack other applications and systems.

If you are interested in more detail regarding these security controls as well as many of the lessons we have learned from implementing them, I will be giving a presentation at RubyHACK in Salt Lake City next month that discusses them in more detail.

I would like to finish this presentation with a couple opinions. First, my opinion on what a successful government IT project needs.

It needs executive management that is willing to try new approaches.

Grace Hopper, one of the creators of COBOL, stated that the most dangerous phrase in the language is “we’ve always done it this way”. Considering the track record for government IT projects of being delivered late, over budget, and full of bugs if they work at all, a new development approach seems worth trying.

Next, the project managers need a deep understanding of the technology used, so they will know what problems the system actually has, and how the problems should be fixed, as soon as possible.

The developers building the system should be discussing the requirements directly with system’s stakeholders, with an eye to reducing the scope of the system to the minimum.

The developers building the system should be responsible for maintaining it, so they have an incentive to make maintenance easy and design the system to be easy to use from the beginning, so that they do not have to make such modifications later.

Finally, all systems should start by developing the simplest possible system that handles the most important requirements, making it work well, and gradually building on top of the working system, to not run afoul of Gall’s Law.

My second opinion comes from a lot of personal experience, and that is that Ruby is a great fit to build the types of systems of the government needs, at least for those systems where runtime performance is not critical.

Ruby makes it very fast to develop systems quickly and get to a working state.

Assuming you have good tests that cover most of the system’s functionality, it is fairly easy to maintain Ruby applications and modify them as requirements change, though a lot of that depends on your choice of Ruby libraries.| My experience is that requirement changes cause much more code changes than refactoring changes, so the limited ability to use automated refactoring in Ruby is not a major issue.

Ruby is easy to learn, and easy to teach to new programmers. This is especially important for government work, since in my experience, applicants for government programming positions will probably not have experience with Ruby.

Importantly, Ruby keeps programming fun. I think the importance of this is underrated. Government work does not pay very well, so having a fun language to program in can help retain talented staff.

After handling all programming for a government department for the past 15 years, first as the sole developer and now as a manager, I can confidently say that Ruby is a great choice for application development, and I highly recommend it.

That concludes my presentation. I would like to thank all of you for listening to me.

If any of you have questions, I will be happy to answer them now.

Photo credits