Apr, 2021

Testing external APIs with Rspec and WebMock

Testing code you don't own can be a bit trickier than writing tests for code you own. If you're interacting with 3rd party APIs, you don't actually want to make real calls every time you run a test.

Reasons why you should not make real requests to external APIs from your tests:

it can make your tests slow
you're potentially creating, editing, or deleting real data with your tests
you might exhaust the API limit
if you're paying for requests, you're just wasting money
...

Luckily we can use mocks to simulate the HTTP requests and their responses, instead of using real data.

I've recently written a simple script that makes a POST request to my external newsletter service provider every time a visitor subscribes to my newsletter. Here's the code:

module Services
  class NewSubscriber
    BASE_URI = "https://api.buttondown.email/v1/subscribers".freeze

    def initialize(email:, referrer_url:, notes: '', tags: [], metadata: {})
      @email = email
      @referrer_url = referrer_url
      @notes = notes
      @tags = tags
      @metadata = metadata
    end

    def register!
      uri = URI.parse(BASE_URI)
      request = Net::HTTP::Post.new(uri.request_uri, headers)

      http = Net::HTTP.new(uri.host, uri.port)
      http.use_ssl = true
      http.request(request, payload.to_json)
    end

    private

    def payload
      {
        "email" => @email,
        "metadata" => @metadata,
        "notes" => @notes,
        "referrer_url" => @referrer_url,
        "tags" => @tags
      }
    end

    def headers
      {
        'Content-Type' => 'application/json',
        'Authorization' => "Token #{Rails.application.credentials.dig(:buttondown, :api_key)}"
      }
    end
  end
end

The first approach could be to mock the ::Net::HTTP ruby library that I'm using to make the request:

describe Services::NewSubscriber do
  let(:email) { 'user@example.com' }
  let(:referrer_url) { 'www.blog.com' }
  let(:tags) { ['blog'] }
  let(:options) { { tags: tags, notes: '', metadata: {} } }

  let(:payload) do
    {
      email: email,
      metadata: {},
      notes: "",
      referrer_url: referrer_url,
      tags: tags
    }
  end

  describe '#register' do
    it 'sends a post request to the buttondown API' do
      response = Net::HTTPSuccess.new(1.0, '201', 'OK')

      expect_any_instance_of(Net::HTTP)
        .to receive(:request)
        .with(an_instance_of(Net::HTTP::Post), payload.to_json)
        .and_return(response)

      described_class.new(email: email, referrer_url: referrer_url, **options).register!

      expect(response.code).to eq("201")
    end
  end

This test passes but there are some caveats to this approach:

I'm too tied to the implementation. If one day I decide to use Faraday or HTTParty as my HTTP clients instead of Net::HTTP, this test will fail.
It's easy to break the code without making this test fail. For instance, this test is indifferent to the arguments that I'm sending to the Net::HTTP::Post and Net::HTTP instances.

Testing behavior with `WebMock`

WebMock is a library that helps you stub and set expectations on HTTP requests.

You can find the setup instructions and examples on how to use WebMock, on their github documentation. WebMock will prevent any external HTTP requests from your application so make sure you add this gem under the test group of your Gemfile.

With Webmock I can stub requests based on method, uri, body, and headers. I can also customize the returned response to help me set some expectations base on it.

require 'webmock/rspec'

describe Services::NewSubscriber do
  let(:email) { 'user@example.com' }
  let(:referrer_url) { 'www.blog.com' }
  let(:tags) { ['blog'] }
  let(:options) { { tags: tags, notes: '', metadata: {} } }

  let(:payload) do
    {
      email: email,
      metadata: {},
      notes: '',
      referrer_url: referrer_url,
      tags: tags
    }
  end

  let(:base_uri) { "https://api.buttondown.email/v1/subscribers" }

  let(:headers) do
    {
      'Content-Type' => 'application/json',
      'Accept'=>'*/*',
      'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
      'User-Agent'=>'Ruby'
    }
  end

  let(:response_body) { File.open('./spec/fixtures/buttondown_response_body.json') }

  describe '#register!' do
     it 'sends a post request to the buttondown API' do
        stub_request(:post, base_uri)
          .with(body: payload.to_json, headers: headers)
          .to_return( body: response_body, status: 201, headers: headers)

       response = described_class.new(email: email, referrer_url: referrer_url, **options).register!

       expect(response.code).to eq('201')
    end
  end
end

Now I can pick another library to implement this script and this test should still pass. But if the script makes a different call from the one registered by the stub, it will fail. This is what I want to test - behavior, not implementation. So if, for instance, I run the script passing a different subscriber email from the one passed to the stub I'll get this failure message:

1) Services::NewSubscriber#register! sends a post request to the buttondown API
     Failure/Error: http.request(request, payload.to_json)                                                                                                                        

     WebMock::NetConnectNotAllowedError:
       Real HTTP connections are disabled. Unregistered request: POST https://api.buttondown.email/v1/subscribers with body '{"email":"ana@test.com","metadata":{},"notes":"","ref
errer_url":"www.blog.com","tags":["blog"]}' with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Authorization'=>'Token 26ac0f19-24f3-4ac
c-b993-8b3d0286e6a0', 'Content-Type'=>'application/json', 'User-Agent'=>'Ruby'}

       You can stub this request with the following snippet:

       stub_request(:post, "https://api.buttondown.email/v1/subscribers").
         with(                              
           body: "{\"email\":\"ana@test.com\",\"metadata\":{},\"notes\":\"\",\"referrer_url\":\"www.blog.com\",\"tags\":[\"blog\"]}",
           headers: {                                                                    
          'Accept'=>'*/*',                                                                                                                                                        
          'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
          'Authorization'=>'Token 26ac0f19-24f3-4acc-b993-8b3d0286e6a0',            
          'Content-Type'=>'application/json',                                                                                                                                     
          'User-Agent'=>'Ruby'
           }).                                                                           
         to_return(status: 200, body: "", headers: {})

       registered request stubs:            

       stub_request(:post, "https://api.buttondown.email/v1/subscribers").                                                                                                        
         with(                              
           body: "{\"email\":\"user@example.com\",\"metadata\":{},\"notes\":\"\",\"referrer_url\":\"www.blog.com\",\"tags\":[\"blog\"]}",                                                    headers: {                       
          'Accept'=>'*/*',
          'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
          'Content-Type'=>'application/json',
          'User-Agent'=>'Ruby'
           })                               

       Body diff:                           
        [["~", "email", "ana@test.com", "user@example.com"]]

There's not much more to it than this. VCR is another tool that also stubs API calls but works a little differently by actually making a first real call that will be saved in a file for future use. For simple API calls, WebMock does the trick for me!