This content originally appeared on DEV Community and was authored by Jeremy Kreutzbender
Webhook Authentication and Processing for Github, Gitlab, and Bitbucket
I was recently tasked with implementing Gitlab support for Release and to complete that task I needed to implement authentication for Gitlab and a way to handle their webhooks. I completed the authentication first and left the webhook implementation for another pull request as I wanted to refactor the way we were handling all webhooks.
As I started the webhook work, the state of how Release handled webhooks was that Github was using the github_webhook gem and Bitbucket was using some custom built code that lived in a Controller Concern. With the need to add a third client I wanted to align everything into a few classes that allow us to easily onboard more providers if the need ever arose.
As I finished up the work I thought it would be useful to share some things I found about the differences between the three providers and share some of the code that I wrote in case anyone else is trying to do a similar implementation. First I'll talk about a few things I came across and later, in the Technical Details section, I'll go over a few classes we're using in our Ruby on Rails project for handling the webhooks.
I enjoyed doing this work because it spanned a lot different aspects of our codebase, including designing a refactor that would allow for processing webhooks from three different sources, reading documentation and understanding three different APIs, and writing tests to ensure that as we move forward we shouldn't need to every worry about breaking our webhook processing.
Authentication Methods
Further down in the Technical Details section, I'll show some code for the Authenticator class so you'll see the implementation around this topic, but I wanted to touch on how each of the three providers handles authenticating the webhooks. We'll start with Github.
Github - With a Github App, when you create the App, you can supply a secret key which will be used as the basis for authenticating the webhooks for all repositories. Their approach is to combine the payload of the webhook and the secret to generate a hash. The generated hash will be passed as a header when the request is sent to you. The documentation gives an example in Ruby on how to generate your own version of the hash. Then, if the comparison of the two hashes matches, you know the authenticity of the request is valid. I would rate this as the most secure way of authenticating the webhooks because if the request were intercepted and decoded, the secret used to generate the hash is not present anywhere in the request. You can read Github's Securing your webhooks documentation for yourself if you want to learn more
Gitlab - Gitlab follows a similar approach to Github, except that a different webhook object must be created on Gitlab for each Repository (as opposed to the singular Github App). Each webhook installation can take in a secret as it is being created and that secret is sent with the webhook request in the headers. There is no generating of a hash like Github, the secret is simply added to the request. Due to the secret being sent in the header, we decided to generate a different secret for every Repository as we save the Repository in the database. This means that if a request were to be intercepted and decoded, at most a single Repository would be compromised. You can read Gitlab's Webhook documentation for yourself if you want to learn more.
Bitbucket Cloud - Bitbucket is set up in a similar fashion to Gitlab where we have to create a webhook object for each Repository. However, unlike Gitlab, Bitbucket Cloud does not offer a way to add a secret. I came across a JIRA Ticket that was created in 2017 outlining this omission of a way to secure the requests but it is currently still open. It is unfortunate that Bitbucket doesn't offer a way to authenticate the webhook requests as it would allow someone to potentially send requests with bad information. They do offer an alternative solution in their documentation about whitelisting specific IPs that the requests could come from, but my opinion is that they should implement adding the secret and at least follow in Gitlab's approach to send the secret in the headers. You can read Bitbucket Cloud's Manange webhooks documentation for yourself if you want to learn more.
The Action Is Separated From The Event
One aspect I really like with Github and Gitlab is that they differentiate their webhook request through an event and an action. The event is sent as a header in the request and an example would be pull_request (on Github) or merge_request (on Gitlab). There are many different things that can happen with a Pull Request though: it might be one of opened, closed, merged, reopened, and so on. Those different actions that could happen on the Pull Request are sent over in the payload as the key action with the value of the aforementioned states. From a coding perspective this event and action pattern allowed me to create a method, say process_pull_request and inside of that method, handle the many different actions that could occur in another method, say pull_request_opened. I found that designing the code this way allowed for a good abstraction and thorough unit testing of all the different action methods.
The outlier is Bitbucket which added the event and the action together in the header. For example, when a Pull Request is created, the header contains pullrequest:created, when closed pullrequest:rejected, and when merged pullrequest:fulfilled. When using Release for ephemeral environments, closing and merging a Pull Request are considered the same type of action: we will destroy that ephemeral environment. But since the header contains two different values, I had to implement two different methods: process_pullrequest_rejected and process_pullrequest_fulfilled which simply call another method. While it is a pretty minor inconvenience, I like the code pattern of the action and event separated compared to having them combined.
Technical Details
First and foremost I want to acknowledge the great work on the gitub_webhook gem as I used a good amount of what they had done to create the foundation for the Authenticator and Processor classes. What follows is the Ruby code I wrote to manage the webhooks from the three providers that Release currently supports.
The Authenticator
First up, we'll look at the Authenticator class. Its purpose is to authenticate the webhooks that we are receiving to ensure that they're valid. You'll see that there is an optional parameter for the Repository and it is optional because as I mentioned in the Authentication Methods above, for Github we have a single secret, while for Gitlab and Bitbucket a secret is generated for each Repository.
Aside from the initialization method, the class has a single public method, authenticate_request! which does as it is named. It will raise an error if the authenticity of the request cannot be validated otherwise the call will return. The expected_signature method follows the different providers implementations with Github needing to create a hash to compare, Gitlab needing only the secret, and Bitbucket currently using a random string due to not offering an authentication method.
module Webhooks
class Authenticator
class SignatureError < StandardError; end
def initialize(request:, vcs_type:, repository: nil)
@request = request
@vcs_type = vcs_type
@repository = repository
end
def authenticate_request!
secret = client_secret(@vcs_type, @repository)
request_signature = signature_header(@vcs_type, @request)
expected_signature = expected_signature(@vcs_type, secret, @request)
unless ActiveSupport::SecurityUtils.secure_compare(request_signature, expected_signature)
raise SignatureError
end
end
private
def request_body(request)
@request_body ||= (
request.body.rewind
request.body.read
)
end
def signature_header(vcs_type, request)
@signature_header ||= (
case vcs_type
when :github
@request.headers['X-Hub-Signature-256']
when :gitlab
@request.headers['X-Gitlab-Token']
when :bitbucket
'bitbucket_cloud'
end
)
end
def expected_signature(vcs_type, secret, request)
digest = OpenSSL::Digest.new('sha256')
case vcs_type
when :github
"sha256=#{OpenSSL::HMAC.hexdigest(digest, secret, request_body(request))}"
when :gitlab
secret
when :bitbucket
'bitbucket_cloud'
end
end
def client_secret(vcs_type, repository)
case vcs_type
when :github
Clients::Github.webhook_secret
when :gitlab, :bitbucket
repository.webhook_secret
end
end
end
end
The Processor
If the request is authenticated, then we need to process the payload that comes with the request and the Processor class does just that. It will look through the payload and try to find the associated Repository in our database, if that Repository cannot be found, then an error occurs. To determine what event occurred, we look through the different headers in the request and parse the value into Ruby method declaration form by replacing any non-word character with an underscore. Based on the provider who sent the request, a service object is initialized and then we attempt to call a process_ method. Some webhooks we receive are for things Release doesn't deal with, for example Github's issues webhooks, so we safely try the method as there may not be an implemented process_ method.
module Webhooks
class Processor
def initialize(request, vcs_type)
@request = request
@payload = json_body(request)
@vcs_type = vcs_type
@repository = repository_from_payload(@vcs_type, @payload)
@webhook_service = webhook_service(@vcs_type, @payload, @repository)
end
def repository
@repository
end
def process_webhook
process_method = "process_#{event_method(@vcs_type, @request)}"
@webhook_service.try(process_method)
end
private
def json_body(request)
payload = request.body.read
ActiveSupport::HashWithIndifferentAccess.new(JSON.load(payload))
end
def repository_from_payload(vcs_type, payload)
provider_repository_id = provider_repository_id(vcs_type, payload)
if provider_repository_id
Repository.find_by!(type: "Repositories::#{vcs_type.capitalize}", provider_repository_id: provider_repository_id)
end
rescue ActiveRecord::RecordNotFound => error
# Re-Raise the error with info from the payload so we know what the repository is
repository_info = payload.dig('repository', 'full_name')
new_error = ActiveRecord::RecordNotFound.new(error.message + "Repository Info: #{repository_info}")
new_error.set_backtrace(error.backtrace)
raise new_error
end
def provider_repository_id(vcs_type, payload)
case vcs_type
when :github
payload.dig('repository', 'id')
when :gitlab
payload.dig('project', 'id')
when :bitbucket
payload.dig('repository', 'uuid')
else
nil
end
end
def event_method(vcs_type, request)
@event_method ||=
(
case vcs_type
when :github
request.headers['X-GitHub-Event']
when :gitlab
request.headers['X-Gitlab-Event']
when :bitbucket
request.headers['X-Event-Key']
else
nil
end
)&.downcase&.gsub(/\W/, '_')&.to_sym
end
def webhook_service(vcs_type, payload, repository)
service_class = "Webhooks::#{vcs_type.to_s.capitalize}"
service_class.constantize.new(payload, repository)
end
end
end
Github Webhook Service
The last method in Processor, webhook_service returns a service class that goes through our internal business logic of what we want to do with
the webhook. I'm going to provide a small snippet of the Github service when we receive a Pull Request webhook. If you recall, I mentioned this method in the The Action Is Separated From The Event section and how I liked this pattern of structuring the code. If someone else were to look at this code, I would hope they would find it easy to understand that anything to do with Github Pull Request webhooks happens inside of the process_pull_request method and the case statement handles all the different actions that can take place.
module Webhooks
class Github
def initialize(payload, repository)
@payload = payload
@action = @payload.dig('action')
@repository = repository
end
def process_pull_request
if @action.nil?
error_message = "ERROR: Pull Request no action received, payload : #{@payload}, do nothing"
Rails.logger.error(error_message)
else
message = "Pull Request with action : *#{@action}*. Received Repository : #{@repository.name}."
Rails.logger.info(message)
case @action
when 'opened', 'reopened'
pull_request_opened
when 'closed'
pull_request_closed
when 'labeled'
pull_request_labeled
else
message = "Pull Request with action : *#{@action}*. Nothing to do for now."
Rails.logger.info(message)
end
end
end
end
end
The Controller
The final piece to tie everything together is the controller. WebhooksController is the base class and each subclass implements only the vcs_type method. Our previous approach had custom code for each of the Webhooks::GithubController and Webhooks::BitbucketController. This meant that each required a ton of specific tests to ensure that we were processing all the different webhooks correctly. My refactored approach moved all that logic out of the controller and aimed for the smallest footprint possible to make testing as simple as possible.
There is only one route in the controller, which is a POST to create. I decided that since the Repository may be optional in the Authenticator that I will store it in the Processor and pass it into the Authenticator. Otherwise you can see that the public methods for each of the classes are called. If an error is raised by either, due to an unauthenticated webhook or possibly a webhook for a Repository we don't have in our database, we'll capture the error and log as much information as possible so that we can look into what went wrong.
class Webhooks::GithubController < WebhooksController
vcs_type(:github)
end
class WebhooksController < ActionController::Base
skip_before_action :verify_authenticity_token
def self.vcs_type(vcs_name)
define_method :vcs_type do
vcs_name
end
end
rescue_from StandardError do |error|
payload = @webhook_service&.payload
backtrace_cleaner = ActiveSupport::BacktraceCleaner.new
cleaned_backtrace = backtrace_cleaner.clean(error.backtrace)
error_message = "Error in #{self}! Message : #{error.message}\nPayload : #{payload}\nBacktrace : #{cleaned_backtrace.join("\n")}"
Rails.logger.error(error_message)
head :bad_request
end
def create
processor = Webhooks::Processor.new(request, vcs_type)
repository = processor.repository
authenticator = Webhooks::Authenticator.new(request: request, vcs_type: vcs_type, repository: repository)
authenticator.authenticate_request!
processor.process_webhook
head :ok
end
end
Conclusion
That's a wrap on my stint in refactoring our webhook code to work with Github, Bitbucket, and Gitlab. If we ever have to add another provider I think it will be quite straightforward and I hope you enjoyed taking a peek inside some development work at Release. If you're interested in having an ephemeral environment created whenever we receive a Pull Request webhook from your Repository, head on over to the homepage and sign up!
This content originally appeared on DEV Community and was authored by Jeremy Kreutzbender
Jeremy Kreutzbender | Sciencx (2021-06-03T15:37:20+00:00) Webhook Authentication Learnings for GitHub, GitLab, and Bitbucket. Retrieved from https://www.scien.cx/2021/06/03/webhook-authentication-learnings-for-github-gitlab-and-bitbucket/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.