Spring Social Bootstrap: Create REST API SDKs and CLIs That Can Record and Replay HTTP Requests

I joined SportsLabs (then still under the Silver Chalice brand) way back in 2011 as one of its earliest employees and the first engineer.

We started work on envisioning and building the Advanced Media Platform - a system to ingest, process, transform, distribute, and stream sports, news, social, and media content to create market leading mobile, web, and social products for clients such as Samsung, the University of Notre Dame, the ACC, the College Football Playoff, IMG College, the Mountain West and Campus Insiders, among others.

Since then, SportsLabs has consumed data from dozens of sources including STATS LLC, Twitter, and Ooyala, but also from proprietary systems that were never foreseen as integration points.

Data providers’ APIs use combinations of JSON, XML and/or CSV. Some are spec-compliant, others are not. Some rely heavily on query parameters, while others favor HTTP headers. Some API providers use OAuth 2.0 plus API rate limits, while others have rolled their own security solutions. Some integrations were with partners willing to work with us on evolving their web services. Others were with competitors who were not motivated to make things easy.

This plethora of ways to configure, consume, learn from, and integrate with APIs led us to create Spring Social Bootstrap, a family of projects intended to aid creating and managing API clients for many of the above scenarios.

Spring Social Bootstrap is comprised of the following:

Spring Social Bootstrap SDK

Spring Social Bootstrap SDK is a simple Spring Social-based framework for rapidly and consistently developing API clients.

It combines the strengths of Spring Social, such as native support for OAuth-based service providers (including support for OAuth 1 and OAuth 2), with consistent and well-defined extension points for new API clients.

Spring Social, despite its name, has no requirement on integrations being only with social networks. In truth, its primarily a set of interfaces for predictable implementations of API clients - which is our goal too.

Spring Social Bootstrap SDK provides CRUD and query capabilities out-of-the-box e.g.

1
2
3
4
5
testApi.testOperations().create(testBaseApiResource);
testApi.testOperations().get(testBaseApiResource.getId());
testApi.testOperations().update(testBaseApiResource);
testApi.testOperations().delete(testBaseApiResource.getId());
testApi.testOperations().query();

as well as guidelines on how to create developer-friendly features like query builders e.g.

1
testApi.testOperations().qb().withPaging(1, 25, "name", Sort.Direction.ASC).query();

I’d advise taking a look at the Spring Social Bootstrap SDK README to learn more on how to create API clients based on Spring Social Bootstrap SDK.

Bootstrap Shell

Bootstrap Shell is a Spring Shell-based framework to aid creating command-line interface (CLI) applications for API clients built on Spring Social Bootstrap SDK.

When developing API clients, SportsLabs learned that is worth leveraging the clients in data monitoring, QA, testing, and simulation scenarios. As such, we created a small CLI framework to allow highly interactive command-line applications as well as taking advantage of the power of command line tools, such as pipes, xargs, jq, and even opening the browser to online side-by-side diff tools like mergely.

We also liked Spring Shell’s Spring-based interactive shell, but the XML-only configuration, the toString()-based command outputs and lack of menu-style multi-step support required a fork of the project to add the following:

The result was a handy framework for developers and QA engineers to quickly start using our API clients in a variety of situations.

e.g. Mockbin CLI: an Bootstrap Shell-based CLI application for Mockbin powered by Spring Social Mockbin CLI Example

Again, check out the Bootstrap Shell README for more information.

HAR Mar Interceptor

HAR Mar Interceptor allows capturing HTTP request/response exchanges, persisting them in a HTTP Archive (HAR) or API Log Format (ALF)-compatible format, and re-executing those requests at a later time.

Often one of the challenging things about creating clients for proprietary APIs is the lack of documentation or samples available to learn from. Also, some API providers can be very strict with rate limits and test API endpoints are rarely provided.

In Sports, this issue is complicated further by the time sensitive nature of the domain - events rarely last longer than a couple of hours and every so often rare scenarios like weather-related postponements, data-entry errors and 22 inning baseball games create challenges for how well our products work for users.

Early on in SportsLabs’ life, we created our own system for storing the request history of particular API endpoints, but I discovered the HTTP Archive (HAR) spec, created by @JanOdvarko, lead of the Firebug project.

If you are familar with Firebug you’ll know you can see the requests the browser is making for a particular page and the time taken for each response to be received:

Firebug

Mashape API Log Format (ALF) is based on HAR and is particularly designed for API consumer use cases.

HAR Mar Interceptor ships with a customized Spring ClientHttpRequestInterceptor for capturing HTTP request/response exchanges and storing them in ALF to a JDBC database:

HAR Table

The persisted data can be loaded into an AlfHar POJO and represented in JSON using Jackson. This JSON can even be stored in a .har file and loaded into HAR-compatible tools like Charles Web Debugging Proxy.

It also provides ReplayAlfHarTemplate to replay entries in the HAR log at either real-time, fixed or immediate intervals.

This allows us to capture what our systems are requesting and/or responding under certain conditions, replay system inputs and/or outputs for debugging, plus design and tweak scenarios based on real data.

It also aids us when integrating with new data providers lacking documentation, as we can learn from the archive log and, for those who wish to limit testing efforts against their infrastructure, use services like Mashape’s Mockbin to mock their endpoints with observed data - using Spring Social Mockbin, naturally.

Comments