Gems: General Marketplace Simulator
By
Community /
Product, Company
Aug 31, 2022
Navigate to:
This article was written by Rushil Shah, Software Engineering Intern (Summer 2022) at InfluxData.
Current system:
InfluxData lists its SaaS products on AWS Marketplace. This means that AWS handles all the product reports and setting up payments to customers. The job of the organization (InfluxData in this case) is to register and publish their product on AWS Marketplace.
Customers can then visit AWS, see InfluxData product options listed on their AWS account page, subscribe to the product, which redirects them to InfluxData’s official product registration page.
This is a straightforward process and AWS has good documentation for it. However, testing this onboarding process is cumbersome in the original application (Quartz) because it depends on the AWS marketplace and its APIs. This means that testing user registration consumes a lot of third-party resources.
In order to remove this dependency and ensure a similar flow while testing User Onboarding in Quartz, the team decided to come up with a simulator that behaves exactly like AWS but runs locally and ensures that Quartz gets the same interaction it would with the AWS Marketplace.
Under the hood of AWS Marketplace registration
Although the Marketplace service is too sophisticated to put in detail here, there are some core features needed when developing a simulator.
REST API:
AWS has its own identifiers to uniquely identify each user. During the interaction with Quartz, it uses REST POST requests to exchange these identifiers and tokens for security and to ensure data integrity while redirecting from the AWS Page to the desired application’s registration page. AWS also sends these identifiers for future use for metering records and billing. More information about specific APIs is available here.
AWS Simple Notification Service:
When the user subscribes to a product, AWS sends some notifications to the application, confirming the subscription on the user side. The application, after receiving these notifications, initiates the billing process. Notifications are also sent when a user unsubscribes from a product, letting the application know that the billing cycle needs to stop. AWS documents the details of these notifications using their SNS service, here.
AWS Simple Queue Service:
The way SNS works is that it sends a one-time non-persistent notification to a receiver. SNS doesn’t retain a record of sent notifications. In order to make sure that we don’t end up missing notifications, we use AWS Queue to store those notifications and poll the queue in regular intervals to fetch and process all the notifications. More about integrating AWS Marketplace to a SaaS product is available here.
Gems
Gems acts just like AWS Marketplace and simulates the services described above. It uses several third-party dependencies to mimic AWS functionality. Gems has its local PostgreSQL database which stores user details like identifiers and AWS subscription statuses. On top of it, all the notifications sent to local queue are persisted in the database for future analysis and testing.
InfluxData has multiple products listed in AW Marketplace and users can subscribe to any of them. Gems, currently, supports a single global product with a static product code, and all the Gems users subscribe to it.
Intro to ElasticMQ
Gems simulates AWS’ API, SNS, and SQS services.
To simulate the behavior of AWS SNS and AWS SQS and send notifications regarding user-subscription to Quartz, we use ElasticMQ, which is an in-memory message queue system. ElasticMQ can run both as a standalone application or via Docker. Its sole purpose in Gems is to mimic SQS.
Gems sends notifications similar to the ones AWS sends, along with required key-value pairs, except the Gems messages go to a local queue. To receive these messages, it is necessary to configure Quartz to listen to that particular local queue while talking to Gems.
Playing with Gems
The Gems UI is very simple and straightforward. Click on the New User button. This will do the following:
- Generate a fake user with all the required identifiers
- Subscribes it to the product. Updates the database, adds required data
- Send subscribe-success notification to the local queue
- Redirects to Quartz registration page
Once created and subscribed, each user can be either deleted or Unsubscribed.
Clicking the Unsubscribe button initiates Gems to send required notifications to the local queue and changes the database statuses accordingly.
Quartz polls the local queue, reads the messages, and updates user and subscription data accordingly.
This is how, at any instance, the state of local queue of ElasticMQ will look, with messages still in the queue.
iex(5)> ExAws.SQS.receive_message(url) |> ExAws.request
{:ok,
%{
body: %{
messages: [
%{
attributes: [],
body: "{\"Message\":\"{\\\"action\\\":\\\"subscribe-success\\\",\\\"customer-identifier\\\":\\\"2e80e7d6-f289-49c4-a123-d054c6d6f263\\\",\\\"product-code\\\":\\\"24rvlw4j78h53941sycw25cwe\\\"}\"}",
md5_of_body: "b620f4bbfe060195a38418c3279f6fe2",
message_attributes: [],
message_id: "0a17fe27-98b7-45fb-a7d9-4512e1416b6b",
receipt_handle: "0a17fe27-98b7-45fb-a7d9-4512e1416b6b#fcba1214-260e-477a-869d-269d4c33bc63"
}
],
request_id: "00000000-0000-0000-0000-000000000000"
},
headers: [
{"Server", "akka-http/10.2.9"},
{"Date", "Fri, 19 Aug 2022 21:01:44 GMT"},
{"Content-Type", "text/xml; charset=UTF-8"},
{"Content-Length", "910"}
],
status_code: 200
}}Future scope
The GEMS application can be extended to include other services like metering and billing to test them without AWS dependency. Gems works similarly with other cloud services like GCP and Azure. Although they have completely different infrastructure and workflows, having access to proper documentation and resources makes doing so possible.
Conclusion
The project covered various aspects of API design, several standard functional programming patterns, and test-driven development. Overall, the experience had a steep learning curve, and was very memorable.
