Clickstream data is a trail of digital breadcrumbs left by users as they click their way through a website or mobile app. It is loaded with valuable customer information for businesses, and its analysis and usage has emerged as a powerful data source. This data is typically collected to analyze and improve the user experience, as well as to track mobile performance and traffic. Clickstream data can include information about the pages that a user visits, the time spent on each page, the order in which pages are visited, and the links that are clicked. It can also include information about the user's device and location, as well as any search queries or other actions that the user takes on the website. This data is often used to optimize website design and layout, to identify popular content, and to target advertising to specific users.

Motivation

The Gojek SuperApp is a crazy highway of varied use cases, which need data to be delivered in real-time to the Gojek backend servers. The main drivers of this initiative were to achieve:

Real-time data

At Gojek, we’d been using third-party analytics providers, none of which provided data to the Gojek backend in real-time or even remotely close to it. The solutions were either very expensive or highly inefficient. The real-time availability of data serves various purposes like detecting driver fraud, serving ads based on impressions, etc.

Data sanity

Gojek is a humongous app, with a zillion events. Hence, it’s important to maintain event data sanity. With a team of over 200 enthusiastic developers on iOS and Android, it’s hard to keep a check or define a Standard Operating Procedure (SOP) for adding new keys to an analytic event. So, we wished to bring in type-safe and reusable event schemas that help achieve that level of data sanity.

Reduced data pipeline costs

With the third party services in place, there is always an overhead of maintaining the data pipelines from the service providers data stores to ours. Sending the data directly to our own data stores reduces the cost of porting the data and maintaining those pipelines.

Architecture

Clickstream on a high level has two parts to it, Clickstream mobile libraries and backend service to collect aka Raccoon

In this blog, we talk about the mobile side of things and how the libraries are built.

Mobile Library Architecture

Event Processor

It ingests the analytics events messages i.e. protobuf objects generated by the client app, add relevant metadata/attributes to the provided proto and forwards the events messages to the next stage.

Event Scheduler

It categorises, prioritises and caches the Event objects passed on by the processor. It is also responsible to batch the events together to be sent to the server.

Network Manager

It is responsible for setting up the required connection to the backend services and forwards the event batches via a retry mechanism. The Network Manager is responsible to handle the socket state and the retries for the failed events.

Configurations

Configurations for each block are initialised with the library. These configurations allow for fine-grained control over the library behaviour. These configurations define various constraints for the library as well, as in, minimum operational battery percentage, duration between retries.

Contracts

Contracts have been defined for each event. These contracts are nothing but protocol buffers that help the team define efficient events and ensure the reusability of the bits. More power to “DRY”. These contracts also ensure the type safety of the data being sent.

How Clickstream is different from other SASS alternatives?

Clickstream via its public API only accepts a Message type and not the conventional dictionary or a map like the other third-party analytical tools. This design allows for enforcing strict schemas, hence making life easier for the PMs and the designers. Using protocol buffers were are able to standardise events across different product groups within Gojek.

Events’ classification (standard/real-time/instant) can be updated on the fly or an event can be dropped altogether simply by updating the remote configurations.

Clickstream collects its own health data. This allows us to gauge the library health as drop rates, event and batch health funnels via the library itself.

Use cases

The data being collected with Clickstream is empowering various use cases for different teams; to mention a few:

Fraud detection

Fraud detection and commitment is a critical part of Gojek’s commitment to drivers and users safety and ensuring safety is tantamount to Gojek’s existence as a whole. Clickstream is used to send encrypted signals which suggest that the application being used is a genuine one or not.

Merchant Ads

Gojek allows merchants to place their ads at various places within its applications. The merchants pay for the number of clicks on the Ads, impressions/views. There are 2 problems that need to be solved in real-time for the merchants & increase revenue for Gojek; Daily ad-spend capping by merchants based on totals clicks/impressions per day and reconcilation of billing based on the total clicks and charge the merchant fairly.

Whats's next?

We at Gojek want to share the tools we’ve created. So we plan to make Clickstream open-source in the near future. This could serve various organizations big and small to send and receive data in real-time with no real effort or time.

Remote Logging and App Observability

Gojek mobile applications are constantly evolving and with that, the need to keep the performance of the applications in check is a must. With Clickstream developers would be able to observe certain critical performance metrics in real-time and turn on or off A/B tests, gather early feedback, collect performance metrics etc.

Real-time alerts and user assistance

Leveraging the real-time capabilities of Clickstream, Gojek’s support team would directly be able to reach out to the affected users and guide/help them with the task they wished to perform but were not due to app performance issues.