Assessing the Need for SB Pay Onboarding Using SQL and GSheets
Lana Hung5 min read·Just now--
Hi everyone! Today I want to share a research project I worked on at ShopBack in Q4 2025.
Previously, I wrote about using PostgreSQL in Metabase to pull data and then applying Xmrit charts to evaluate issue severity. This time, I took a different approach — using Google Sheets formulas to run statistical testing instead. I’ll explain later why I didn’t use Xmrit charts for this case.
Also, at the time, I relied quite a bit on ChatGPT. However, the statistical testing results it generated weren’t as accurate as I expected. I also wanted to strengthen my fundamentals in selecting and running statistical tests myself, so I decided to go through the process manually. That said, I now feel much more confident with statistical testing — and with my company now subscribing to Claude Enterprise, building a Claude skill to automate this is officially on my to-do list.
But for today, I’ll focus on the research itself and the thinking behind my approach. Stick around to see what I learned along the way!
Context
ShopBack Pay is one of our key product lines in Singapore, Australia, Hong Kong, and Malaysia. Here’s how the business model works: for example, when a user pays at a physical Starbucks store via ShopBack, they receive a tracked Cashback. This Cashback is only confirmed after they return to the same merchant within a certain period and make another purchase. At the same time, they earn a new tracked Cashback for the next cycle. This creates a win-win-win situation: merchants benefit from retention, users get Cashback, and ShopBack earns commission.
However, while the flow seems straightforward to those familiar with it, it can be confusing for first-time users. In one conversation, we heard a new ShopBacker and a first-time Pay user say, while looking at the interface, “I thought I needed to return to the exact store on the exact date.” Although we found it amusing, it’s slightly concerning. It raised an important question: Is this confusion more widespread than we think? And if so, do we need to introduce onboarding for new users?
That’s where I came in — to assess how serious this issue actually is.
Goal
- Understand what percentage of users are confused about how ShopBack Pay works, and how severe this issue is
- Determine whether there’s a need to introduce onboarding flows for new ShopBack Pay users
Collecting raw data
Since we already had a recurring in-app survey triggered after a ShopBack Pay transaction (i.e., when users receive a tracked Cashback), I used this existing data instead of launching a new survey.
In addition to a CSAT question, the survey includes a follow-up for detractors (users who rated 1–3), asking which aspect of ShopBack Pay they struggled with, and the first option is directly related to onboarding.
So I went directly into Metabase and used SQL to query the last three months (Aug 1 — Nov 1, 2025) of survey data.
Question: What made it difficult to earn Cashback from ShopBack Pay? Select all that apply.
Answer options:
1. I didn't know I had to buy again at the same store to secure the Cashback.
2. Having to return and buy in-store is inconvenient for me.
3. This is not a store I shop frequently at.
4. The offer is only available for a limited time.
5. Other (please specify)Statistical testing using Google Sheets
My first goal was to determine: (1) whether option 1 was selected at a statistically significant level, (2) whether its selection rate was meaningfully higher than other options.
I exported the raw survey data from Metabase into Google Sheets and applied a z-test for proportions, which is suitable for percentage-based comparisons.
I found out that from around 130 users, although ~45% of respondents selected option 1, its proportion is not significantly selected (p-values: AU 0.90, MY 0.84, HK 0.88 at 95% CI). It is also not selected significantly more often than other options (p-values: AU 0.67, MY 0.13, HK 0.56 at 95% CI).
Based on this, the issue is neither affecting a majority of users nor statistically significant in severity.
Here, I didn’t use Xmrit charts to evaluate the severity since in this case I needed to compare: % selecting option 1 vs. a fixed benchmark (50%), and % selecting option 1 vs. the next highest option. Whereas Xmrit charts are better suited for monitoring a single metric over time.
Joining survey and transaction data
In Metabase, I also joined the survey responses (specifically users who selected option 1) with Pay transaction data using user IDs.
The goal was to understand: Are these users simply expressing confusion but still successfully earning Cashback? Or are they genuinely confused and failing to complete the flow?
The results were quite interesting: Around 130 users selected option 1, yet all of them eventually received confirmed Cashback, and about 50% had already received it before answering the survey. This suggests that while users may express confusion, they are still able to navigate the system successfully.
Looking across all three data points, the issue does not appear to be severe at this stage. And as a result, I recommended: No immediate need for additional onboarding or education for ShopBack Pay in AU, MY, and HK. (This recommendation got accepted by the product team. 🎉)
Takeaways
Although the scope of this research was relatively small, I believe studies like this are especially valuable, as they help teams prioritize and allocate resources to higher-impact areas.
As a researcher, this project also sharpened my data analysis skills. For example, analyzing data by country instead of aggregating, since each market behaves differently, and choosing the right statistical test for different scenarios (e.g., one-sample vs. two-sample z-tests).
These learnings made my recommendations more convincing and also boosted my confidence as a researcher. More importantly, it helped me build a solid foundation in quantitative analysis before relying heavily on AI tools!
If you find this article helpful, let me know by clapping👏 🥰 And if you’d like to discuss any topics, hit me up through my email [email protected]. Thank you and..Peace out!
