Introduction

1.1 Introduction

Theory

2.1 Definition of Customer Satisfaction

2.2 Importance

2.3 How to measure?

Customer Effort Score (CES)

Customer Effort Score is very similar to customer satisfaction score, but instead of asking how satisfied customers are, they are required to rate the ease of interacting with a business. In other words, CES measures the effort exerted by a customer to attain goods or services. If customers have to do plenty of work to purchase from a company, they will be likely to shop elsewhere. The idea of a survey is to help find out if customers have a hard time interacting with a brand, and take the necessary actions to streamline processes with the help of the survey data.

Net Promoter Score (NPS)

Net Promoter Score asks “How likely are you to recommend your business to a friend or family?” — a question called “likelihood to recommend” or LTR. It attempts to measure not only customer satisfaction but also customer loyalty. NPS is used by many companies to measure customer experience and predict business growth. It serves as an alternative to traditional customer satisfaction research and is believed to correlate with revenue growth [8]. NPS has been widely adopted with more than two-thirds of Fortune 1000 companies using the metric. [9]

The company can also easily segment responses into three categories: detractors, passives, and promoters.

Promoters (9–10) are loyal customers who will keep buying from you and refer to others, helping your business grow.

Passives (7–8) are mostly satisfied but unenthusiastic and are potentially vulnerable to competitors.

Detractors (0–6) are unhappy customers who can actually damage your brand and growth through negative word-of-mouth.

Net Promoter Score is then calculated by subtracting the percent of “detractors” from the percent of “promoters” and is a number that ranges from -100 (worst case scenario: all responses are detractors) to +100 (best case scenario: all responses are promoters. [10]

Big companies like Apple, Amazon and Airbnb, among others, rely on this measurement tool to help them grow. There is a direct correlation between NPS, the likelihood of subscription renewal, and expansion of the business. NPS is an easy way for firms to receive feedback from customers and for customers to provide feedback. Over time, it has proved to be a valuable and credible system for customer-focused companies. NPS score and feedback are less likely to be affected by particular events. As a result, a business gets specific and meaningful feedback, with fewer outliers caused by recent positive or negative customer experience.

We have covered the most popular approaches, but who should fill out customer satisfaction surveys? Ideally, every customer that interacted with a business should fill in a questionnaire. It is also important through what channel survey is delivered, for example, email, in-product, website or app, among others. The frequency of delivery and the target audience within the customer base may vary from business to business.

The customer satisfaction metrics mentioned above are commonly used and straightforward, but they do not cover the full scope of customer satisfaction surveys. Depending on its goals, a company can also use longer surveys and more than one methodology.

2.4 Tools

There is also a range of customer satisfaction survey tools that companies can use to build their surveys, including «traditional» customer satisfaction survey methodologies. I am going to mention some of them.

Delighted

Delighted uses the Net Promoter System methodology to gather feedback from customers as the main one. It is possible to set up a CSAT or CES survey, but the company suggests to work with NPS surveys. The channels of distribution are email, website, and SMS.

Wootric

Wootric enables companies to track Net Promoter Score, Customer Satisfaction Score and Customer Effort Score metrics. Wootric runs surveys from the web and native iOS/Android mobile apps. Its greatest feature is so-called sentiment analytics, also known as Opinion Mining (a field within Natural Language Processing that builds systems that identify and extract opinions within the text) used for understanding customer comments.

Survey Monkey

Survey Monkey is probably the most well-known name in the realm of online survey tools. Survey Monkey gives access to an extensive library of a massive amount of pre-written questions, NPS, CSAT, CES surveys, and other survey templates. It includes open-ended and multiple-choice questions. You can send them via email, web and social networks. However, in my opinion, even though this service offers templates, most of them are long, not optimized or unreasoned from the user’s point of view.

Problem Statement

3.1 Problem Statement

Speaking about the NPS survey, let us imagine we received 1000 customer surveys and a score of 35. We had 50% promoters, 15% detractors and 25% passives. How can be passives turned into promoters? What can be done about the detractors? How can the score and the customer experience be improved?

Net Promoter Score is used to measure the loyalty of a company’s customer relationships. The best way to do so is to let customers express their opinion in their own words. NPS is used by many companies to measure customer experience and predict business growth. It is primarily built around one simple question - «How likely is it that you would recommend our company to a friend or colleague?» Many companies would follow-up with another question «Tell us a bit more about why you gave «9» as a score?; or «What can we do to improve?». Does one question alone give companies the insights they need? An open-ended question can be used to collect feedback on different topics without forcing the customer to explain what exactly they did not like about a product or service. A firm might, for example, get a response “I do not like your service”. What aspect of the service were they unhappy with? Due to such shortcomings, surveys cannot provide all the necessary deep customer insights that lead to action. Also, if the response rate is too low, survey results can misrepresent real feelings of the broader customer base. This could lead to companies drawing wrong conclusions about their business and features.

I tried to find some examples of what big companies that use or used NPS surveys think about the future of NPS. Data scientist Lisa Qian from Airbnb mentions «We find that higher NPS does, in general, correspond to more referrals and rebookings. But we find that controlling for other factors does not significantly improve our ability to predict if a guest will book on Airbnb again in the next year. Therefore, the business impact of increasing NPS scores may be less than what we would estimate from a naive analysis.» [11]

NPS is about learning and improving, and a number alone will not provide the information necessary for improving customer experience. One needs to go deeper to understand the reasons behind a score. One needs to have a conversation. A question that is being asked can vary depending on the customer’s previous choices. If they are unhappy with delivery, it should automatically route to relevant follow-up questions. Not only does it enable better analysis, but it also shows respondents that a company listens to their feedback.

3.2 Conclusion

In today’s highly competitive market, every touch point counts. Why should capturing feedback not be treated as a touch point with customers? Every organization seeks feedback, but standard web questionnaires do not provide good user experience, especially on mobile.

While working with the NPS platform Zenloop, I came up with an idea to improve the user experience of traditional surveys and make it possible for the user (company) to deploy conversational surveys with multiple questions. Conversational surveys allow users (companies) not only to collect customer satisfaction metrics but also ask customers how they feel and why they chose a particular score. In my thesis, I would like to build a prototype of the multiple question survey for a user. As an example, I will be providing the architecture of the survey for a bike sharing app on mobile. So found out all the ‘pain points‘ of bike sharing and built the prototype based on them. Afterwards I tested the Survey on different users and compared the results (effectiveness) of this survey with the results of a normal NPS Survey.

Is it possible to get more insights into the customer experience by using conversational surveys? Can a conversational survey become a solution to understanding customer needs? Can they become a more effective tool for measuring customer satisfaction?

Conversational design

4.1 Conversational design

Before speaking about conversational surveys, I would like to understand what conversational design and conversational interfaces mean.

Conversation is the most natural form of human interaction, but until recently, we were not able to use it to interact with computers. Over the last years, there has been a revolution in artificial intelligence. As a result, conversational interfaces reached another level, allowing people to speak with almost every device. Conversational interfaces are powerful because they do not need to be taught for users to understand them - we all know how to hold a conversation.

Conversation design is a design language based on human conversation, similar to how material design is a design language based on pen and paper. It is a combination of many different segments of the design industry, including voice user interface design, interaction design, visual design, motion design, audio design, and UX writing. [12]

Nowadays for almost every user, it’s absolutely natural to interact online with others. When making conversations with a computer, it is crucial to consider how words and format represent the business. It is important to build a conversation in a way that ensures the best experience for users. The process of conversation design must consider these and other challenges involved in the conversation flow.

I mentioned conversation design, now let us speak about conversation design for customer surveys and understand what a conversational survey is and how they can improve traditional surveys.

4.2 Conversational Surveys

As I already mentioned in the problem statement, some companies are afflicted with falling response rates that can probably mean that customers do not want to talk. Often customers are using social media such as Twitter or Facebook to contact companies, so probably the problem is not that customers do not want to talk, that they maybe suffering from using outdated traditional survey methods. Too many surveys are long and complicated.

Research from Stanford University shows that the quality of data deteriorates because respondents need too much time to fill out a survey. [13] As for traditional surveys they do not always produce the results that brands are looking for. That is why conversational surveys can become a new generation of customer experience surveys.

Conversational surveys enable a conversation with customers about how they feel and why they gave a particular score. Giving feedback through conversation makes the experience more enjoyable, and increases the chances that customers will take the time to provide the information you need. There are different ways to build a conversational survey. Let us think of a survey as a one-sided conversation. One person asks questions, the other answers them. Great and smart surveys, similar to great and smart conversations, have a natural flow and feel. We can create a conversational survey manually, providing relevant questions and answers that a user can choose. Another way is to use conversational AI (chatbot) and real-time reactions to give the impression a bot understands feedback and listens. It can encourage each customer to deliver valuable insight into the different features of a product or service. Some chatbots use natural language processing (NLP) systems, but many simpler ones, scan for keywords within the input and pull a reply with the most matching keywords. Most of the process occurs manually. Today, most chatbots are accessed via virtual assistants such as Facebook Messenger, Google Assistant, and Amazon Alexa among others. [14]

Giving feedback through conversation makes the experience more enjoyable and increases the chances that customers will take the time to provide the information. From a company’s point of view, conversational surveys provide customer insight that helps understand what features user like or dislike and more importantly, why.

In the practical chapter of this thesis, «Practice», I research the bike sharing thematic by finding out the biggest user ‘pain points’ of bike sharing services, manually build a conversational survey, conduct user tests of this survey and compare with a traditional NPS survey and compare the results.

Practice

5.1 Workshop

Before designing a customer survey for a bike-sharing company, I conducted detailed research on the topic. It included common problem areas of bike-sharing from the user perspective and the reasons for which users might not like their rides. To get started quickly, I organized brainstorming with peers who use bike-sharing daily. I raised the following question ‘What problem can you anticipate or have experienced before renting a bike; or during and after your ride’. After 30 minutes, I collected written responses from the group and combined user problems in five categories: 1) bike 2) external conditions, 3) app, 4) service, 5) branding & emotions.

The brainstorming proved to be very productive. I gained a comprehensive overview of bike- sharing and its problem areas. While interviewing people from two different cities - Berlin and Moscow - it became clear that bike sharing issues differ in each country. In Berlin, for example, one does not need to park a bike at a special station - it can be left within a parking zone area, such as Ring Bahn. In Moscow, however, a bike must be returned to a parking station. A problem emerges when Muscovites cycle to work during rush hour, and cannot find a free spot at the station to park. Berliners, on the other hand, argue that parking zones are too small. Another pain point of bike sharing that I found out is that people in Berlin are disgruntled a lot by sharing bikes, mostly Mobike, that are just laying on the streets, broken and not used anymore. There was even such an assumption that it is a marketing strategy to make the company catchy. The group also mentioned the lack of cycling lanes in Russia, which is one of the main issues.

Different countries have different problems, and this brainstorming workshop inspired me to start the design process of my own customer survey.

5.2 Low fidelity Wireframe

My work focuses on mobile surveys; thus, all wireframes are for mobile devices. First, I created wireframes for testing purposes. I compiled a list of screens to cover all possible scenarios, namely problem arias of a bike-sharing service discovered at the workshop. I began with grayscale wireframes to detail out the flows. I also made a low fidelity prototype to test the idea with users and fix potential problems at an early stage. The main purpose of these wireframes was to create multiple flows and explore different ideas. Before moving on to high fidelity visual designs and implementation, I created five prototypes to understand how the new solutions work. These prototypes were used to run the first round of user tests. I decided to run the tests by users, who cycle daily and are familiar with the NPS and other customer surveys so that I can check which flow has better performance. In some of the flows, I use a scale from 0 to 10 as NPS, in some scales from 0 to 5, as well as stars. The overall goal was to understand what works better for users.

To measure the success of these improvements I tracked:

• how well does each flow attract the attention of the user;
• how many users finish each flow or get bored and drop out;
• how much information they have provided in the survey.

At this stage of the design process, early user feedback was crucial to eliminate drawbacks and pick the best flow for my survey.

5.3 Logotype

This work not only aims to build a prototype of a bike-sharing survey and test it but also to demonstrate how bike-sharing companies or any other company can create similar surveys. I want to create a prototype of a service that specializes in building different kinds of surveys for any sort of business.

When I first thought about a name and a logo for my service I realized that I want choice options to have a form of bubbles of different sizes that one can click on. I did not want to develop a boring survey design, but a playful and an attractive one so that it catches the interest and is fun to complete. Ultimately I came up with Bubble. In preparation for a logo design, I researched the design trends of 2019. [17] While working on Bubble, I applied ‘Bold Colors’, ‘Gradients’, ‘Geometrical & Asymmetrical Shapes’ and ‘Big.Bold.Better’ fonts.

5.4 High fidelity Prototype

Based on the user tests and feedback from the first round, I moved on to a high fidelity prototype for the survey. My design choices were based on simplicity and design trends 2019. tried to create an emotional impact with the bubbles as well as used the shape of the logotype as an overlapping element. When the high-fidelity prototype was completed, I converted it into an interactive prototype using Invision software. The screen with the comment is programmed and also linked to the Invision prototype so that I get the answers of the users directly through email. This prototype was used to run the second round of user tests.

I would also like to explain my idea behind the bubbles. A reader typically scans a vertical line down the left side of the text while looking for keywords or points of interest (F-Shaped Pattern). [19] For every respondent, the position of the answer is different, and the bubbles can be bigger or smaller. The interface should present content in a way that matches how users prioritize information. Random bubble allocation makes all answers equivalent. Bubbles also become animated per click and disappear smoothly, becoming smaller before you get to the next screen.

You can take a look here 

5.5 A/B Testing

As previously mentioned, I wanted to test my survey on different users and compare their insights with the results of traditional NPS testing. Would I receive more insights from it? Would it help to understand customer needs better than with an NPS survey? For the user tests, I also designed a traditional NPS survey, based on familiarity and simplicity.

Before starting with the user tests I made a small research about the best practices for it. First of all, it was important for me to understand how many users are needed for a successful test. There are two types of metrics to measure the interface - qualitative and quantitative. The quantitative metric is used to quantify the problem by generating numerical data. Qualitative research relies on the observation and collection of non-numerical insights, such as opinions and motivations. For a quantitative study, it is recommended to test with twenty users, whereas the maximum number for a qualitative test, is five. [20] [21] [22] In my work I decided in favour of a quantitative study and tested each survey on twenty respondents. In addition, I measured satisfaction and behavioural responses.

For my tests, I needed to find respondents who use bike-sharing. The survey should appear after the user locks his bike, so I came up with the idea to spend some time and ask people that were returning their bikes in the city center of Berlin. Before raising any questions, I explained my motivation to the respondents so that they would engage. I also conducted some user tests remotely, with the help of my Russian friends, who submitted surveys after returning their shared bikes. It enabled me to collect opinions from respondents from different countries. The task was formulated as follows. ‘You are returning your bike. Please fill in a survey about your ride. Feel free to answer the questions you want. If you do not wish to answer any questions, please close the survey’. Ultimately, I ran the test on 32 users, eight of which participated in both surveys. Upon the completion of the survey, participants were also asked about their visual satisfaction with the questionnaire, whether they enjoyed its simplicity. Another question was if users would complete the survey in the real world if it appeared in the app after returning a bike; and how often it should appear before it starts to annoy them.

5.6 Test Result

In general, the results of the tests were very helpful and provided plenty of input. First, let us see the results of the traditional NPS survey.

As we can see from the graph, 35% (15+20) of the respondents are detractors, and 65% (35+30) are passives. Thus the NPS of -35 is negative. Only four people answered the follow-up questions. All of them were detractors, and the answers were: «oldfashioned bike», «poorly branded», «I don’t like it», «I use it only because it’s free for 30 minutes».

Now let us see the results of the survey that I built.

The graph demonstrates that users experienced the most issues with bicycles themselves. From this input, I can draw definite conclusions on what users disliked about the bikes, the app, service and check-in and check-out. For example, I know that all Mobike users were dissatisfied because the bike was too heavy and too small for them. Another interesting fact is that all four users who participated remotely had problems with parking slots or transactions.

The results also have more than twenty answers as the survey gives the opportunity to choose more than one answer. Participants enjoyed the simplicity of the survey; as well the idea of choosing the bubbles as answers. They found it intuitive, clean and easy to interact with. But testing the prototype with the first seven users detected some pain points:

The “Submit” button on the first screen is superfluous and requires an extra click, which can be avoided.

All seven users disliked the purple colour.

Therefore I took the button out and changed the colours of the prototype for the next participants. As a result, the subsequent feedback was positive.

All participants of this survey stated that if they were dissatisfied with something, they would go through the survey because it was quick and straightforward. But they would also appreciate it if it did not appear every single time but appear only when they return the bike every 5 to 10 rides.

It is impressive how much better the insights of a manually built survey were if compared to a traditional NPS survey. Each dissatisfied user made it clear with their answer exactly what they did not like about the ride. Two users, although they rated the service with five stars, left the following feedback. «I love the idea of discovering and exploring new cities by bike. Moreover, it’s great for the environment!». «Good bike and easy experience with the app». Nevertheless, my impression is that people who give a service five to four stars do not want to leave any feedback. Unsatisfied customers are ready to go through a short survey in order to let the company know what the problem was.

5.7 Bubble

I already mentioned that I want to build a prototype of a service that specializes in creating a different kind of surveys, that I named «Bubble».

To align the new service and key functionalities with user needs, I created a user persona. While developing it, I referred to possible consumers and tried to understand their job responsibilities and goals. 

Job

Marketing analyst at Mobike for two years.

Job responsibilities

Performs analysis with the use of sales and marketing data to report on customer metrics and KPIs. Assists with customer segmentation, and reports across multiple marketing channels. Provides analysis for Product, IT and Customer Service teams.

Goal

To provide best reports for the product team with the help of customer feedback data analysis to improve the service.

As the next step, the user persona was used to create a user flow. Creating UX flow enabled me to understand the user journey and to cover all the screens. For creating the user flow, I used an online tool called FlowMapp.

I usually start the design process with low fidelity sketches. This is the way I explore more technical aspects of the design. I sketched a draft of the service with elements and screens that were necessary for users’ goals on paper to see if the idea works. I concentrated only on the desktop version of the service.

After completing the sketches, I finalized the visual design and uploaded the prototype in Invision to show the interaction between different pages, and how different features of the service, such as adding or editing a question, will work.

I mostly concentrated on the flow, in which a user creates a new survey, adds questions and sets up the design. From the design, it is clear that the service provides industry-specific survey templates. It shows that the user can choose a bike-sharing survey, the design of which I provided and tested.

In addition, the service has a dashboard. There you can select a survey you created and shared previously; you can also see the response analytics. On the left side menu, you can find Plans and Billings, where you can choose a plan and pay for the service; Account settings, Help Center and Integrations, where you can manage integrations with different CRM systems - Salesforce, PipelineDeals, and others.

Since the test revealed that the colors needed a change, I replaced the Bubble colors. You can see the difference in the subsequent pictures.

You can take a look here

Conclusion

6.1 Conclusion

Traditional online surveys are losing relevance, but it does not mean that customer feedback is no longer relevant. By providing an example of a manually created conversational survey for a bike-sharing company, I proved that it is possible to have a more engaging and personalized survey experience, which guarantees higher completion rates and better insight into the user needs. Conversational surveys can be used to track and improve customer satisfaction by measuring customer emotion. They allow us to go beyond the metrics and gain a deeper understanding of user responses.

I concur the fact that it might not be useful for big companies, where responses can reach several thousand per day, to conduct conversational surveys. Big companies understand their customers feedback by using sentiment analysis of the comments. Nevertheless, I believe that small companies will have a better understanding of their business issues, and customer needs with the use of conversational surveys.

In the course of this work, I dived deeper into the understanding of the business perspective and worked out how to create a powerful tool to measure customer satisfaction. I learned that creating something new or different is exciting, especially from a designer’s point of view, but continuous testing is crucial for success.

6.2 Future work

This work demonstrates how to create a conversational survey manually, providing relevant questions and answers for a user to choose. Another way of providing conversational survey involves chatbots.

Chatbots should make it possible to create a virtual interviewer and provide conversational surveys that offer better user experience. Respondents will be able to answer questions on their favourite platforms, such as Facebook messenger. Combined with the power of artificial intelligence, it offers the next level of real-time understanding and routing and creates a better user experience.

This work can be continued by building a survey powered by an AI chatbot. It should be tested when finished as there is a big possibility that users will not like a conversation with a machine. The onus is on the designer to build an AI chatbot that provides human-centric experience and reacts to the conversation in a way that only a human can.

References

Thank you

I would like to thank Prof. Constanze Langer and Prof. Reto Wettach for their feedback.

Also I would like to say thank you to my parents, Shilpa and Katia for advice and all friends for support and participation.