With more than 245 million customers visiting 10,900 stores and with 10 active websites across the globe, Walmart is definitely a name to recognize with in the retail sector. Whether it is in-store purchases or social mentions or any other online activity, Walmart has always been one of the best retailers in the world. The Global Customer Insights analysis estimates that Walmart sees close to 300,000 social mentions every week. With 2 million associates and approximately half a million associates hired every year, Walmart’s employee numbers are more than some of the retailer’s customer numbers. It takes in approximately $36 million dollars from across 4300 US stores every day.
American multinational retail giant Walmart collects 2.5 petabytes of unstructured data from 1 million customers every hour. One petabyte is equivalent to 20 million filing cabinets; worth of text or one quadrillion bytes. The data generated by Walmart every hour is equivalent to 167 times the books in America’s Library of Congress. With tons of unstructured data being generated every hour, Walmart is improving its operational efficiency by leveraging big data analytics. Walmart has created value with big data and it is no secret how Walmart became successful.
“The most important thing about Wal-Mart is the scale of Wal-Mart. Its scale in terms of customers, its scale in terms of products and its scale in terms of technology. Walmart want to know what every product in the world is. Walmart want to know who every person in the world is. And Walmart want to have the ability to connect them together in a transaction.”
Walmart was the world’s largest retailer in 2018 in terms of revenue. Walmart makes $36 million dollars from across 4300 retail stores in US, daily and employs close to 2 million people. Walmart started making use of big data analytics much before the term. Big Data became popular in the industry. In 2012, Walmart made a move from the experiential 10 node Hadoop cluster to a 250 node Hadoop cluster. The main objective of migrating the Hadoop clusters was to combine 10 different websites into a single website so that all the unstructured data generated is collected into a new Hadoop cluster. Since then, Walmart has been speeding along big data analysis to provide best-in-class e-commerce technologies with a motive to deliver pre-eminent customer experience. The main objective of leveraging big data at Walmart is to optimize the shopping experience of customers when they are in a Walmart store, or browsing the Walmart website or browsing through mobile devices when they are in motion. Big data solutions at Walmart are developed with the intent of redesigning global websites and building innovative applications to customize shopping experience for customers whilst increasing logistics efficiency. Hadoop and NOSQL technologies are used to provide internal customers with access to real-time data collected from different sources and centralized for effective use.
Walmart acquired a small startup Inkiru based in Palo Alto, California to boost its big data capabilities. Inkiru Inc. helps in targeted marketing, merchandising and fraud prevention. Inkiru's predictive technology platform pulls data from diverse sources and helps Walmart improve personalization through data analytics. The predictive analytics platform of Inkiru incorporates machine learning technologies to automatically enhance the accuracy of algorithms and can integrate with diverse external and internal data sources.
How Walmart uses Big Data?
Walmart has a broad big data ecosystem. The big data ecosystem at Walmart processes multiple Terabytes of new data and petabytes of historical data every day. The analysis covers millions of products and 100’s of millions customers from different sources. The analytics systems at Walmart analyze close to 100 million keywords on daily basis to optimize the bidding of each keyword. Big data solutions at Walmart are developed with the intent of redesigning global websites.
Walmart has transformed decision making in the business world resulting in repeated sales. Walmart observed a significant 10% to 15% increase in online sales for $1 billion in incremental revenue. Big data analysts were able to identify the value of the changes Walmart made by analyzing the sales before and after big data analytics were leveraged to change the retail giant’s e-commerce strategy.
First Applications to Ride the Hadoop Data at Walmart
Savings Catcher –An application that alerts the customers whenever its neighboring competitor reduces the cost of an item the customer already bought. This application then sends a gift voucher to the customer to compensate the price difference.
eReceipts application provides customers with the electronic copies of their purchases.
A mapping application at Walmart uses Hadoop to maintain the most recent maps of 1000’s of Walmart stores across the globe. These maps specify the exact location where a small bar of soap resides in the widespread Walmart store.
Mupd8- Map Update Application
To fulfil the need for a general purpose real time stream processing platform which can tackle issues like performance and scalability, Walmart developed Mupd8 for Fast Data. With Mupd8, stream processing applications could emphasize on the quality of generated data. Mupd8 does for fast data, what Hadoop map reduce computational model does for big data.
Mupd8 allows developers to write applications easily and process them using the Map Update framework (a workflow of Map and Update operators), an easy way to express streaming computation. Writing an application as a combination of customized map and update operators, big data developers can focus on the business logic of the application and let Mupd8 handle load and data distribution across various CPU cores.
For example, an application can be written to subscribe to the Twitter firehose of every tweet written; such an application can analyze the tweets to determine Twitter's most influential users, or identify suddenly prominent events as they occur. Alternatively, an application can be written to subscribe to a log of all user activity on a Web site; such an application can detect service problems users’ face as they occur, or compute suggestions for users' next steps based on up-to-the-moment activity.
How Walmart is tracking its customers?
Walmart uses data mining to discover patterns in point of sales data. Data mining helps Walmart find patterns that can be used to provide product recommendations to users based on which products were bought together or which products were bought before the purchase of a particular product. Effective data mining at Walmart has increased its conversion rate of customers. A familiar example of effective data mining through association rule learning technique at Walmart is – finding that Strawberry pop-tarts sales increased by 7 times before a Hurricane. After Walmart identified this association between Hurricane and Strawberry pop-tarts through data mining, it places all the Strawberry pop-tarts at the checkouts before a hurricane. Another noted example is during Halloween, sales analysts at Walmart could look at the data in real-time and found that thought a specific cookie was popular across all Walmart stores, there were 2 stores where it was not selling at all. The situation was immediately investigated and it was found that simple stocking oversight caused the cookies not being put on the shelves for sales. This issue was rectified immediately which prevented further loss of sales.
Walmart tracks and targets every consumer individually. Walmart has exhaustive customer data of close to 145 million Americans of which 60% of the data is of U.S adults. Walmart gathers information on what customer’s buy, where they live and what are the products they like through in-store Wi-Fi. The big data team at Walmart Labs analyses every clickable action on Walmart.com-what consumers buy in-store and online, what is trending on Twitter, local events such as San Francisco giants winning the World Series, how local weather deviations affect the buying patterns, etc. All the events are captured and analyzed intelligently by big data algorithms to discern meaningful big data insights for the millions of customers to enjoy a personalized shopping experience.
How Walmart is making a real difference to increase sales?
Launching New Products
Walmart is leveraging social media data to find about the trending products so that they can be introduced to the Walmart stores across the world. For instance, Walmart analysed social media data to find out the users were frantic about “Cake Pops” .Walmart responded to this data analysis quickly and Cake Pops hit the Walmart stores.
Better Predictive Analytics
Walmart has recently modified its shipping policy for products based on big data analysis. Walmart leveraged predictive analytics and increased the minimum amount for an online order to be eligible for free shipping. According to the new shipping policy at Walmart, the minimum amount for free shipping is increased from $45 to $50 with addition of several new products to enhance the customer shopping experience.
· Customized Recommendations
Just the manner in which Google tracks tailor made advertisements, Walmart's big data algorithms analyze credit card purchases to provide specialized recommendation to its customers based on their purchase history.
Big Data Analytics Solutions at Walmart
Social Media Big Data Solutions: -
Social Media Data is unstructured, informal and generally ungrammatical. Analyzing and mining petabytes of social media data to find out what is important and then map it to meaning products at Walmart is an arduous task.
Social Media Data driven decisions and technologies are more of a norm than an exception at Walmart. A big part of Walmart’s data driven decision are based on social media data- Facebook comments, Pinterest pins, Twitter Tweets, LinkedIn shares and so on. Walmart Labs is leveraging social medial analytics to generate retail related big data insights.
Walmart launched a social media crowdsourcing contest that helped entrepreneurs get their products on the shelf. The contest attracted more than 5000 entries and more than 1 million votes across US. Anybody could pitch in their products and get exposure to millions of audience. The best products were declared as winners and sold at Walmart stores to be made available to millions of customers.
Social Genome: -
Social Genome is a big data analytics solution developed by Walmart Labs that analyses millions and billions of Facebook messages, tweets, YouTube videos, blog postings and more. Through the Social Genome analytics solution, Walmart is reaching customer or friends customers who tweet or mention something about the products of Walmart to inform them about the product and provide them special discount.
The Social Genome product combines public data from the web, social media data and proprietary data like contact information, email address and customer purchasing data. This data helps Walmart better analyze the context of their users.
For example, if the Social Genome identifies that a lady frequently tweets about movies, then when she tweets something like “I love Salt”, the social genome solution of Walmart is able to understand that the lady is referring to the popular Hollywood movie Salt and not the condiment salt.
Shopycat-Gift Recommendation Engine at Walmart: -
If you are confused on finding the perfect gift for your friends then Walmart’s Shopycat app will help you buy the ideal gift for your friend during the holiday buying rush. Walmart’s Shopycat recommends gifts for friends based on the social data extracted from their Facebook profiles. The app also provides links to the Walmart products so that users can easily purchase the product without any hassle and strive towards creating a broader marketplace. Shopycat is a part of Walmart’s Facebook page that has close to 10 million fans.
The app also suggests friends for whom users must by gifts depending on the level of interaction with them. When people click on a suggested gift, Shopycat also tells why a particular gift was suggested. For instance, the suggestions can show that a friend has liked the product on Facebook or has commented on a wall post or has a status update related to the product.
Shopycat allows the users to message their friends mutually through Facebook and ask them if they would like to buy a gift voucher or a product.
Inventory Management at Walmart using Predictive Analytics: -
Predictive analytics is at the heart of supply chain process that helps Walmart reduce overstock and stay properly stocked on the most in-demand products. Suppliers to Walmart are required to use the real-time vendor inventory management system that helps them minimize the inventory for a particular product if there are no significant sales for it. This helps retailers to save funds to buy products that have greater demand and have increased probability for greater profits.
Improving the Store Checkout Process for Customers: -
Big data analytics is begin leveraged to determine the best form of checkout for a particular customer - facilitated checkout or self-checkout. It is using predictive analytics to predict the demand at specific hours and determine how many associate would be needed at specific counters.
Mobile Big Data Analytics Solutions: -
According to Deloitte, the mobile influenced offline sales are anticipated to reach $700 billion by end of 2016. Walmart is harnessing the power of big data to drive tools and services in order to get its mobile strategy in order.
More than half of the Walmart’s customers use Smartphones and among these 35% of the shoppers are adults which is close to 3/4th of its overall customer base. Mobile phone customers are extremely important to Walmart as smartphone shoppers make 4 more trips and spend 77% more in-store. Thus, mobile users account for 1/3rd of the Walmart traffic every year and approximately 40% during holidays.
Walmart is leveraging big data analysis to develop predictive capabilities on their mobile app. The mobile app generates a shopping list by analyzing the data of what the customers and other purchase every week. Walmart’s mobile application consists of a shopping list that can tell customers the position of their wants and helps them by providing discounts to similar products on Walmart.com.
Another way in which Walmart is harnessing the power of big data analysis is by leveraging analytics in real-time- when a customer actually enters the Walmart store. The geofencing feature of Walmart’s mobile app senses whenever a user enters the Walmart store in US. The app asks the user to enter into the “Store Mode”. The store mode of the mobile app helps users to scan QE codes for special discounts and offers on products they would like to buy.
Walmart’ Carts – Engaging Consumers in the Produce Department
With the intent of reduce waste and increasing consumer engagement, Walmart is introducing quality carts in produce departments across its stores. Walmart has employed quality carts in across 500 stores now and are expected to be present in all 5000 US stores by end of third quarter. Walmart knows that keeping its customers in the fresh produce department is the key to customer engagement and the implementation of quality carts has attractive offerings for them. Walmart is using big data and IoT sensors to find out how long people loiter in the fresh produce department. Big data analysis has helped them find that if the fresh produce looks fresh enough then people loiter for longer and this is the secret to make customers buy more things from the Walmart stores.
Walmart repurposed 200 of its existing outlets to provide grocery pickup in 30 cities. After knowing that consumers were increasingly concerned about the freshness of food, Walmart trained personnel to evaluate the quality of produce and showed food items to the customers before packing them. If the wrap of frozen chicken is ripped or if the mango is not ripe, an exchange can be made immediately. All that the customers need to do is tap in their order through the app. Big data analytics helped Walmart win a bright spot in terms of grocery pickup.
World's Biggest Private Cloud at Walmart- Data Cafe
Walmart is in the process of creating the world’s biggest private cloud for processing 2.5 PB of data every hour. Walmart has created its own analytics hub known as Data Café in Bentonville, Arkansas headquarters. At the data café, more than 200 streams of external and internal data along with 40 PB of transactional data can be manipulated, modelled and visualized. The data cafe pulls information from 200 varied sources that include Telecom data, social media data, economic data, meteorological data, Nielsen data, gas prices and local events databases that accounts for 200 billion rows of transactional data for just few weeks. The solution to any particular problem can be found through these varied datasets and Walmart's analytic algorithms are designed to scan through the data in microseconds to come up with a real-time solution for a particular problem.
How Walmart is fighting the battle against big data skills crisis?
Walmart Big Data is increasing exponentially at a rapid pace every day and the dearth of big data talent is a major roadblock for Walmart in performing analytics. With limited number of personnel possessing required big data skills –Walmart is taking every necessary step to overcome this challenge is that it does not have to fall behind its competitors. Whenever a new team member jobs the analytics team at Walmart Labs, he/she has to take part in the analytics rotation program. During this program the candidates are required to spend some time with the different departments in the company to understand how big data analytics is being leveraged across the company.
Walmart is having a tough time finding professionals with experience in cutting edge analytics applications and working knowledge of data science programming languages like Python and R to build machine learning models. Walmart used the hashtag #lovedata for its recruitment campaign to raise its profile amongst the growing data science community in Bentonville and Arkansas.
Mandar Thakur, senior recruiter for Walmart’s Technology division said – “The staffing supply and demand gap is always there, especially when it comes to emerging technology”. With more than 40 petabytes of data available for analysis daily at Walmart, he says that there is going to be an unprecedented demand always for people who can do data science and analytics.
The secret to successful retailing of Walmart lies in delivering the right product at the right place and at the right time. Walmart continues to climb the retailing success ladder with remarkable results by leveraging big data analysis.
Walmart is fighting the big data skills gap by crowd sourcing analytics talent. Walmart hosted a Kaggle competition in 2014 where professionals where provided with historical sales dataset from sample of stores together with related sales events, price rollbacks and clearance sales. Candidates has to develop models that showed the impact of these events on the sales across various departments. The result of the competition helped Walmart find highly skilled and competent analytics talent.
In 2015, Walmart crowd sourced analytic talent with another Kaggle competition where candidates were required to predict the impact of weather on sales of different products in the store. Walmart has been able to hire skilled talent through these competition which they would not consider even interviewing based on the resume alone.
2014 Kaggle Competition Walmart Recruiting – Predicting Store Sales using Historical Data
The biggest challenge for retailers like Walmart is to make predictions with limited historical data. If Thanksgiving or New Year comes once a year, retailers like Walmart have to make strategic decisions about how the sales will impact the bottom-line during the festive season. Walmart hosted a recruiting competition where job seekers were provided with historical sales data of 45 Walmart stores from different regions. Each store has multiple departments and the candidates participating in the crowdsourcing competition were required to predict the sales for each department in the store. Walmart also has promotional markdown events for prominent holidays like Christmas, Super Bowl, Labor Day, New Year, Thanksgiving, etc. Holiday markdown events were also included in the dataset provided by Walmart to add up to the challenge as the sales for holiday seasons were evaluated 5 times higher than the sales for non- holiday weeks.
The most challenging part of the competition was to predict which departments were largely affected by the holiday markdown events and what was the level of impact they had on the sales.
Description of Walmart Dataset for Predicting Store Sales
stores.csv – This file contains data about all the 45 stores indicating the type and size of each Walmart store.
train.csv-This file has historical training dataset from 2010 to 2012 containing the below information-
The Store Number
The Department Number
The Week
Weekly Sales of a particular department in a particular store.
IsHoliday to indicate if it is a holiday week or not.
Features.csv- This file contains additional information about each store, the department, and regional activity for the mentioned dates with details like the store number, the average temperature in the region, the cost of fuel in that region, the unemployment rate, the consumer pricing index, whether the give date/week is a special holiday week or not, data related to promotional markdowns that Walmart is running.
Test.csv- It is just similar to train.csv except that the weekly sales are withheld in this file and the sales predictions have to be made for every triplet of the store, department and the date.
What kind of big data and Hadoop projects you can work with using Walmart Dataset?
Use market basket analysis to classify shopping trips
To serve its customers better, Walmart enhances customer experiences by segmenting their store visits based on different trip types. Regardless of whether a customer is- on a last minute run looking for new puppy supplies or is just taking a leisurely troll down the store shopping for weekly grocery.
Classifying different trip types helps Walmart enhance customer shopping experience. Initially, Walmart’s trip types are created by combing art i.e. existing customer insights and science i.e. purchase history data. A new challenge that can be solved using the Walmart dataset is to classify customer trips to the Walmart store using only transactional dataset of the products purchased so that the segmentation process can be refined.
***