The second portion of my REIbot application is to grab data about the products themselves. I had to spend a significant amount of time trying to determine how to get to the .json files that REI uses to populate their website. After a little bit of online searching, I figured out how to use Chrome's developer tools and see what urls were being fetched. In particular, REI appears to have a pattern that builds via the following lines of code (after being given a complete url for a category page):
category = url.substr(url.indexOf('/c/')+3);
urlOrigin = url;
var urlStart = "http://www.rei.com/rest/search/results?ir=category%3A"
var urlEnd = "&page="+pageNumber+"&pagesize=90";
url = urlStart + category + "&r=category%3A" + category + urlEnd;
This took a bit of digging, but it eventually built the component I needed to get to the data populating the page. The next step was to parse the .json file to find the product ids and the current price (or sale price). This part just took a couple looks at the data and revealed the following code:
for (name in $page['results']) {
if ($page['results'][name]['prodId']){
console.log($page['results'][name]['prodId']);
if ($page['results'][name]['displayPrice']['priceDisplay']['price'] == null){
console.log($page['results'][name]['displayPrice']['priceDisplay']['salePrice'])
} else {
console.log($page['results'][name]['displayPrice']['priceDisplay']['price'])
console.log(name);
}
} else {
moreResults = false;
}
}
if (moreResults) {
REIbot.runbot(urlOrigin,parseInt(pageNumber)+1);
}
One of the interesting aspects I found was that the framework loads regardless of how many products there are, so we need to check to see if there is a product before wanting to ask for another page. Unfortunately, the application would not let me ask for an excessive number of results on each page.
The next step for my REIbot is to move through all of the pages that are category pages and gather some amount of data.