Amazon Order History Encryption Bypass

The Amazon US website allows you to export your Order History easily by visiting the “Order History Reports” page. No such option seems to exist for the Amazon websites for other countries. I was trying to write a simple scraper for the Amazon India Order History Page to get the same data, and discovered something interesting: Amazon encrypts the Order history page, and decrypts it using client side cryptography¹. If you were to visit the page, and check the response HTML, you’d see something like this in the source code (fairly simplified):

// Define encrypted content in JS
var payload = {
  "kid": "b70014",
  "iv": "/HenfXwYrGrrw8ff",
  "ct": "Wt78pPcibe8HAdVtoJ8+E9EGwt4IQYNghBMubBy7Zy/..."
}
// The HTML div to be populated with the decrypted HTML
var elementId = "csd-encrypted-889C1D02..";
// if client side decryption library failed to load
if (!window.SiegeClientSideDecryption) {
  window.location.href = "?disableCsd=missing-library";
  return;
}
// Decrypt and populate the div
SiegeClientSideDecryption.decryptInElementWithId(
  elementId, payload, {callSource: "now"}
);

The easiest way to scrape with such hurdles is often to just run a complete browser to scrape the site. The browser runs the javascript code with the decryption routine so you can scrape the actual content. However, it is much slower, and wastes CPU cycles - I try to avoid it if I can.

I could have spent time to parse the encryption routine, extract the key and decrypt the payload. But I found a much simpler solution - Amazon offers an alternate URL which disables encryption. As a fallback, in case the decryption code fails, it adds a query parameter ?disableCsd=missing-library. That disables the server side encryption entirely.

So if you’re trying to scrape Amazon and stumped at the missing order history in the HTML, try visiting the following URLs instead:

• https://www.amazon.in/gp/css/order-history?disableCsd=missing-library
• https://www.amazon.co.uk/gp/css/order-history?disableCsd=missing-library
• https://www.amazon.com/gp/css/order-history?disableCsd=missing-library

Amazon also sets a cookie csd-key=disabled but I didn’t experiment with that much.

Request My Data

Another alternative to scraping is to request Amazon for your data. Check the Retail.OrderHistory CSV files in the data export. The export from amazon.com includes data for other countries as well. The feature is also available on other Amazon sites:

• Amazon US - Request My Data
• Amazon India - Request My Data
• Amazon UK - Request My Data
• Amazon Germany - Request My Data

Writing on books - Fantastic Beasts and Where to Find Them

“Fantastic Beasts and Where to Find Them” is a curious book:

• The in-universe book is written by Newt Scamander and was published in 1927.
• The first edition of the companion book was published in 2001. This is apparently the 52nd in-universe edition with a foreword from Dumbledore and was released to the muggle world for charity.
• The 2001 edition pretends to be Harry’s copy of the book as of the end of Harry’s 4th year at Hogwarts. As such it includes hand-written comments from the trio. (Yes, Hermione writes on books!)

However, with the release of the film of the same name in 2016 - a new edition was released with lots of changes:

1. 6 new beasts that made an appearance in the film were added to the book¹:
   • Hidebehind
   • Hodag
   • Horned Serpent
   • Snallygaster
   • Thunderbird
   • Wampus cat
2. The hand-lettering was removed.
3. Dumbledore’s foreword is removed from the book, in favor of a in-universe foreword from Newt Scamander.
4. “About the Author” section changes Newt’s background. He no longer graduates from Hogwarts, just “leaves” it, as portrayed in the film.

All of these changes are meant to fix the inconsistencies in the book with the canon, however that also makes the book much less charming. I got myself a copy of the Hogwarts Library boxset a few years ago, which includes the newer edition of the book (Bloomsbury) - that means no witty comments from Ron.

Since it didn’t have the hand-lettering, I took it upon myself to fix that mistake. Thankfully, lists of all the comments in 2001 edition are available on the internet. The trickiest part was the “this book belongs to” page, which is missing from the newer edition. I ended up creating a faux-library card for that instead.

Here is what it looks like:

Thanks to Bhavya for helping with the troll illustration.

Analysing the Indian government cyberspace

I recently did some work on analysing the Indian government cyberspace, thought I should document them somewhere outside of my Twitter¹.

List of GoI websites

I’d made a list of Indian government websites in Jan 2019:

I ran @18F /pulse on Indian Government websites to see how many of them support HTTPS. A quick summary:

Total Websites: 14183
Total Live Websites: 11710 (82%)
Websites with Valid HTTPS: 4753 (40% of all live websites)

Raw Dataset for now: docs.google.com/spreadsheets

— Nemo (@captn3m0) January 15, 2019

The dataset was from 2 sources:

1. GoI Directory
2. crt.sh (All certificates ending in .gov.in were used)

I re-ran the scripts to get an updated list (12842 domains), then tabulated them against the public-suffix² for each. There is a long-tail, and I’ve published results here. Here are the top public suffixes for Indian government sites:

Sanskari Proxy

This was a long standing idea on my ideas repo:

A lot of Indian Government websites are inaccessible on the public internet, because they geo-fence it to within Indian Boundaries. The idea is to make a Indian Proxy service that specifically works only for the Geo-fenced Indian government websites.

For eg, if uidai.gov.in is inaccessible, hitting uidai.gov.sanskariproxy.in will get you the same result, proxied via our servers.

Since I’d made an updated list of GoI websites, this seemed easy enough. I realized that setting up uidai.gov.sanskariproxy.in would likely count as impersonation under the Indian law, so I did the next best thing: run an actual proxy. Here’s the announcement tweet:

Are you a security researcher outside India? Do you hate getting geoblocked to Indian government websites?

Well, I made a proxy for security researchers outside India to access Indian government websites without resorting to shady VPNs.
captn3m0/sanskari-proxy

— Nemo (@captn3m0) September 5, 2020

Project page is https://github.com/captn3m0/sanskari-proxy, and if you’d like to get access - please reach out.

Cyberspace Ownership

I’d planned to get a complete list of geoblocked websites next. While I’m progressing on this front, the results have been inconsistent/inaccurate so far. As an intermediate step, I’d made a list of IPs against every domain³, which looked like this:

While running numerous nmap scans (and failing), I start checking the ASN⁴ for some of these IPs to see who was hosting each website - especially the ones I was finding were blocked.

I stumbled upon a bulk IP to ASN service by Cymru, ran all the IPs against it and published the results. Here’s the important graph:

As you can expect, NIC⁵ has the highest share, with NKN⁶, BSNL, and CtrlS following at roughly 5% each. There are a few other chart on the twitter thread, and the raw data is available here with interactive versions of each visualization.

What next?

I’m working on running and comparing connectivity scans to these IPs to get a better understanding of the geoblocking situation. There’s also some issues with the domain list, as it seems to be missing lots of domains - so more corrections are needed.