MDN's under construction PWA page defines a PWA as

"Progressive Web Apps are web apps that use emerging web browser APIs and features along with traditional progressive enhancement strategy to bring a native app-like user experience to cross-platform web applications. Progressive Web Apps are a useful design pattern, though they aren't a formalized standard."

If this definition is too vague or broad for you, you can safely assume that PWAs are web apps which run in the browser but can be "installed" on the user's smartphone or computer to provide a native app like behavior but in the browser. Generally, some offline functionality or graceful application behavior without network connectivity is expected from PWAs.

To better understand how PWAs are implemented and what better experience this web application design pattern provides, let's try to dig into and reverse engineer the Twitter PWA using the Chrome Dev Tools.

The Manifest 📜

If your web app is going to be "installed" it needs to have some metadata, e.g the app's name, icons for different resolutions and a url to which the app opens to by default. All of this information is added in a json file present at the /manifest.json path.

Here is a screenshot of the Twitter's manifest.json file. You can see the detail you can provide and the support for the possible interoperability of the Twitter Android native and the PWA app using the android_package_name property. Pretty powerful and feature rich! Read more about the manifest file and what it can support here.

Service Worker 🧑‍🏭

The ❤️ of PWA - service worker is a JS script that lives separate from the browser page. Among other features, it provides a network proxy for the web page it is running for. This means every network request originating from that page passes through the service worker. This opens up possibilities and provides more control over resource caching, offline application behavior and requests for resources or API calls that are sent to the server.

The Twitter service worker is registered under the https://twitter.com/ path which means it will work as service worker for all paths under the twitter.com host. Service workers once installed against a web app keep running even when you navigate away from the page and become active every time user goes to the URL for which a service worker is registered in the browser. Read more about the lifecycle, features and nuances of service workers here.

Cache Storage 🛍️

The Cache Storage Web API provides a simple way to implement caching of resources in the browser. It is important not to confuse this API with the default browser caching implemented by default.

As you can see there is a list of js files stored in the cache storage and each one of them has a path in the name field. Remember when we said that the service worker acts as a network proxy for the page. This means every request for these js resources originating from twitter.com first passes through the service worker.

The service worker is implemented in such a way that for every resource request it first checks if the resource is present in the cache storage. If it is, the page is sent a 200 OK response along with the required resource, without ever hitting the twitter.com servers. You can see a live demo of it in the Network console. As you can see in this screenshot, for the js resource bundle.HomeTimeline...js the request never hits the servers and the twitter.com page still gets the 200 OK response along with the script.

Similarly, the HTML skeleton/shell of the Twitter app is also cached.

Even if there is no internet connection, the Twitter web app's HTML and JS will be served to the page from the Cache Storage by the Service Worker, making the app resilient to slow or no internet connectivity.

IndexedDB 💾

Another one of the emerging web browser API that PWAs use is the IndexedDB API. The IndexedDB is an interface for storing and accessing structured data within the browser.

Twitter makes interesting use of this feature for optimizing user experience by storing different kinds of less frequently changing metadata for the logged in user and some recent user activity so that it can be fetched directly from within the browser when needed. A quick look under the IndexedDB category for twitter.com shows multiple IndexedDB instances, some with only a single Object store, some with multiple.

Expanding in and looking at the contents of one of the DBs with an object store called keyvaluepairs we can see that it has stored a list of my recently used emojis on Twitter.

The Object store in the DB with the identifier localforge contains some more metadata about the user.

This kind of data could have been stored using good old Local Storage but newer features like IndexedDB are meant to server as an objectstore database for your application right within the browser.

Installing the Twitter PWA ⬇️

You can use a little button in the Chrome browser URL field to install the PWA as an app.

Once installed, you now have a Twitter app icon in your dock/task bar and the app opens in a separate window different from your browser.

The Offline Test 🚫

Let's see what happens if I turn off my network and try opening the newly installed Twitter PWA.

As you can see, the app's UI gets loaded. Most of the navigations work, but no tweets or trends data is loaded. It is important to understand:

• The UI works because the app's HTML and CSS are in Cache Store and are being served through the service worker
• The interactions and navigations work because the JS implementing those is also stored in Cache Store.
• The profile page shows some of the user metadata, which means that data is stored somewhere maybe in the IndexedDB from which the app is fetching it rather than the API.
• As the installed Twitter app is actually only a separate Chrome instance which only goes to https://twitter.com, but we are not seeing the 'No Internet Connection' page with a trex because the Service Worker is acting as a network proxy and still serving content and responding to the network requests of the twitter.com page even without a connection to the Twitter servers.

Phew you reached the end! Hopefully, you learned something new. If you want to ask a question, leave your thoughts about PWAs or you think there's a mistake in the article above, I'll be glad to hear from you in the comments below. You can also reach out to me on Twitter @haseebelaahi 🙂

Facebook allows you to download your data as browser viewable HTML, but that view isn’t really interesting and doesn’t reveal much (as Facebook would probably want 🤷‍♂️). The HTML view just gives you a boring list of coordinates like this.

I downloaded the data in JSON which is another of the supported formats and did some basic stats & visualizations.

I have been a Facebook user since 2011 but started using the Facebook app on my phone 3–4 years later. All of the stats and visualizations here are from location data ranging from January 2015–April 2019.

Number of Times Facebook Stored My Location

Facebook accessed and stored my location for a total of 2818 times in the period from start of 2015 to April of 2019 averaging almost 650 times a year , 54 times a month 🙀 or twice every day 😮. When you think about these numbers for a moment, it almost feels like basically being tracked literally all the time!

Stored Locations By Year

Facebook’s location collection seems to be tied with how often your location changes and obviously having your phone’s location turned on is a prerequisite. This graph shows a trend of location entries by each year.

The reason for the huge spike in data collection from 2016 to 2017 is maybe due to some change in Facebook’s location collection policy or probably I just bought a good smartphone in 2017 and started having the location turned on all the time on my Android smartphone.

Busiest Location Collection Days

Digging deeper into the data, I went ahead and filtered out the ten days which had the highest location collection frequency. Can you guess how many times at maximum my location got stored on Facebook on any given day?

50, 100? Nope, Two Hundred and Thirty-Eight times!

Turns out I was traveling on that particular day and was on the road for the most part of my day and Facebook like any worried parent would do, checked on me for 238 times…

What would all of this data look like on a map?

Naturally I wanted to put all of this latitude and longitude data on an actual map to visualize where I have been moving around in all these years!

So this above is a visual representation of my location data that Facebook has almost in a 15km radius of my house in my hometown (Lahore, PK). Each blue dot represents one instance of a location captured and stored by Facebook.

For some of the locations marked here I didn’t even remember visiting that area but thanks to Facebook for providing hard evidence I recalled everywhere I’ve been in the past 4 years 🙃

With the location data, timings for each location capture and Facebook’s resources it is very easy to infer any Facebook app user’s home address, workplace, where do they go on weekends regularly, which restaurants they eat at and so on. All of this definitely makes targeting ads easier and of course classifying specific user personas for political and social purposes.

Considering the latest speculation about Facebook launching a dating service of their own, this data could be of a lot more use in “👫 bringing people together”!

So, next time when you meet someone and the next thing is that they start appearing in your ‘Suggested Friends’ list don’t get surprised 😆

You can find detailed analysis on my Facebook data here on Kaggle

https://www.kaggle.com/helahi/facebook-messages-analysis

Thank you for making it through! If you think I should dig in some more into this data to find any other stats or patterns, let me know in the comments or get in touch with me on Twitter @haseebelaahi!