Planning the Web Almanac 2019

This year we’re setting a new goal to create an annual report of the state of the web, called the Web Almanac. The purpose of the Almanac is to take a snapshot of how the web is built and how it has changed over the year. The end result will be a web app that takes the reader through the different building blocks of the modern web and presents our findings about how they are performing.

Current Status: As of June 3, we are wrapping up our first milestone of finalizing all chapters’ authors, reviewers, and metrics. The next milestone on our roadmap is to get the technical infrastructure ready for the Almanac content to drop and to get all of the metrics ready to query once the 2019_07_01 dataset becomes available. All of the Author roles have been filled, but we’re still actively looking for Reviewers, Analysts, Developers, Designers, and Translators.


We’re looking for contributors to fill the following roles:

  • Subject Matter Expert (Author)
    • Responsible for providing an interpretation of the HTTP Archive data for specific chapters aligned with their area of expertise. For example, in the JS chapter, explain what the implications are for the number of bytes and CPU execution time to be rising. 1-2 paragraphs per metric are expected plus an intro and summary conclusion. Think of it like one blog post in a series written by many authors.
    • During the planning phase, experts should ensure that their chapters contain the necessary metrics to accurately capture their state. Also help us identify if there are any metrics that we are currently unable to capture so we can work on getting them in place for next year’s edition.
    • Time commitment: 2 hours in April to coordinate on chapters’ metrics, 4 hours in August to write interpretations.
  • Peer Reviewer
    • Responsible for providing feedback on the technical correctness of experts’ written interpretations and ensuring a consistent voice is followed throughout the report.
    • Assist expert with brainstorming metrics during the planning phase.
    • Time commitment: 1 hour in April, 2 hours in August.
  • Data Analyst
    • Responsible for writing queries to extract each metric used in the report. Results will be passed to experts for analysis.
    • Time commitment: 4 hours in May.
  • Designer
    • Responsible for creating the responsive UX of the report including static landing pages and data visualization. We’re looking for a design that complements the primary website but is not necessarily a clone of it.
    • Time commitment: 20+ hours throughout July and August.
  • Developer
    • Responsible for the full stack including bringing the design to life and providing the infrastructure to serve content on App Engine.
    • Developers are expected to include a11y and SEO best practices and ensure the report is extensible for i18n and future annual editions.
    • Time commitment: 5-20 hours throughout May, June, August, and September (depending on how much work you want to take on).
  • Translator
    • Responsible for translating the finished Almanac content from English into international languages like Japanese, Spanish, Russian, French, Portuguese, German, Dutch, Italian, and Polish (prioritized in that order).
    • Time commitment: 3-5 hours in September and October.

If you’re interested in contributing, please join us in the #web-almanac channel on the HTTP Archive Slack. We’re also tracking project progress on GitHub so feel free to jump into any of the open issues.

7 Likes

This is an awesome idea and I’ll contribute where I can. Possibly some writing and/or gather of interesting pieces published on the web that may add to the EOY story telling.

Something like this, so we can track what has changed from Feb 1st onwards - among many other things.

Really looking fwd to this.

Great initiative Rick!

H!

2 Likes

Awesome idea.

I can help with web development, writing and tracking the features, changes over time.

1 Like

Hi Rick,

This is a fantastic idea! I would absolutely love to help out with the writing side of things especially. Let me know how I can help.

I agree about looking at the current HTTP archive reports as well as some mobile/desktop comparisons. Maybe you could do some analysis on different devices and their CPUs too, and do some load time benchmarking and comparisons? I’m always interested to see how websites are performing in the real world for users on their actual devices.

I’ll be happy to provide some more ideas as well when more of the topic planning is under way.

1 Like

Hello Rick !

Great idea. I could help on data analysis and writing.

Some ideas :

  • Distinctions between different locations in the world (bandwith, latency, performance and devices)
  • If we could track performance of cms vOv (is WP getting faster on latests versions ?..)
  • Would be interesting to have some perf insights according to business field (how e-commerce websites are performing, medias ? etc)
  • Try to zoom on the long tail (the slowest, the poorest performance %, excluded users…Etc)

Looking forward !
Aymen

1 Like

@matijagrcic @AymenLoukil are you on the HTTP Archive Slack? If not, please join. I’ve created a channel for us to discuss this project in a more high bandwidth way.

It’s also an open channel, so if there’s anyone else wants to help out, feel free to join.

Brainstorming Almanac content in this public doc. Contributors welcome!

Awesome. Looking forward to this :smiley:

This is very cool. I can help with writing, and cross-promotion in web.dev.

1 Like

Hey Meggin, I think there’s definitely a good opportunity to connect this back to web.dev. Did you have anything in mind? For example, maybe we can write a summary of the interesting insights and talk more broadly about what we should be doing as a community to improve the state of the web. Also, your writing expertise would be invaluable! If you’re on Slack, please join us in the #web-almanac channel.

Hi @rviscomi what kind of web development do you guys need? I can also help in writing

1 Like

Hey @tjmonsi! Thanks for helping. The web dev work we’ll need will be to extend our existing website (source code):

  • writing new Flask templates to render different types of Almanac pages
  • working with a designer to implement the UI in HTML and CSS
  • visualizing the data using the Highcharts JS API

Forking it now and will try to submit a PR on one of the good first issues. Thanks :slight_smile:

1 Like

Awesome stuff. Looking forward to helping with the Media chapter.

1 Like

Join the Data Analyst team!

Over the upcoming days I’ll be posting a new thread in the forum for each of the 20 chapters in the Almanac. The goal is to map the list of metrics curated by authors and reviewers to corresponding queries.

Contributors who have signed up for the Data Analyst team will be resolving the mapping of metrics to queries in the forum. It’s a great opportunity to see how to extract insights from the HTTP Archive dataset, ask questions, and get your hands dirty if you want to help out with the research project. Everyone who helps will be credited in the published Almanac and receive a special BigQuery credit to help offset the cost of querying.

Authors and reviewers from those 20 chapters have cumulatively compiled a list of about 250 metrics about the state of the web. As of now the Data Analyst team is just 6 members strong, which means each person must map ~40 metrics. We could really use the help of the HTTP Archive community to break this challenge down into more manageable chunks by adding more people to the team. Our goal is to reach 25 members, with each person responsible for a much more doable ~10 metrics.

If you’re new to BigQuery or even a little rusty, mentors are available to get you started. @paulcalvano has written an excellent Getting Started with BigQuery guide and a Guided Tour of schema explorations and example queries for select tables. This forum also contains a treasure trove of queries over the years with great analyses, so browsing old threads is a great way to learn the dataset.

If you’re interested, please request to join at https://github.com/HTTPArchive/almanac.httparchive.org/issues/23 and feel free to hop into any of the threads here on the forum as they appear.