Technology behind the big web sites

Disclaimers:

Very few of these sites has published details on their architecture and tech stack. So this is based on whatever incomplete information they have published and various internet gossip that may be more or less accurate. And it is almost certain that there are errors in this. And it is certain that this is not complete (such huge companies use lots of different technologies). But hopefully most of it is correct and cover at least the most important stuff.

The technology behind the biggest web sites is not the same as the most used technology on the web. By definition the biggest web sites has unique requireements that often will lead to unique solutions. Mucg of the software used has specifically been designed for extreme volume and may not be needed by more normal web sites.

10 years ago:

Company	Application	Frontend	Backend	Data storage
Amazon		proprietary web server (C++) / Linux Java / JBoss / Linux Perl / Linux		Oracle / Linux
EBay		Java / WebSphere / Windows and Linux (primary) Java / Windows and Linux		Oracle / Solaris
Facebook	Facebook	PHP / Apache / Linux		MySQL / Linux HBase / Linux files @ CDN
Facebook	Chat	C++	Haskell and Erlang / Linux	-
Google	Search engine	static HTML proprietary web server (C++) / Linux	C++ / Linux Python / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Advertising	-	Java	MySQL / Linux
	GMail	JavaScript / Closure Tools proprietary web server (C++) / Linux	Java / proprietary servlet engine (Java) / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Google Docs	Closure Tools proprietary web server (C++) / Linux	Java / proprietary servlet engine (Java) / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Google Calendar	Java / GWT	Java / Jetty / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Google Wawe	Java / GWT	Java / Jetty / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Youtube	Apache / Linux lighthttpd / Linux	Python / Linux	files @ CDN MySQL / Linux
LinkedIn		Java, Tomcat and Jetty / Linux memcached / Linux		Oracle / Solaris MySQL / Linux
MicroSoft	General web sites	C# / ASP.NET / IIS / Windows		SQLServer / Windows
	Hotmail	C# / ASP.NET / IIS / Windows		SQLServer / Windows
	Bing search	C# / ASP.NET / IIS / Windows C# / Windows		proprietary graph database / proprietary storage system / Windows
MySpace		C# / ASP.NET / IIS / Windows		SQLServer / Windows proprietary file system (C) / Linux
Twitter		Ruby / RoR / Mongrel / Linux	Java and Scala / Linux memcached / Linux	MySQL / Linux
Wikipedia		PHP / Apache / Linux		MySQL / Linux Lucene / Mono / Linux
Yahoo	Portal	PHP / Symfony / Apache / FreeBSD and Linux		MySQL / FreeBSD and Linux
	Mail	JavaScript / YUI	PHP / Apache / FreeBSD and Linux	proprietary storage system (C++) / proprietary file system (C) / FreeBSD and Linux
	Groups	JavaScript / YUI	PHP / Apache / FreeBSD and Linux	proprietary storage system (C++) / proprietary file system (C) / FreeBSD and Linux
	Flickr	PHP / Apache / Linux (primary) Perl / Apache / Linux Java / Linux		MySQL / Linux HBase / Linux files @ CDN

Today:

Company	Application	Frontend	Backend	Data storage
Amazon		JavaScript / Angular JavaScript / React Java / GWT	Java / Linux Java / JBoss / Linux Perl / Linux	Aurora / Linux MySQL / Linux Amazon DynamoDB
EBay		Java / WebSphere / Linux (primary) Java and Scala / Linux JavaScript / node.js / Linux C++ / Linux Kafka / Linux		Oracle / Linux HBase / Linux
Facebook	Facebook	JavaScript / React (for Web) Java and Kotlin (for Android) Swift (for iOS)	PHP and Hack / Apache and HHVM / Linux (primary) Java / Linux	MariaDB / Linux HBase / Linux
	Instagram	JavaScript / React (for Web) Kotlin (for Android) Swift (for iOS)	Python / Django / Linux RabbitMQ / Linux memcached / Linux	PostgreSQL / Linux SOLR / Linux Amazon storage
	WhatsApp	JavaScript / React (for Web) Java (for Android) Swift (for iOS)	Erlang / FreeBSD Erlang / Yaws / FreeBSD PHP / Apache / FreeBSD lighthttpd / FreeBSD RabbitMQ / FreeBSD	Mnesia / FreeBSD MySQL / FreeBSD
Google	Search engine	JavaScript / Closure Tools proprietary web server (C++) / Linux	C++ / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Advertising	-	Java	F1 / Spanner / Linux
	GMail	JavaScript / Closure Tools proprietary web server (C++) / Linux	Java / proprietary servlet engine (Java) / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Google Docs	JavaScript / Closure Tools proprietary web server (C++) / Linux	Java / proprietary servlet engine (Java) / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Google Calendar	Java / GWT	Java / Jetty / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Google Play	JavaScript / Closure Tools proprietary web server (C++) / Linux	Java / proprietary servlet engine (Java) / Linux	proprietary storage system (C++) / proprietary file system (C) / Linux
	Youtube	Apache / Linux lighthttpd / Linux	Python / Linux (primary) Java / Linux (for Music, TV) Go / Linux (for games)	files @ CDN MariaDB / Linux
MicroSoft	General web sites	C# / ASP.NET / IIS / Windows		SQLServer / Windows
	Office 365	JavaScript / React	C# / ASP.NET / IIS / Windows	SQLServer / Windows ESE / Windows
	Bing search	C# / ASP.NET / IIS / Windows C# / Windows		proprietary graph database / proprietary storage system / Windows
	LinkedIn	Java and Scala / Tomcat and Jetty / Linux Kafka / Linux memcached / Linux		Oracle / Linux Lucene / Java / Linux Espresso / Linux Voldemort / Linux
NetFlix		JavaScript / React (for web) Kotlin (for Android) Swift (for iOS)	JavaScript / node.js / Linux Python / Linux Kotlin and Java / Linux Kafka / Linux	PostgreSQL and MySQL / Linux Cassandra / Linux Amazon DynamoDB
SalesForce		-	Java / Jetty and Resin / Linux (primary) Java and Kotlin / Linux Go / Linux Kafka / Linux memcached / Linux	Oracle / Solaris SOLR / Solaris proprietary storage system (C++) / Solaris
Slack		JavaScript / React (for web) JavaScript / Electron (for desktop) Java and Kotlin (for Android) Objective-C + Swift (for iOS)	PHP / Apache / Linux (primary) Java / Linux Go / Linux Kafka / Linux Redis / Linux	MySQL / Linux Lucene / Java / Linux
StackExchange (StackOverflow)		C# / ASP.NET / IIS / Windows Redis / Windows		SqlServer / Windows
Twitter		JavaScript / React JavaScript / node.js / Linux	Java and Scala / Linux Go / Linux Kafka / Linux Redis / Linux	MySQL / Linux Lucene / Java / Linux Cassandra / Linux
Wikipedia		PHP / Apache / Linux		MariaDB / Linux Lucene / .NET / Linux
Wordpress		PHP / Apache / Linux		MariaDB / Linux

Trends:

Some statistics:

Category	10 years ago	Today
Programming language	Java : 12 / 22 (2 for platforms) C++ : 10 / 22 (8 for platforms) C : 8 / 22 (8 for platforms) PHP : 6 / 22 C# : 4 / 22 JavaScript : 3 / 22 Perl : 2 / 22 Python : 2 / 22	Java : 18 / 22 (3 for platforms) JavaScript : 13 / 22 C++ : 8 / 22 (6 for platforms) C : 5 / 22 (5 for platforms) PHP : 5 / 22 Kotlin : 5 / 22 Swift : 5 / 22 C# : 4 / 22 Go : 4 / 22 Python : 3 / 22 Scala : 3 / 22
Web/App servers	Apache : 8 / 22 Jetty + Tomcat + JBoss + WebSphere : 3 + 1 + 1 + 1 / 22 ASP.NET/IIS : 4 / 22	Jetty + Tomcat + JBoss + WebSphere : 3 + 1 + 1 + 1 / 22 Apache : 5 / 22 ASP.NET/IIS : 4 / 22 node.js : 3 / 22 lighthttpd : 2 / 22
MQ servers		Kafka : 6 / 22 RabbitMQ : 2 / 22
Cache servers		Redis : 3 / 22 memcached : 3 / 22
Text search		Lucene : 4 / 22 SOLR : 2 / 22
Databases	MySQL : 8 / 22 Oracle : 3 / 22 SQLServer : 3 / 22 HBase : 2 / 22	MySQL + MariaDB : 5 + 4 / 22 Oracle : 3 / 22 SQLServer : 2 / 22 PostgreSQL : 2 / 22 HBase : 2 / 22 Cassandra : 2 / 22
OS	Linux : 20 / 22 Windows : 5 / 22 FreeBSD : 3 / 22 Solaris : 2 / 22	Linux : 17 / 22 Windows : 4 / 22

Comments:

Programming language:
- C/C++ is still popular but almost entirely for platform work not business application work
- Java is still popular
- JavaScript usage has increased many times
- React is very popular for web sites of these sizes
- PHP, C# and Python are used
- Newer languages like Kotlin, Scala and Go has seen some adaptation in recent years
- Lot of frontend has switched to mobile apps (Java/Kotlin and Swift)
Web/App servers:
- Apache usage has decreased due to increasing popularity of lighthttpd and nginx
- Java EE container are still widely used but their role within companies seems to be decreasing
- node.js has become popular in recent years
- ASP.NET/IIS is used
MQ servers:
- Usage of MQ servers has increased many times
- Kafka is king for web sites of these sizes
Cache servers:
- Usage of cache servers has increased many times
- Redis and memcached share the market
- None of the big web sites use the Java base cache servers (Hazelcast, Infinispan, Ignite etc.)
Text search:
- Usage of dedicated text search engines has increased many times
- The Apache projects Lucene and SOLR dominate the market
Databases:
- MySQL/MariaDB is still very popular
- Oracle and SQLserver are used
- NoSQL database (HBase, Cassandra, Voldemort) usage has increased (and noticeable the Java based ones)
- PostgreSQL has become popular in recent years
OS:
- Linux totally dominates this market
- Windows is used
- Real Unix usage is decreasing

Article history:

Version	Date	Description
1.0	February 6th 2010	Initial version (in Danish) published on Eksperten.dk
2.0	July 29th 2021	Translation to English and complete reformatting and publishing here
2.1	September 16th 2021	Add WordPress

Comments:

Please send comments to Arne Vajhøj

Technology behind the big web sites

Disclaimers:

10 years ago:

Today:

Trends:

Article history:

Other articles:

Comments: