{"id":114,"date":"2014-06-15T12:22:52","date_gmt":"2014-06-15T12:22:52","guid":{"rendered":"http:\/\/jobshuk.com\/yirmi\/?p=114"},"modified":"2020-07-30T19:13:51","modified_gmt":"2020-07-30T19:13:51","slug":"optimizing-high-traffic-massive-data-websites","status":"publish","type":"post","link":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/","title":{"rendered":"Optimizing High Traffic Massive Data Websites"},"content":{"rendered":"<p dir=\"LTR\"><span style=\"font-family: Calibri;\"><span style=\"color: #000000;\">The big dilemma in web development is that, on the one hand, web pages are expected to do more and more, while on the other hand, are expected to load instantaneously.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">A web page that takes longer than 2 seconds to load is considered slow.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">By 10 seconds many users give up, and 20 seconds load time is considered completely unusable.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">On the other hand, with dynamic data, hundreds of data queries need to be performed to display a single web page.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">When thousands of users are simultaneously accessing the page, the server access can get clogged up, increasing the server response time to render the page.<\/span><\/span><\/p>\n<p dir=\"LTR\"><b><span style=\"color: #000000; font-family: Calibri;\">Client Side Optimization<\/span><\/b><\/p>\n<p dir=\"LTR\"><span style=\"font-family: Calibri;\"><span style=\"color: #000000;\">What makes a page take time to be displayed? Loading time of additional files and script execution time.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">To decrease initial script execution time, the display initial should be as independent as possible from script execution, and scripts should be reserved for event handling.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">To decrease loading time of additional files, there are a few helpful methods.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">First of all, each http request has its own overhead, so combine JavaScript and CSS files to make less http requests.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Image files may also be combined into one larger sprite file.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Then, use background position to determine which image to be displayed.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Additionally, since these files are generally static, they may be loaded faster by storing them on a CDN, so that the end user will download them from a distributed server, decreasing network time.<\/span><\/span><\/p>\n<p dir=\"LTR\"><b><span style=\"color: #000000; font-family: Calibri;\">Database Optimization<\/span><\/b><\/p>\n<p dir=\"LTR\"><span style=\"font-family: Calibri;\"><span style=\"color: #000000;\">On the server side, the biggest bottleneck is database access.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">To optimize database response time, there are 2 types of optimizations \u2013 query optimization and server distribution.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">To optimize the queries, the first thing is to make sure that all the necessary indexes are in place.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Every field that is used in a &#8220;WHERE&#8221;,<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">&#8220;JOIN&#8221;, or &#8220;ORDER BY&#8221; should be indexed.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">This should eliminate the most sever inefficiencies.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">After that, overhead may be further reduced by monitoring the server for queries that take more resources and use EXPLAIN to find the inefficiencies.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Server distribution is increasing the number of database servers, so that each server will not have a long queue of requests.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">To do this, there should be a master server where are write command, such as &#8220;INSERT&#8221;, &#8220;UPDATE&#8221;, etc. are executed.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">All changes should be replicated on the slave servers.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">All read commands, such as &#8220;SELECT&#8221; should be performed on the slave servers.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">To speed things up further when there is greater traffic, additional slave servers should be created, and the requests should be distributed across the slave servers.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">The disadvantage of this is that the information queried could be a split-second out of date due to replication time.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">It also creates another place where data retrieval can go wrong, when the replication fails.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Still, with large data and traffic, server distribution is absolutely needed for the web site to run at reasonable speed.<\/span><\/span><\/p>\n<p dir=\"LTR\"><span style=\"font-family: Calibri;\"><b><span style=\"color: #000000;\">Caching<\/span><\/b><\/span><\/p>\n<p dir=\"LTR\"><span style=\"font-family: Calibri;\"><span style=\"color: #000000;\">After all the queries are optimized, the website may take time because there are still a large number of queries being executed.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">If the data is not constantly changing, there is no need for all the queries to be executed each time a page is loaded.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Instead, the content should be store in memcache.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Content will then load almost instantaneously.<\/span><span style=\"color: #000000;\">\u00a0\u00a0\u00a0 <\/span><span style=\"color: #000000;\">Sometimes, the specific page content is dynamic, but much of the data is static.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Here, the page may still load quicker when the model caches the static data.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Just make sure to invalidate the memchaced content when the data is updated.<\/span><\/span><\/p>\n<p dir=\"LTR\"><b><span style=\"color: #000000; font-family: Calibri;\">External Indexing<\/span><\/b><\/p>\n<p dir=\"LTR\"><span style=\"font-family: Calibri;\"><span style=\"color: #000000;\">Still, if I want to search the website&#8217;s data, I need dynamic results that cannot always be cached.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">How do I avoid increased database usage? Use and external data index, such as Sphinx to search the data.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">You can have it return the ids of each record of the search results.<\/span><span style=\"color: #000000;\">\u00a0 <\/span><span style=\"color: #000000;\">Once you have the ids, you just look up the records in memcache, and no direct database access is now required to display the search results. You can visit this <a href=\"https:\/\/www.couchbase.com\/pricing\">site<\/a> to find further information.\u00a0<\/span><\/span><\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>The big dilemma in web development is that, on the one hand, web pages are expected to do more and more, while on the other hand, are expected to load instantaneously.\u00a0 A web page that takes longer than 2 seconds to load is considered slow.\u00a0 By 10 seconds many users give up, and 20 seconds &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Optimizing High Traffic Massive Data Websites&#8221;<\/span><\/a><\/p>\n\n\n","protected":false},"author":375,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[73,28,72,71],"class_list":["post-114","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-big-data","tag-database","tag-high-traffic","tag-web-site-optimization","entry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Optimizing High Traffic Massive Data Websites - Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Optimizing High Traffic Massive Data Websites - Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer\" \/>\n<meta property=\"og:description\" content=\"The big dilemma in web development is that, on the one hand, web pages are expected to do more and more, while on the other hand, are expected to load instantaneously.\u00a0 A web page that takes longer than 2 seconds to load is considered slow.\u00a0 By 10 seconds many users give up, and 20 seconds &hellip; Continue reading &quot;Optimizing High Traffic Massive Data Websites&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/\" \/>\n<meta property=\"og:site_name\" content=\"Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer\" \/>\n<meta property=\"article:published_time\" content=\"2014-06-15T12:22:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-07-30T19:13:51+00:00\" \/>\n<meta name=\"author\" content=\"yirmi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"yirmi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/\"},\"author\":{\"name\":\"yirmi\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/#\\\/schema\\\/person\\\/3ad2fc75c4041622134d8f0f14a2102e\"},\"headline\":\"Optimizing High Traffic Massive Data Websites\",\"datePublished\":\"2014-06-15T12:22:52+00:00\",\"dateModified\":\"2020-07-30T19:13:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/\"},\"wordCount\":672,\"commentCount\":0,\"keywords\":[\"big data\",\"database\",\"high traffic\",\"web site optimization\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/\",\"url\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/\",\"name\":\"Optimizing High Traffic Massive Data Websites - Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/#website\"},\"datePublished\":\"2014-06-15T12:22:52+00:00\",\"dateModified\":\"2020-07-30T19:13:51+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/#\\\/schema\\\/person\\\/3ad2fc75c4041622134d8f0f14a2102e\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/2014\\\/06\\\/15\\\/optimizing-high-traffic-massive-data-websites\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Optimizing High Traffic Massive Data Websites\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/#website\",\"url\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/\",\"name\":\"Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer\",\"description\":\"Blog of a PHP Web Programmer\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/#\\\/schema\\\/person\\\/3ad2fc75c4041622134d8f0f14a2102e\",\"name\":\"yirmi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/wp-content\\\/uploads\\\/avatars\\\/375\\\/8c2adaf748317e659d83f249c32af044-bpfull.jpg\",\"url\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/wp-content\\\/uploads\\\/avatars\\\/375\\\/8c2adaf748317e659d83f249c32af044-bpfull.jpg\",\"contentUrl\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/wp-content\\\/uploads\\\/avatars\\\/375\\\/8c2adaf748317e659d83f249c32af044-bpfull.jpg\",\"caption\":\"yirmi\"},\"url\":\"https:\\\/\\\/jobshuk.com\\\/yirmi\\\/author\\\/yirmi\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Optimizing High Traffic Massive Data Websites - Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/","og_locale":"en_US","og_type":"article","og_title":"Optimizing High Traffic Massive Data Websites - Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer","og_description":"The big dilemma in web development is that, on the one hand, web pages are expected to do more and more, while on the other hand, are expected to load instantaneously.\u00a0 A web page that takes longer than 2 seconds to load is considered slow.\u00a0 By 10 seconds many users give up, and 20 seconds &hellip; Continue reading \"Optimizing High Traffic Massive Data Websites\"","og_url":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/","og_site_name":"Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer","article_published_time":"2014-06-15T12:22:52+00:00","article_modified_time":"2020-07-30T19:13:51+00:00","author":"yirmi","twitter_card":"summary_large_image","twitter_misc":{"Written by":"yirmi","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/#article","isPartOf":{"@id":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/"},"author":{"name":"yirmi","@id":"https:\/\/jobshuk.com\/yirmi\/#\/schema\/person\/3ad2fc75c4041622134d8f0f14a2102e"},"headline":"Optimizing High Traffic Massive Data Websites","datePublished":"2014-06-15T12:22:52+00:00","dateModified":"2020-07-30T19:13:51+00:00","mainEntityOfPage":{"@id":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/"},"wordCount":672,"commentCount":0,"keywords":["big data","database","high traffic","web site optimization"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/","url":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/","name":"Optimizing High Traffic Massive Data Websites - Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer","isPartOf":{"@id":"https:\/\/jobshuk.com\/yirmi\/#website"},"datePublished":"2014-06-15T12:22:52+00:00","dateModified":"2020-07-30T19:13:51+00:00","author":{"@id":"https:\/\/jobshuk.com\/yirmi\/#\/schema\/person\/3ad2fc75c4041622134d8f0f14a2102e"},"breadcrumb":{"@id":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/jobshuk.com\/yirmi\/2014\/06\/15\/optimizing-high-traffic-massive-data-websites\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jobshuk.com\/yirmi\/"},{"@type":"ListItem","position":2,"name":"Optimizing High Traffic Massive Data Websites"}]},{"@type":"WebSite","@id":"https:\/\/jobshuk.com\/yirmi\/#website","url":"https:\/\/jobshuk.com\/yirmi\/","name":"Yirmi&#039;s World of Web Widgets by Yirmiyahu Fischer","description":"Blog of a PHP Web Programmer","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jobshuk.com\/yirmi\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/jobshuk.com\/yirmi\/#\/schema\/person\/3ad2fc75c4041622134d8f0f14a2102e","name":"yirmi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jobshuk.com\/yirmi\/wp-content\/uploads\/avatars\/375\/8c2adaf748317e659d83f249c32af044-bpfull.jpg","url":"https:\/\/jobshuk.com\/yirmi\/wp-content\/uploads\/avatars\/375\/8c2adaf748317e659d83f249c32af044-bpfull.jpg","contentUrl":"https:\/\/jobshuk.com\/yirmi\/wp-content\/uploads\/avatars\/375\/8c2adaf748317e659d83f249c32af044-bpfull.jpg","caption":"yirmi"},"url":"https:\/\/jobshuk.com\/yirmi\/author\/yirmi\/"}]}},"_links":{"self":[{"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/posts\/114","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/users\/375"}],"replies":[{"embeddable":true,"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/comments?post=114"}],"version-history":[{"count":0,"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/posts\/114\/revisions"}],"wp:attachment":[{"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/media?parent=114"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/categories?post=114"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jobshuk.com\/yirmi\/wp-json\/wp\/v2\/tags?post=114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}