2023-04-17T23:57:27+00:00https://codeahoy.comSemantic Search - Word Embeddings with OpenAI2023-03-28T00:00:00+00:00https://codeahoy.com/2023/03/28/semantic-search-intro<p>According to Wikipedia, <a href="https://en.wikipedia.org/wiki/Semantic_search">Semantic Search</a> denotes <strong>search with meaning</strong>, as <strong>distinguished from lexical search</strong> where the search engine looks for <strong>literal matches</strong> of the query words or variants of them, without understanding the overall meaning of the query.</p>
<p>For example a user is searching for the term “jaguar.” A traditional keyword-based search engine might return results about the car manufacturer, the animal, or even the Jacksonville Jaguars football team. However, semantic search would analyze the context and intent behind the user’s query, such as whether they are interested in cars or wildlife, and then prioritize results accordingly.</p>
<p>In this blog post, we will explore the underlying principles of semantic search, discuss its advantages over other types of search, and examine real-world applications that are transforming the way we access and consume information.</p>
<h2 id="lexical-search-engines">Lexical Search Engines</h2>
<p>Lexical (Traditional) search engines have served us well using keyword-based search methods, looking for matching exact words or phrases in users’ queries with those in documents/database. For example, if we search for the term “<em>computer science intro</em>” in a lexical / traditional search engine, it will return results that match one or more of my search terms.</p>
<p><img src="/img/lexical-search-results.png" alt="lexical-search-results" class="center-image" /></p>
<p>As you can imagine, the keyword matching approach often falls short when it comes to understanding what the user actually meant, often producing less accurate results.</p>
<h2 id="semantic-search">Semantic Search</h2>
<p>Enter semantic search — a context-aware search technology that aims to improve search results by focusing on understanding the meaning and context behind queries.</p>
<p>When a user inputs the query “<em>computer science intro</em>” in a semantic search engine, it would first attempt to understand the intent behind the query. In this case, the user is likely looking for introductory resources related to computer science. Based on this understanding, the search engine would prioritize search results such as introductory computer science courses or textbooks or other learning materials that cover fundamental topics in computer science.</p>
<p>NLP plays a crucial role in semantic search, as it provides the necessary tools and techniques to analyze the context and relationships between words in a query, ultimately helping the search engine understand the meaning and intent behind user queries.</p>
<p>Semantic search can also be implemented using embeddings and vector databases. In this approach, both the user query and the documents in the search database are represented as <strong>vectors</strong> in a multi-dimensional space. By comparing the distance between vectors, we can determine the most relevant results.</p>
<h3 id="word-embeddings">Word Embeddings</h3>
<p>Word embeddings are a critical component in the development of semantic search engines and natural language processing (NLP) applications. They provide a way to represent words and phrases as numerical vectors in a high-dimensional space, capturing the semantic relationships between them. The key idea behind word embeddings is that similar words or phrases, like “big mac” and “cheeseburger,” should have closely related vector representations. This proximity in the vector space reflects their semantic relationship, enabling algorithms to better understand the meaning behind words and phrases.</p>
<p>There are several popular algorithms for generating word embeddings, including Word2Vec, GloVe, <a href="https://platform.openai.com/docs/guides/embeddings">OpenAI embeddings</a> and FastText. These algorithms work by analyzing large corpora of text data, learning the context and co-occurrence patterns of words, and then generating vector representations to capture these patterns. Words that often appear in similar contexts will have similar vector representations.</p>
<p>In short, word embeddings is powerful technique to represent words and phrases as <strong>numerical vectors</strong>. The key idea is that <strong>similar words</strong> have vectors in close proximity. Semantic search finds words or phrases by looking at the vector representation of the words and finding those that are close together in that multi-dimensional space.</p>
<p>For example, here’s how the process works in general if we use OpenAI Embeddings APIs for generating vector representations:</p>
<ol>
<li>
<p><strong>Document (or Dataset) embeddings:</strong> First, you would need to generate embeddings for the documents in your search database. You can use <a href="https://openai.com/blog/introducing-text-and-code-embeddings">OpenAI embeddings</a> to obtain these embeddings. Then you can store these vectors in a vector database or an efficient data structure like an approximate nearest neighbors (ANN) index.</p>
</li>
<li>
<p><strong>Query embeddings:</strong> When a user submits a query, you’d call the GPT-3 API to <em>generate an embedding</em> for the query using the same method as the document embeddings. This returns a single query vector.</p>
</li>
<li>
<p><strong>Similarity search:</strong> Compare the query vector to the document vectors stored in the vector database or ANN index. You can use <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a>, Euclidean distance, or other similarity metrics to rank the documents based on their proximity (or closeness) to the query vector in the high-dimensional space. The closer the document vector is to the query vector, the more relevant it is likely to be.</p>
</li>
<li>
<p><strong>Retrieve and display results:</strong> Finally, retrieve the top-ranked documents sorted by their similarity scores and display them to the user as search results.</p>
</li>
</ol>
<p>Keep in mind that this is a simplified example, and a full-fledged semantic search engine would involve additional considerations like query expansion, context-awareness, and other techniques to improve search results.</p>
<h2 id="word-embeddings-complete-example-on-github">Word Embeddings Complete Example on Github</h2>
<p>In the Python notebook linked below, we walk through the process of building a simple semantic search engine using word embeddings from OpenAI to illustrate basic concepts.</p>
<p>First let’s define our dataset of a few words that we’ll be searching against.</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="s">"hamburger"</span><span class="p">,</span> <span class="s">"cheeseburger"</span><span class="p">,</span> <span class="s">"blue"</span><span class="p">,</span> <span class="s">"fries"</span><span class="p">,</span> <span class="s">"vancouver"</span><span class="p">,</span> <span class="s">"karachi"</span><span class="p">,</span> <span class="s">"acura"</span><span class="p">,</span> <span class="s">"car"</span><span class="p">,</span> <span class="s">"weather"</span><span class="p">,</span> <span class="s">"biryani"</span><span class="p">]</span>
<span class="n">dataset</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">l</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="p">[</span><span class="s">'term'</span><span class="p">])</span>
<span class="k">print</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
</code></pre></div></div>
<p>This prints:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> term
0 hamburger
1 cheeseburger
2 blue
3 fries
4 vancouver
5 karachi
6 acura
7 car
8 weather
9 biryani
</code></pre></div></div>
<p>Next, we covert our dataset to embeddings by calling the embeddings API of OpenAI and storing their vector representations in our “database” (dataframe.)</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">openai.embeddings_utils</span> <span class="kn">import</span> <span class="n">get_embedding</span>
<span class="n">dataset</span><span class="p">[</span><span class="s">'embedding'</span><span class="p">]</span> <span class="o">=</span> <span class="n">dataset</span><span class="p">[</span><span class="s">'term'</span><span class="p">].</span><span class="nb">apply</span><span class="p">(</span>
<span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">get_embedding</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">engine</span><span class="o">=</span><span class="s">'text-embedding-ada-002'</span><span class="p">)</span>
<span class="p">)</span>
<span class="c1"># print terms and their embeddings side by side
</span><span class="k">print</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
</code></pre></div></div>
<p>This outputs:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">term</span> <span class="n">embedding</span>
<span class="mi">0</span> <span class="n">hamburger</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.01317964494228363</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.001876765862107277</span><span class="p">,</span> <span class="p">...</span>
<span class="mi">1</span> <span class="n">cheeseburger</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.01824556663632393</span><span class="p">,</span> <span class="mf">0.00504859397187829</span><span class="p">,</span> <span class="mf">0.</span><span class="p">...</span>
<span class="mi">2</span> <span class="n">blue</span> <span class="p">[</span><span class="mf">0.005490605719387531</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.007445123512297869</span><span class="p">,</span> <span class="p">...</span>
<span class="mi">3</span> <span class="n">fries</span> <span class="p">[</span><span class="mf">0.01848343200981617</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.030745232477784157</span><span class="p">,</span> <span class="o">-</span><span class="p">...</span>
<span class="mi">4</span> <span class="n">vancouver</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.011030120775103569</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.023991534486413002</span><span class="p">,...</span>
<span class="mi">5</span> <span class="n">karachi</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.004611444193869829</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.001336810179054737</span><span class="p">,...</span>
<span class="mi">6</span> <span class="n">acura</span> <span class="p">[</span><span class="mf">0.0055086081847548485</span><span class="p">,</span> <span class="mf">0.013021569699048996</span><span class="p">,</span> <span class="p">...</span>
<span class="mi">7</span> <span class="n">car</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.007495860103517771</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.021644126623868942</span><span class="p">,...</span>
<span class="mi">8</span> <span class="n">weather</span> <span class="p">[</span><span class="mf">0.011580432765185833</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.013912283815443516</span><span class="p">,</span> <span class="p">...</span>
<span class="mi">9</span> <span class="n">biryani</span> <span class="p">[</span><span class="o">-</span><span class="mf">0.009054498746991158</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.015499519184231758</span><span class="p">,...</span>
</code></pre></div></div>
<p>Now we are ready to search. We prompt user to enter a keyword. Once the user enters the keyword, we call the API to get its vector representation:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">openai.embeddings_utils</span> <span class="kn">import</span> <span class="n">get_embedding</span>
<span class="n">keyword</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s">'What do you want to search today? '</span><span class="p">)</span>
<span class="n">keywordVector</span> <span class="o">=</span> <span class="n">get_embedding</span><span class="p">(</span>
<span class="n">keyword</span><span class="p">,</span> <span class="n">engine</span><span class="o">=</span><span class="s">"text-embedding-ada-002"</span>
<span class="p">)</span>
<span class="c1"># print embedings of our keyword
</span><span class="k">print</span><span class="p">(</span><span class="n">keywordVector</span><span class="p">)</span>
</code></pre></div></div>
<p>To find the matching results, we iterate through the vectors of our dataset and apply <em>cosine similarity</em> to find the distance between keyword vector and each vector in the dataset. We printing top 3 results, sorted by the distance between vectors (keyword and dataset) in descending order.</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">openai.embeddings_utils</span> <span class="kn">import</span> <span class="n">cosine_similarity</span>
<span class="n">dataset</span><span class="p">[</span><span class="s">"distance"</span><span class="p">]</span> <span class="o">=</span> <span class="n">dataset</span><span class="p">[</span><span class="s">'embedding'</span><span class="p">].</span><span class="nb">apply</span><span class="p">(</span>
<span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">cosine_similarity</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">keywordVector</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">dataset</span><span class="p">.</span><span class="n">sort_values</span><span class="p">(</span>
<span class="s">"distance"</span><span class="p">,</span>
<span class="n">ascending</span><span class="o">=</span><span class="bp">False</span>
<span class="p">).</span><span class="n">head</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
</code></pre></div></div>
<p>Here’s are the top 3 results for the keyword “big mac” from our dataset. As you can see, it correctly inferred that I was referring to a burger and found the right matches!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>term distance
0 hamburger 0.853306
1 cheeseburger 0.841594
3 fries 0.823209
</code></pre></div></div>
<p>Here’s the complete code example:</p>
<h3 id="github---example-word-embeddings-and-semantic-search-using-openai"><a href="https://github.com/umermansoor/EmbeddingSemanticSearchDemo/blob/main/Embeddings101UM.ipynb">(GitHub - Example) Word Embeddings and Semantic Search using OpenAI</a></h3>
Brief Overview of Caching and Cache Invalidation2022-04-03T00:00:00+00:00https://codeahoy.com/2022/04/03/cache-invalidation<p>Caches are present everywhere: from the lowest to highest levels:</p>
<ul>
<li>There are hardware caches inside your processor cores (L1, L2, L3),</li>
<li>Page/Disk cache that our Operating Systems</li>
<li>Caches for databases such as using MemCached, Redis or DAX for DynamoDB</li>
<li>API caches</li>
<li>Layer-7 (Application layer) HTTP caches like Edge level caching in CDNs</li>
<li>DNS caching</li>
<li>Cache in your browser</li>
<li>Microservices can have internal caches to improve their performance for complex and time consuming operations</li>
<li>You are reading this post thanks to many intermediary caches</li>
</ul>
<p>I can keep going but you get the point: <strong>Caching is ubiquitous</strong>. Which begs the question, why do we need caching? Before you scroll down for the answer, take a few seconds to think about the answer.</p>
<h2 id="what-is-a-cache">What is a Cache?</h2>
<p>A cache is a fast data <strong>storage</strong> layer for storing a <strong>subset</strong> of data on a <strong>temporary</strong> basis for a duration of time. Caches are faster than original sources of data so they <strong>speed</strong> up future data retrievals by accessing the data in cache as opposed to fetching it from the actual storage location. Caches also make data retrievals <strong>efficient</strong> by avoiding complex or resource intensive operations to compute the data.</p>
<p>When the application needs data, it <em>first</em> checks if it exists in the cache. It if does, the data is read directly from the cache. If the data is not in cache, it is read from primary data store or generated by services. Once the data is fetched, it is stored in the cache so for future requests, it can be fetched from the cache.</p>
<p><img src="https://codeahoy.com/img/read-through.png" alt="read-through" /></p>
<h2 id="why-do-we-need-caching">Why do we need Caching?</h2>
<p>Typically, there are two main reasons for caching data:</p>
<ol>
<li>We cache things when the <strong>cost</strong> of generating some information is high (resource intensive) and we don’t need fresh information each time. We can calculate the information once, and then store it for a period of time and return the cached version to the users.</li>
<li>Arguably the top reason why we use caching is to <strong>speed</strong> up data retrieval. Caches are faster than original sources of data and cached information can be retrieved quickly resulting in faster responses to users.</li>
</ol>
<h3 id="caching-example">Caching Example</h3>
<p>Let’s look at an example. Suppose we have a webpage that displays “Related Content” links on the sidebar. This related content is generated by machine learning algorithms by <em>processing large volumes</em> of data in the main database, and can take <em>several seconds</em> to compute.</p>
<ul>
<li>This is a complex and resource intensive operation: each user request has to calculate this information. For popular pages on the website, a significant amount of time and resources will be spent computing the same data over and over again. <em>Impact</em>: Increased load on backend servers and databases, and higher cloud infrastructure costs.</li>
<li>Generating “Related Links” takes time and holds up the final response that’s sent to users. <em>Impact</em>: The <em>response times</em> increase that hurt user experience and page performance metrics such as the Core Web Vitals that search engines use.</li>
</ul>
<p>To <em>address</em> both these issues, we can use a “Cache”. We can computed the Related Links <strong>once</strong>, then store it in the cache and return the cached copy for several hours or even days. The next time the data is requested, rather than performing a costly operation and waiting for several seconds for it to complete, the result can be fetched from cache and returned to users faster. (This type of <strong>caching strategy</strong> is called <a href="/2017/08/11/caching-strategies-and-how-to-choose-the-right-one/#cache-aside">Cache Aside</a>.)</p>
<figure class="figure d-block">
<p><img src="/img/blogs/cache-website.png" class="rounded mx-auto d-block" alt="Why we use caching? Because it speeds up information delivery and reduces the cost of calculating that information over and over again." /></p>
<figcaption class="figure-caption text-center">
<p>Why we use caching? Because it speeds up information delivery and reduces the cost of calculating that information over and over again.</p>
</figcaption>
</figure>
<h2 id="cache-invalidation">Cache Invalidation</h2>
<p>We have seen how useful caches can be: they save costs, scale heavy workloads, and reduce latency. But like all good things, there’s a catch or rather trade-offs that developers must be aware of.</p>
<p><em>Phil Karlton</em>, an accomplished engineer who was an Architect at Netscape famously said the following which also happens to be my favorite quote:</p>
<blockquote>
<p>There are only two hard things in Computer Science: cache invalidation and naming things - Phil Karlton</p>
</blockquote>
<p>Cache invalidation is the process of <strong>marking the data in the cache as invalid</strong>. When the next request arrives, the corresponding invalid data must be treated as a cache-miss, forcing it to be generated from the original source (database or service.)</p>
<p>Caches are not the <strong>source of truth</strong> for your data. That’d be your database (or a service.) The problem happens when the data in your database (source of truth) changes, leaving <em>invalid</em> data in the cache. If the data is the cache is not invalidated, you’ll get inconsistent, conflicting or <strong>incorrect</strong> information. For example, suppose we cached the price of an item and the supplier increases it in their system.</p>
<p>Cache invalidation is indeed a hard problem. <em>Why?</em> Because we effectively need to deal with the <em>dependency graph</em> of all the inputs that gave us the result we cached. Any time even a single input changes, we have a stale or invalid result in the cache. Miss just one subtle place, and we have an issue. The program will still work making is very difficult to track down the exact issue and fix the cache invalidation logic. If you have a function with well defined inputs and outputs then it’s not that hard to catch issues. In fact, when it doesn’t work at all is usually one of the simpler things to find and fix. Cache invalidation bugs leading to the program “mostly working” makes somewhat trivial bugs fiendishly hard to discover.</p>
<p>Let revisit our earlier example of caching “Related Content” links (links to other related pages for a webpage.) Suppose one of the linked pages is no longer present in the system: it was taken down by an admin because of a complaint. We forgot to capture this input for cache invalidation. Now we get a “mostly working” system that results in users getting an HTTP 404 error when they click on a broken link. Debugging is very difficult because the actual page (that’s hosting the broken link) is not broken in any way. We only see HTTP 404 errors in the logs and troubleshooting turns into a nightmare.</p>
<p>In distributed systems with several inter-connected caches, invalidation becomes even more difficult thanks to many dependencies, race conditions and invalidating all the caches that need to be updated. <a href="/learn/tutorials/distributed-caching-at-scale/">Distributed caching</a> has its own challenges at scale and some complex systems like Facebook’s Tao use <strong>cache leaders</strong> for handling invalidations for all data under their shards.</p>
<p>Heck, it is easy to run into cache issues during the course of normal software development. Modern CPUs have several cores and each has its own cache (L1) that’s periodically synced with the main memory (RAM). In the absence of proper synchronization, values stored in variables on one thread may not be visible to threads. For example:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">foo</span> <span class="o">=</span> <span class="mi">2</span><span class="o">;</span>
</code></pre></div></div>
<p>In Java, the JVM might update the value of foo in the local cache and not commit the result to memory. A thread running on another core may see a stale value for <code class="language-plaintext highlighter-rouge">foo</code>. (This is one of the primary reasons why writing multithreading applications is hard.)</p>
<p>In summary, caching is a super useful technique. But it can easily go wrong if we are not careful. When using a cache, it’s important to understand how and when to invalidate it and to build proper invalidation processes.</p>
<h2 id="when-to-not-use-a-cache">When to Not Use a Cache</h2>
<p>Caches are <strong>not</strong> always the right choice. They may not add any value and in some cases, may actually degrade performance. Here are some questions you need to answer to determine if you need a cache or not.</p>
<ol>
<li>The original source of data is slow (e.g. a query that does complex JOINs in a relational database.)</li>
<li>The data doesn’t need to change for each request (e.g. caching real-time sensor data that your car needs when it’s in the self-driving mode or live medical data from patients… not good ideas.)</li>
<li>The operation to fetch the data must not have any <strong>side-effects</strong> (e.g. a Relational DB Transaction that fetches data and updated KPI counters is not a good caching candidate due to side-effect of updating counters.)</li>
<li>The data is frequently accessed and needed more than once.</li>
<li>Good cache hit:miss ratio and total cost of cache misses. For example, suppose I put a cache for user requests as they come in and it takes <strong>10 ms</strong> to check if the data <em>exists</em> in the cache or not, vs the original time of <strong>60 ms</strong>. If only <strong>5%</strong> of requests are cached, I’m <strong>adding</strong> an additional 10ms to 95% of the requests that result in a cache-miss. Doing rough calculations, we can see that cache is actually hurting performance:</li>
</ol>
<ul>
<li>Before cache: <code class="language-plaintext highlighter-rouge">1,000,000 requests * 60 milliseconds per request = 60,000,000 milliseconds total</code></li>
<li>After cache: <code class="language-plaintext highlighter-rouge">(0.05 * 1,000,000 * 10) + (0.95 * 1,000,000 * (60 + 10) ) = 67,000,000 milliseconds total</code> Each cache miss results in 60+10 millisecond That’s poorer than using no cache, assuming all requests are equal in value/distribution.</li>
</ul>
<h2 id="caching-strategies">Caching Strategies</h2>
<p>There are <strong>many</strong> different ways to configure and access caches. Various <a href="/2017/08/11/caching-strategies-and-how-to-choose-the-right-one/">cache strategies that are covered in this post</a>.</p>
How To Manage Employees Who Are Going Through a Difficult Period2022-01-02T00:00:00+00:00https://codeahoy.com/2022/01/02/managing-employee-with-personal-issues<p>Managing a team member who’s going through a difficult period is challenging scenario for any manager. It is not as uncommon as one might think. Major life events like death of a family member, prolonged illness of a child or spouse, separation or divorce, etc. are very stressful situations for almost everyone. Some might do a better job than others keeping their work and personal lives separate, but it is not uncommon for these these major life events to creep in and start affecting work performance of even top performers</p>
<p>So what can managers to do when one of their reports is going through a difficult period and it is affecting their ability to do their work? Two things are key here:</p>
<ol>
<li>You have to show compassion and help the employee,</li>
<li>You are also responsible for your team’s productivity and producing results for your company.</li>
</ol>
<p>Depending on the situation, it can like walking a tightrope. Managers must navigate carefully and come up with a plan. Here are some things managers can do in these situations.</p>
<h2 id="1-dont-wait---early-detection-goes-a-long-way">1. Don’t Wait - Early Detection Goes a Long Way</h2>
<p>As a manager, you shouldn’t expect your employees to come to you and share that they are going through a stressful period. If they do, great. But good managers must detect early warning signs and start the conversation early. If they start showing up late to meetings consistently, miss deadlines, quality of work goes down, etc. sit down and talk to them.</p>
<p>If you wait too long and don’t take an action, the entire team might suffer or the situation could get so bad that the employee might just directly resign, without ever giving you the opportunity to explore a solution together.</p>
<h2 id="2-understand-the-situation">2. Understand the Situation</h2>
<p>When you talk, and he or she opens up and shares that they are going through difficult times, the best thing you can do is listen intently and understand the situation as much as possible. Show compassion and care, but avoid being a therapist. Understand if it’s short-term challenge or something that would potentially drag for many months.</p>
<h2 id="3-dont-solve-solo-brainstorm-with-your-employee">3. Don’t Solve Solo. Brainstorm With Your Employee</h2>
<p>Many managers when faced with this scenario are tempted to offer <strong>time off</strong> or <strong>flexible hours</strong> to their employees. While these are good solutions in many situations, they aren’t always. Instead, <strong>brainstorm</strong> solutions with your employees. Start the conversation by expressing your support and genuine sympathy and asking the employee what the two of you and the company can do to support them during this difficult period and how to best manager their responsibilities.</p>
<p>May be the employee is working on a challenging project that requires a lot of collaboration and meetings and those are just not feasible at the moment. May be the on call responsibilities are too much for them for the next few weeks. The point is there could be ways to support your employee during personal crisis. It could even a combination of different things: flexible schedule plus independent tasks, as an example.</p>
<h2 id="4-make-a-plan">4. Make a Plan</h2>
<p>After brainstorming, make a plan that works for the employee and helps them go through their difficult situation while also meeting modified work expectations. Set realistic goals. Also make sure that the plan is in-line with your company’s HR policies. If you are not sure, review your company policies or if you have a good HRBP, talk to them.</p>
<h2 id="5-share-the-arrangement-with-the-team">5. Share the Arrangement With the Team</h2>
<p>The team usually knows about these situations because chances are they are a part of the solution in some way or other. However, if they don’t, you should let them know about any adjustments that you are making. The team will notice and if they don’t have the context, the team culture might suffer.</p>
<p>I must add that you should respect the employee’s privacy at all times. If they don’t want to share details with the team, then don’t share any details and keep it high level.</p>
<h2 id="6-follow-up">6. Follow up</h2>
<p>Once you have a plan, follow up with your employee time to time in your 1-1’s or even over Slack, Email. Don’t pry, instead show care, and ask how they are doing, how the plan is working and if any adjustments need to be made.</p>
<h2 id="summary">Summary</h2>
<p>An employee going through a personal crisis or difficult period can have a serious impact on their and teams’ productivity. Managers should be on the look out for any warning signs. Once the issue is detected, proceed with careful planning and compassion towards the employees who personal problems are affecting their work performance.</p>
Code Reviews During Emergencies2022-01-01T00:00:00+00:00https://codeahoy.com/2022/01/01/emergency-code-reviews<p>It’s 3:40pm on a fine Friday afternoon. You are about to wrap up the main logic for a feature you’ve been working on for a couple of days when you notice you have unread Slack notifications. One catches your eye in particular:</p>
<blockquote>
<p>“Hey! Emergency. Need approval on my PR. I have tested everything on dev so if you can quickly approve, I can merge it before the weekend.”</p>
</blockquote>
<p>Emergency. Alright, they have your attention. You could stop what you are doing and do a <em>superficial</em> code review on their <em>pull request</em> (<strong>PR</strong>) because after all, it’s an <em>emergency</em>. But you are curious and fire back seeking to understand the context:</p>
<blockquote>
<p>“Sure, I can take a look right now. What’s the emergency? Is it a bug in production or does the legal team want something updated ASAP?”</p>
</blockquote>
<p>They reply back right away:</p>
<blockquote>
<p>“No, I want to merge this code that does <em><insert something innocuous></em> because I’m OOO on Monday and want to get this released today”</p>
</blockquote>
<p>Urgent? Perhaps. But does this constitute an emergency? Definitely not.</p>
<p>In this post, we’ll discuss how to review code during emergencies. Let’s first begin by establishing an understanding of what constitutes a <em>real emergency</em>.</p>
<p><img src="/img/blogs/michael-scott-meme-emergency.png" class="rounded mx-auto d-block" alt="Michael Scott Meme 911 Emergency" /></p>
<h2 id="emergencies">Emergencies</h2>
<p>An emergency is a critical <strong>bug</strong> in production that’s affecting users, a major security issue, an urgent legal concern or something that’s blocking a major feature release that has a significant impact on KPIs.</p>
<p>I think it’s easier to understand if we look at a few examples of what is <strong>not</strong> an emergency:</p>
<h3 id="not-emergencies">Not Emergencies</h3>
<p>The following examples are not emergencies.</p>
<ul>
<li>The author has been working very hard and long hours on the feature and wants to get it out ASAP.</li>
<li>It’s 30-minutes to the weekend code-freeze deadline so it’ll be nice to get it merged.</li>
<li>The author’s manager told them that it’d be great if they can release the feature this week.</li>
<li>The author will be away for the next couple of days.</li>
</ul>
<p>These could relate to urgent situations or special circumstances that could definitely be important. Delaying a release may not be ideal, but it’s not usually <em>catastrophic</em>, hence not an emergency. (The only <em>exceptions</em> are feature releases where something bad would happen if it were to get delayed.)</p>
<h2 id="code-review-protocol-during-emergencies">Code Review Protocol During Emergencies</h2>
<p>You might ask yourself that if it’s so important to fix the bug during emergencies as soon as possible, should we <em>even bother with code reviews</em>? The answer is that yes, you should definitely review <strong>all</strong> changes, including during emergencies. However, during emergencies, we should <strong>adapt</strong> our code review process and make some changes to it.</p>
<p>Emergencies require special attention and a modified code review process. For example, during normal <a href="/2016/04/03/effective-code-reviews/">code reviews</a>, you might look at various aspects such as complexity, style, code documentation, unit tests, etc. and provide suggestions or nitpick. However, that won’t work during emergencies, where the goal is to complete the code review process as <strong>fast</strong> as possible.</p>
<p>Here are four things code review authors and reviewers should do differently during emergencies:</p>
<h3 id="1-pr-authors-keep-the-change-small-and-focused">1. PR Authors: Keep the Change Small and Focused</h3>
<p>The authors of the PR should make sure that it is a small change going out to specifically address the emergency only. Do not add anything extra that could confuse reviewers and introduce potential delays. Keep your change small and focused.</p>
<h3 id="2-reviewers-complete-review-as-fast-as-possible">2. Reviewers: Complete Review as Fast as Possible</h3>
<p>While normal code reviews can take anywhere from a few hours to a few days to complete, during emergencies it is critical that code reviews are completed as fast as possible. Speed is the main priority and takes precedence over everything else. Just like the author of the PR dropped everything to get the fix out, the reviewers should do the same and make code review their top priority.</p>
<h3 id="3-reviewers-limit-code-review-focus">3. Reviewers: Limit Code Review Focus</h3>
<p>During the normal code review process, a reviewer may look at various aspects of the code such as its design, complexity, <a href="/2016/05/22/effective-coding-standards/">code style</a>, to name a few. However, during emergencies, the reviewers should limit and restrict their focus to one and one thing only: “<em>does this address and resolve the emergency?</em>” In other words, the main criteria is that if we merge this code into production, will it make the bug go away? Refrain for providing suggestions or opinions as they’d have no impact on the PR’s ability to address the issue. I’ll repeat myself here: <strong>speedy code review is your top priority here</strong>.</p>
<h3 id="4-everyone-follow-up-with-more-through-code-review">4. Everyone: Follow-up With More Through Code Review</h3>
<p>Once the change has been released to production and the emergency has been resolved, and you have all taken a deep breath and relaxed, it’s time to <strong>revisit</strong> and do the regular or more <strong>thorough</strong> code review. Do not skip this step if reviewers noticed things during ‘emergency code review’ that you didn’t call out due to urgency at the time. For example, you may have noticed that the code was poorly structured and too complex to be understood in the future and maintained. Call it now so the author can follow up with another PR to address these issues.</p>
<h2 id="summary">Summary</h2>
<p>In this post, we looked at how to do code reviews during emergencies such as handling critical bugs in production. <strong>In emergency situations, we should not skip the code review process</strong>, but rather <em>adapt</em> it to focus on the <em>speed</em> of completing the review, keeping the code review focused on the code’s ability to address the issue and then following up with a more formal code review process.</p>
<p class="message">
I would love to hear your feedback, comments, and thoughts. Please leave a
comment below sharing your experience or anything that would add value to this article and its future readers.
</p>
Burnout in Software Development - Survey Results 20212021-10-01T00:00:00+00:00https://codeahoy.com/2021/10/01/software-developer-burn-out-survey<p>Burnout is very common in software development. Intense mental focus, <a href="https://ea-spouse.livejournal.com/274.html" rel="nofollow">heavy workloads</a>, never ending roadmaps, under-staffed teams, unclear targets, and many other factors lead to developer burnout.</p>
<p>I have experienced burnout in my career, perhaps twice. Early on, I had mild symptoms and my manager recognized it and helped me manage it. Later, after a very long stretch of working very long hours, 7 days a week on some super unrealistic goals with very little support, I ended up getting a bad case of burnout. It took me weeks to recover from it.</p>
<p><a href="https://www.unlaunch.io" target="_blank"> <img class="rounded img-fluid border" src="/img/caads/unlaunch-ad2.png" /> </a></p>
<p>When COVID-19 forced remote work shift began in companies, it was a step in the right direction by giving developers the flexibility to work from anywhere. However, I started hearing more and more about burnout. In this article, I will share results of a developer survey we did to explore if COVID-19 is somehow making the burnout situation worse.</p>
<h2 id="what-is-burnout">What is burnout?</h2>
<blockquote>
<p>Burnout <a href="https://www.webmd.com/mental-health/burnout-symptoms-signs" rel="nofollow">is</a> a form of <strong>exhaustion</strong> caused by constantly feeling swamped. It’s a result of excessive and prolonged emotional, physical, and mental <strong>stress</strong>. In many cases, burnout is related to one’s job.</p>
<p>Burnout happens when you’re overwhelmed, emotionally drained, and unable to keep up with life’s incessant demands.</p>
</blockquote>
<p>If it sounds very <em>broad</em>, it actually is. Burnout doesn’t have a test that your doctor can prescribe and be certain whether you are suffering from it or not. Regardless, if it’s not identified and addressed, it will continue to grow until it grinds you down and become the new normal. When that happens, it is not easy to pull yourself back out. It might take many months or even years to shake it off.</p>
<p>Here are some of the common symptoms of burnout:</p>
<ul>
<li>You stop enjoying working on things you used to enjoy and lose motivation for weeks.</li>
<li>You feel exhausted or fatigued all or most of the time. You have low energy at work and at home. You struggle to sleep and wake up feeling tired.</li>
<li>You don’t feel a sense of accomplishment or feel sort of hopeless about results.</li>
</ul>
<p>(The symptoms are pretty similar to <a href="https://en.wikipedia.org/wiki/Major_depressive_disorder" rel="nofollow">Depression</a>. Unlike burnout, depression is a clinical condition which requires immediate medical attention.)</p>
<h2 id="what-causes-burnout">What causes burnout?</h2>
<p>Here are some of the common <a href="https://www.mayoclinic.org/healthy-lifestyle/adult-health/in-depth/burnout/art-20046642" rel="nofollow">causes</a> of burnout:</p>
<ul>
<li>Lack of control. An inability to influence decisions that affect your job — such as your schedule, assignments or workload — could lead to job burnout. So could a lack of the resources you need to do your work.</li>
<li>Unclear job expectations. If you’re unclear about the degree of authority you have or what your supervisor or others expect from you, you’re not likely to feel comfortable at work.</li>
<li>Dysfunctional workplace dynamics. Perhaps you work with an office bully, or you feel undermined by colleagues or your boss micromanages your work. This can contribute to job stress.</li>
<li>Extremes of activity. When a job is monotonous or chaotic, you need constant energy to remain focused — which can lead to fatigue and job burnout.</li>
<li>Lack of social support. If you feel isolated at work and in your personal life, you might feel more stressed.</li>
<li>Work-life imbalance. If your work takes up so much of your time and effort that you don’t have the energy to spend time with your family and friends, you might burn out quickly.</li>
</ul>
<p>How to deal with burnout? Here’s a a good <a href="https://alphacolin.com/burnout-is-caused-by-resentment/" rel="nofollow">must-haves</a> list to prevent burnout:</p>
<ul>
<li>Exercising 2 - 3 times a week</li>
<li>Eating well (fruits and veggies)</li>
<li>8 hours of sleep at least 4 nights a week</li>
<li>Unplugging totally for at least a week a year</li>
</ul>
<p>I’ll add the following as well:</p>
<ul>
<li>Set aside some time (at least 30 minutes) to go for a walk everyday. Don’t check your Slack or emails when walking.</li>
<li>Don’t work from your bedroom. If you can, move your desk to a different room or another area. At the end of the day, walk away from your desk marking the end of the work day. Maintain proper <a href="https://codeahoy.com/2019/10/19/do-software-developers-work-weekends-work-life-tech/">work-life balance</a>.</li>
<li>Strongly second sleeping at least 8 hours more than once a week.</li>
<li>Spend time with your friends and family.</li>
<li>Meditate.</li>
</ul>
<h1 id="developer-survey-results">Developer Survey Results</h1>
<p>Let’s look at the results of the software developers survey to understand the scope and impact of burnout given COVID-19 related changes in workplaces. Here are the survey stats:</p>
<ul>
<li>A total of <em>504</em> developers participated in the survey.</li>
<li>71% of the participants who took the survey were from the US, while the remaining 29% were from rest of the world.</li>
<li>87% of the participants were individual contributors with titles such as Software/Lead/Staff/
Principle Engineer or DevOps. Rest were in (engineering) or product management.</li>
<li>62% worked in companies with more than 500 employees.</li>
</ul>
<h2 id="82-of-all-developers-indicated-that-they-have-experienced-burnout-in-last-6-to-8-months">82% of all developers indicated that they have experienced burnout in last 6 to 8 months</h2>
<p>The first question on the list asked software developers if they are experiencing or have experienced burnout in the last 6 to 8 months. A whopping 82% said yes. Almost 50% developers cited feelings of burnout to a “great” or “moderate” extent.</p>
<p><img src="/assets/images/burnout/burnout.001.jpeg" alt="Software developers burnout survey chart and stats - 1" /></p>
<hr />
<h2 id="73-developers-said-burnout-is-negatively-impacting-their-productivity-or-personal-life">73% developers said burnout is negatively impacting their productivity or personal life</h2>
<p>This was not a surprise. Lack of energy and motivation directly leads to lower productivity at work. Burnout also affects personal life and relationships.</p>
<p><img src="/assets/images/burnout/burnout.002.jpeg" alt="Software developers burnout survey chart and stats - 2" /></p>
<hr />
<h2 id="developers-indicated-increased-workload-and-and-poor-work-culture-as-the-main-reason">Developers indicated increased workload and and poor work culture as the main reason</h2>
<p>This was a multiple choice question. There were many options to choose from and participants could even come up with their own answer. I have summarized work-related reasons as “Poor work culture” and pandemic related as “COVID-19”.</p>
<p><img src="/assets/images/burnout/burnout.003.jpeg" alt="Software developers burnout survey chart and stats - 3" /></p>
<hr />
<h2 id="57-developers-indicated-that-covid-19-has-made-the-situation-worse">57% developers indicated that COVID-19 has made the situation worse</h2>
<p><img src="/assets/images/burnout/burnout.004.jpeg" alt="Software developers burnout survey chart and stats - 4" /></p>
<p>Participants were also given an option to provide more details by leaving comments. The top theme was not having to commute to the workplace every day as the silver lining.</p>
<hr />
<h2 id="77-of-the-developers-indicated-that-their-management-is-not-aware-of-the-burnout-or-not-taking-any-steps-to-help-its-employees-manage-it">77% of the developers indicated that their management is not aware of the burnout or not taking any steps to help its employees manage it</h2>
<p>Only 23% said that their company has formally recognized burnout and other challenges and taking concrete steps such as additional days off, reasonable worloads, etc. to ensure employee well-being. Kudos to companies who are doing this!</p>
<p><img src="/assets/images/burnout/burnout.005.jpeg" alt="Software developers burnout survey chart and stats - 5" /></p>
<hr />
<h2 id="78-of-the-developers-said-they-are-planning-to-switch-their-job-within-the-next-12-months">78% of the developers said they are planning to switch their job within the next 12 months</h2>
<p>I was expecting this number to be high, but not this high. Only 22% cited burnout as the reason behind their desire to switch jobs. I wish I had a comments enabled for this question to better understand the reasons why such large percentage of developers are considering a career change.</p>
<p><img src="/assets/images/burnout/burnout.006.jpeg" alt="Software developers burnout survey chart and stats - 6" /></p>
<h2 id="other-comments">Other comments</h2>
<p>Developers surveyed were asked to leave their comments. I can’t post them all, but here are few of them.</p>
<blockquote>
<ul>
<li>Lots of engineers left. The workload has increased significantly and management can’t hire enough people. Working longer hours and wait for the weekends to recharge. But weekends are mostly groceries, chores, helping kids with their homework.. the cycle continues</li>
<li>Inability to disconnect from work, expectations of being always reachable and no thought for planning for alternate coverage.</li>
<li>I work as a QA for 3 years and I’ve recently got hired after being fired by low productivity. My understanding is the overload of what clients consider on our job. It should be related to software Quality Assurance, but because I only dealt with legacy systems (bank and flight), QAs don’t even know the expected behavior, not even the users themselves. I’m planning on switching jobs.</li>
<li>I’m still fairly new to my position, and it takes time to get used to a new codebase, company, and processes, which makes my job more stressful and creates pressure to get up to speed.</li>
</ul>
</blockquote>
<p>There you have it. Thanks to everyone who participated in the survey and provided their feedback. Leave your comments in the comments section below.</p>
<h3 id="if-you-enjoyed-this-post-please-sign-up-for-unlaunch-which-lets-you-release-your-code-whenever-you-want-and-change-its-behavior-on-the-fly-using-feature-flags">If you enjoyed this post, please sign up for <a href="https://www.unlaunch.io/">Unlaunch</a>, which lets you release your code whenever you want and change its behavior on the fly using Feature Flags.</h3>
How to use Feature Flags in Node.js2021-09-12T00:00:00+00:00https://codeahoy.com/2021/09/12/how-to-use-feature-flags-in-node-js<p>Feature flags (or feature toggles) is a powerful technique used by modern software teams to control the behavior of their code and features in production. Feature flags are used for:</p>
<ul>
<li>rolling new features out gradually</li>
<li><a href="https://martinfowler.com/bliki/BranchByAbstraction.html">branch by abstraction</a></li>
<li>experimentation and testing</li>
<li>migrations</li>
<li>geographic restrictions</li>
<li>permissions or kill switch</li>
<li>testing in production</li>
</ul>
<p>In this article, I’ll show you how to use feature flag in Node.js. For this example, let’s assume we have an e-commerce store and we’re implementing a <strong>new feature</strong> to change the sort order of the items because in our hypothetical example, the marketing team thinks changing the sort order will lead to more revenue. I’ll be using an open-source Node project called <a href="https://github.com/Mahrukhizhar/crate">Crate</a> which is an eCommerce subscription service for trendy clothes and accessories.</p>
<h2 id="1-put-your-feature-behind-feature-flag-in-code">1. Put your feature behind feature flag in code</h2>
<p>First, you’ll need to define the new feature in such a way it can be shown or hidden easily.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if featureFlag == "on" then
// show the new feature: new sort order
else
// show the old feature: old sort order
</code></pre></div></div>
<p>Once you have wrapped your code (new and old features) in a feature flag, you can easily control the behavior by changing the state of the feature flag or its targeting rules. If the flag is enabled (“on” variation,) we show the new feature. Otherwise, we don’t.</p>
<h2 id="2-create-a-feature-flag">2. Create a feature flag</h2>
<p>Next, create a new feature flag in <a href="https://app.unlaunch.io">Unlaunch</a> that will control the feature you’re building. You can call it the “default-sort-order” or “crate-products”.</p>
<h3 id="targeting-users">Targeting Users</h3>
<p>A user is any object such as an email address, unique user id, a hash etc. to represent a unique user for which a feature flag is evaluated. Let’s define an internal user (engineering, QA or product team member) by email address to always show the “on” variation so they always get the new feature.</p>
<h3 id="targeting-rules">Targeting Rules</h3>
<p>The rules that define which variation is evaluated are known as flag targeting rules. Here you can target by user attributes such as new users vs old, or by geo-location. Default Rule is the “catch-all” rule when targeting rules don’t match. In this case, the goal is to enable the feature for 10% of the users. To do that, we’ll define percentages under Default Rule.</p>
<p><img src="/img/feature-flags/nodejs-feature-flag.png" alt="Feature flag setup" /></p>
<h2 id="3-integrate-unlaunch-nodejs-sdk-in-your-app">3. Integrate Unlaunch Node.js SDK in your app</h2>
<p>You’ll need to use the Unlaunch Node.js SDK to evaluate the feature flag you just defined from your app. If you’re using NPM, you can easily install the SDK:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm install --save unlaunch-node-sdk
</code></pre></div></div>
<h2 id="4-call-the-feature-flag-and-use-variation-to-show-or-hide-the-feature">4. Call the feature flag and use variation to show or hide the feature</h2>
<p>Open the file where wrapped the new feature in step #1. Let’s call the feature flag we just defined and use its result instead.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">UnlaunchFactory</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">"</span><span class="s2">unlaunch-node-sdk</span><span class="dl">"</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">variation</span> <span class="o">=</span> <span class="nx">unlaunch</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nx">variation</span><span class="p">(</span><span class="dl">"</span><span class="s2">crate-products</span><span class="dl">"</span><span class="p">,</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">user</span><span class="p">.</span><span class="nx">email</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="nx">variation</span> <span class="o">==</span> <span class="dl">'</span><span class="s1">on</span><span class="dl">'</span><span class="p">){</span>
<span class="k">return</span> <span class="k">await</span> <span class="nx">models</span><span class="p">.</span><span class="nx">Product</span><span class="p">.</span><span class="nx">findAll</span><span class="p">({</span> <span class="na">order</span><span class="p">:</span> <span class="p">[[</span><span class="dl">'</span><span class="s1">id</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">ASC</span><span class="dl">'</span><span class="p">]]</span> <span class="p">})</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">await</span> <span class="nx">models</span><span class="p">.</span><span class="nx">Product</span><span class="p">.</span><span class="nx">findAll</span><span class="p">({</span> <span class="na">order</span><span class="p">:</span> <span class="p">[[</span><span class="dl">'</span><span class="s1">id</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">DESC</span><span class="dl">'</span><span class="p">]]</span> <span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here’s the actual <a href="https://github.com/Mahrukhizhar/crate/blob/6dc37fbd5a3b37e35f5c43e27c80a4fe215f9573/code/api/src/modules/product/resolvers.js">commit</a>.</p>
<p>Here’s what the store looks like when the feature is disabled.</p>
<p><img src="/img/feature-flags/nodejs-ff-feature-disabled.png" alt="Feature flag is disabled - showing old sort order" /></p>
<p>When you enable the feature, we can see the new sort order.</p>
<p><img src="/img/feature-flags/nodejs-ff-feature-enabled.png" alt="Feature flag is enabled - showing new sort order" /></p>
<p>The cool thing is that now, we can enable or disable the feature on demand. We can roll it out to more users, kill it if it’s misbehaving, without changing the code or deploying new releases.</p>
<h2 id="summary">Summary</h2>
<p>Feature flags are easy to set up and add a lot of flexibility to your application. Even if you use a feature flag service provider or create your own solution, I’m confident that your team will appreciate the idea of decoupling feature releases from code deployments because it removes a major source of stress.</p>
<h2 id="github-repo">GitHub Repo</h2>
<p>Here’s the <a href="https://github.com/Mahrukhizhar/crate">complete source code</a> used in this blog post is available on GitHub. Here’s the commit with comments describing changes: <a href="https://github.com/Mahrukhizhar/crate/commits/main">https://github.com/Mahrukhizhar/crate/commits/main</a>.</p>
<h2 id="learn-more">Learn More</h2>
<ul>
<li><a href="https://martinfowler.com/articles/feature-toggles.html">Blog post</a> from Pete Hodgson at MartinFowler.com is a good read.</li>
<li><a href="https://docs.unlaunch.io/docs/getting-started/">Getting Started with Feature Flags</a></li>
<li><a href="hhttps://docs.unlaunch.io/docs/sdks/nodejs-sdk">Unlaunch Node.js SDK</a></li>
</ul>
How to Toggle Features in C# with Feature Flags2021-08-26T00:00:00+00:00https://codeahoy.com/2021/08/26/how-to-toggle-features-in-c-sharp<blockquote>
<p>This blog post was contributed by Tuan Nguyen, Software Developer at Getty Images.</p>
</blockquote>
<p>Using feature flags to release new features to customers is a powerful technique. I have been using feature flags for many years to release features or changes to production safely and with the peace of mind. The releases are done without any ceremony, and the features aren’t visible to customers until we turn the feature flag on. Using feature flags to toggle functionality in .NET Core apps is easily achievable using a feature flag management platform.</p>
<p>In this tutorial, I’ll show you how to toggle features on demand in a simple web application using <a href="https://unlaunch.io">Unlaunch</a> to create and manage feature flags. We’ll also see how to show or hide our features with a click of a button, without merging branches, rollbacks or deploys.</p>
<h2 id="feature-flags">Feature Flags</h2>
<p>Feature flags allow developers to control who sees new features irrespective of code deployment. For example, developers can deploy a new feature to production environment and keep it hidden from all users (except themselves.) This way, they can do testing on real systems, and when the management is ready to release the feature, they can turn on the feature flag to let the users in on it. To me, here are some of the benefits of using feature flags:</p>
<ul>
<li>Developers eliminate some of the risk knowing they can roll things back instantly if things go wrong. On a few occasions, I rolled back features because of negative impact to KPI we didn’t consider before or one of the underlying system wasn’t quite ready.</li>
<li>Developers can launch the feature on production behind feature flag and show it to product or marketing team for their feedback.</li>
<li>QA in production. This doesn’t mean that we skipped testing on the ‘Dev’ environment, but rather launching the feature on production just for the development team give extra boost of confidence. It also allowed testing with real data which is not always up to date on ‘Dev’ environment.</li>
<li>Gradually roll features out starting with a small percentage of users. This is also known as canary releases.</li>
</ul>
<h2 id="basics-of-feature-flags">Basics of Feature Flags</h2>
<p>A feature flag is a simple <em>object</em> that has a Boolean value or variations such as <em>“on”</em> or <em>“off”</em> Based on the value of the variation, you either do one thing or the other.</p>
<div class="language-cs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">string</span> <span class="n">feature</span> <span class="p">=</span> <span class="nf">getFeatureFlag</span><span class="p">(</span><span class="s">"FLAG-ID"</span><span class="p">);</span><span class="c1">// Get feature flag</span>
<span class="k">if</span> <span class="p">(</span><span class="n">feature</span> <span class="p">==</span> <span class="s">"on"</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// show new code </span>
<span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span>
<span class="c1">// show old code</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You can use a use a config file (<code class="language-plaintext highlighter-rouge">appsettings.json</code>) or a database to keep feature flags. You can also use a number of feature flag platforms which provide a web UI to manage and control feature flags and SDKs to make it easy to integrate across the stack.</p>
<h2 id="implementing-feature-flags-in-net-core">Implementing Feature Flags in .NET Core</h2>
<p>I’ll show you how to implement and use feature flags in .NET Core and ASP.NET. I will be using the <a href="https://github.com/unlaunch/dotnet-sdk">Unlaunch SDK for.NET</a> to manage and retrieve feature flags from my code. We’ll use the eShop Web application. Our imaginary <strong>use case</strong> (inspired from real-life) is to change the sort order on our store. We want to show items in reverse order.</p>
<p><img src="/img/feature-flags/normal.jpg" width="75%" alt="eShopWeb - Normal sort order" class="img-thumbnail mx-auto d-block" /></p>
<p>I’ll show you how to implement the new feature (sort items in reverse order) and put it <em>behind</em> a feature flag. When the feature flag is turned “on”, the items will be sorted in reverse order. If the feature flag is turned “off”, the sort order will not change.</p>
<h2 id="before-your-get-started">Before Your Get Started</h2>
<h3 id="register-an-unlaunch-account">Register an Unlaunch Account</h3>
<p>Go to <a href="https://unlaunch.io">Unlaunch.io</a> and create a free account. As part of sign up, create a new project. (The name of the project should match that of your application. I named my project ‘eShopWeb’.) Next, I created a new feature flag called ‘catalog_reverse’. Leave everything to its default value. Make sure you enable the flag by clicking on the “Enable Flag” button in the top right corner. To make the feature flag return “on” variation (to signal that we should show the feature to all users), choose “on” variation under “Default Rule”.</p>
<p><img src="/img/feature-flags/csharp-feature-flag-turned-on.png" alt="C# feature flag returning on variation for all users" /></p>
<h3 id="access-the-feature-flag-from-net-core-application">Access the Feature Flag from .NET Core Application</h3>
<h4 id="install-the-unlaunch-sdk">Install the Unlaunch SDK</h4>
<p>To use the feature flag ‘catalog_reverse’ in our application, we need to integrate the <a href="https://github.com/unlaunch/dotnet-sdk">Unlaunch .NET SDK</a>.</p>
<p>Install the <a href="https://www.nuget.org/packages/unlaunch/">Nuget package</a> or run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Install-Package unlaunch
</code></pre></div></div>
<h4 id="initialize-unlaunch-client-as-a-singleton">Initialize Unlaunch Client as a Singleton</h4>
<p>In <a href="https://github.com/tnguyenquy/eShopOnWeb/commit/116ee293f79b7bac52fd7957dfa2feb033384c2b#diff-7428d2e4bcb9fdecb7a556159a82f21be0ed7fd99ae85a4402460609b67677fdR166"><code class="language-plaintext highlighter-rouge">src/Web/Startup.cs</code></a>, initialize the Unlaunch client. The client will download the feature flag right away and will poll for changes every 60 seconds. You can <a href="https://docs.unlaunch.io/docs/sdks/dotnet-sdk#configuration">customize</a> the polling interval, but in our case, 60 seconds works great. A number of feature flag platforms like LaunchDarkly uses a streaming architecture but I prefer polling to avoid any extra overhead and in most cases, it is okay to wait up to a minute to disable the feature if the flag is turned off using the web UI.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">services</span><span class="p">.</span><span class="n">AddSingleton</span><span class="p"><</span><span class="n">IUnlaunchClient</span><span class="p">>(</span><span class="n">UnlaunchClient</span><span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="s">"YOUR_SDK_KEY"</span><span class="p">));</span>
<span class="n">services</span><span class="p">.</span><span class="n">AddTransient</span><span class="p"><</span><span class="n">UnlaunchService</span><span class="p">>();</span>
</code></pre></div></div>
<p>To find your SDK key, in the <a href="https://app.unlaunch.io">Unlaunch web UI</a>, and click on “Settings” in the sidebar. Copy the <strong>Server Key</strong> for the environment you’re working with (Production in my case.) Additional information: Unlaunch provides 3 different types of SDK keys for security purposes. For example, the “Browser Key” is for client-side SDKs like React that run on client side. To access feature flags with this SDK key, you must update flag settings to make it available on client-side.</p>
<p><img src="/img/feature-flags/sdk-keys-find.png" alt="Find Unlaunch SDK Keys" /></p>
<h4 id="evaluate-feature-flag-and-get-on-or-off-variation">Evaluate feature flag and get “on” or “off” variation</h4>
<p>I created a new service for Unlaunch called <code class="language-plaintext highlighter-rouge">UnlaunchService</code> within the application to handle feature flag logic in there and inject this service in other places.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">namespace</span> <span class="nn">Microsoft.eShopWeb.Web.Services</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">UnlaunchService</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">const</span> <span class="kt">string</span> <span class="n">CatalogReverseFlag</span> <span class="p">=</span> <span class="s">"catalog_reverse"</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IUnlaunchClient</span> <span class="n">_client</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">UnlaunchService</span><span class="p">(</span><span class="n">IUnlaunchClient</span> <span class="n">client</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_client</span> <span class="p">=</span> <span class="n">client</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="kt">bool</span> <span class="nf">IsCatalogReverseFlagEnabled</span><span class="p">(</span><span class="kt">string</span> <span class="n">userIdentity</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">variation</span> <span class="p">=</span> <span class="n">_client</span><span class="p">.</span><span class="nf">GetVariation</span><span class="p">(</span><span class="n">CatalogReverseFlag</span><span class="p">,</span> <span class="n">userIdentity</span><span class="p">);</span>
<span class="k">return</span> <span class="n">variation</span> <span class="p">==</span> <span class="s">"on"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I created a wrapper function (<code class="language-plaintext highlighter-rouge">IsEnabled</code>) to return whether the feature flag is enabled or not. It takes a string argument called <code class="language-plaintext highlighter-rouge">userIdentity</code>. This is important when doing gradual or phased rollouts as Unlaunch SDK uses hash of userIdentity to determine whether to show the feature or not to a particular user. For this example, this argument is not used because we are turning the flag on or off for <strong>all</strong> users. But we’ll pass it anyways because we’ll be using it later.</p>
<p>Then in my <code class="language-plaintext highlighter-rouge">Index.cshtml</code>, I check if the feature flag is enabled or not and if enabled, I sort the items in the reverse order.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">class=</span><span class="s">"esh-catalog-items row"</span><span class="nt">></span>
@{
if (UnlaunchService.IsCatalogReverseFlagEnabled(HttpContext.Connection.RemoteIpAddress.ToString()))
{
Model.CatalogModel.CatalogItems.Reverse();
}
foreach (var catalogItem in Model.CatalogModel.CatalogItems)
{
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"esh-catalog-item col-md-4"</span><span class="nt">></span>
<span class="nt"><partial</span> <span class="na">name=</span><span class="s">"_product"</span> <span class="na">for=</span><span class="s">"@catalogItem"</span> <span class="nt">/></span>
<span class="nt"></div></span>
}
}
<span class="nt"></div></span>
</code></pre></div></div>
<p>As we can see, items in the shop are <strong>reverse sorted</strong> when the feature flag is turned “on”.</p>
<p><img src="/img/feature-flags/reverse.jpg" width="75%" alt="eShopWeb - Reverse sort when feature flag is on" class="img-thumbnail mx-auto d-block" /></p>
<h2 id="targeting-user-segments-with-feature-flags">Targeting User Segments with Feature Flags</h2>
<h3 id="targeting-specific-users-by-id">Targeting Specific Users By ID</h3>
<p>Suppose you want to turn the feature on only for internal users so they can see how things look. For example, if I wanted to keep the feature off for everyone and only enable it for one of my colleagues whose working from home, I can target by IP address. (Alternatively, you can also target users by other criteria e.g. all users whose email address ends with <code class="language-plaintext highlighter-rouge">xyz@company.com</code>, but if you recall, our shop is visible to logged our users and we’re passing IP address as identity.)</p>
<p><img src="/img/feature-flags/csharp-ff-targeting-ip.png" alt="feature flag IP address targeting in C#" /></p>
<h3 id="targeting-by-geo-location">Targeting by Geo-location</h3>
<p>Often times, we roll out feature starting with countries that would have limited impact on revenue or other KPIs if things go south. For example to target everyone from Germany (DEU) and Austria (AUT). This often allow to learn the behavior and impact of a new feature before opening it up for broader audience. To do this, <a href="https://docs.unlaunch.io/docs/attributes/attributes">create a new Attribute</a> in Unlaunch of type “Set” and call name “country”.</p>
<p><img src="/img/feature-flags/csharp-ff-targeting-by-geo.png" alt="feature flag targeting by country in C#" /></p>
<p>To make this work, in our code we’d have to pass an additional attribute that contains user’s country so the SDK can decide which variation to show. In the Unlaunch service class, I updated the <code class="language-plaintext highlighter-rouge">isEnabled</code> to resolve user country by IP and pass it to the SDK.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="kt">bool</span> <span class="nf">IsEnabled</span><span class="p">(</span><span class="kt">string</span> <span class="n">userIdentity</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">variation</span> <span class="p">=</span> <span class="n">_client</span><span class="p">.</span><span class="nf">GetVariation</span><span class="p">(</span>
<span class="n">FlagKey</span><span class="p">,</span>
<span class="n">userIdentity</span>
<span class="n">UnlaunchAttribute</span><span class="p">.</span><span class="nf">NewSet</span><span class="p">(</span><span class="s">"country"</span><span class="p">,</span> <span class="nf">GetLetterCountryCodeByIP</span><span class="p">(</span><span class="n">userIdentity</span><span class="p">)),</span>
<span class="p">);</span>
<span class="k">return</span> <span class="n">variation</span> <span class="p">==</span> <span class="s">"on"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="targeting-by-email">Targeting by Email</h3>
<p>Suppose, we want to target all logged in users in your company. We can define a targeting rule to show the feature to everyone whose email ends with <code class="language-plaintext highlighter-rouge">@@yourcompanydomain.com</code>.</p>
<p><img src="/img/feature-flags/csharp-ff-targeting-by-email.png" alt="feature flag targeting by email in C#" /></p>
<p>You’d have to start passing the email address to the SDK so it can make the decision.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="kt">bool</span> <span class="nf">IsEnabled</span><span class="p">(</span><span class="kt">string</span> <span class="n">userIdentity</span><span class="p">,</span> <span class="kt">string</span> <span class="n">email</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">variation</span> <span class="p">=</span> <span class="n">_client</span><span class="p">.</span><span class="nf">GetVariation</span><span class="p">(</span>
<span class="n">FlagKey</span><span class="p">,</span>
<span class="n">userIdentity</span>
<span class="n">UnlaunchAttribute</span><span class="p">.</span><span class="nf">NewString</span><span class="p">(</span><span class="s">"email"</span><span class="p">,</span> <span class="n">email</span><span class="p">),</span>
<span class="p">);</span>
<span class="k">return</span> <span class="n">variation</span> <span class="p">==</span> <span class="s">"on"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I just gave you a few examples that I’ve been using to target segments using feature flags. The possibilities are endless and in my opinion, this is where feature flags truly shine. You can release a feature on production and turn in on only for specific users (just your product manager or QA engineer,) or everyone in the company, or everyone in certain countries, etc. Try it out and have fun with it.</p>
<h2 id="summary">Summary</h2>
<p>Feature flags are easy to set up and add a lot of flexibility to your application. Even if you use a feature flag service provider or create your own solution, I’m confident that your team will appreciate the idea of decoupling feature releases from code deployments because it removes a major source of stress.</p>
<h2 id="github-repo">GitHub Repo</h2>
<p>Here’s the <a href="https://github.com/tnguyenquy/eShopOnWeb">complete source code</a> used in this blog post is available on GitHub. Here’s the commit with comments describing changes: <a href="https://github.com/tnguyenquy/eShopOnWeb/commit/116ee293f79b7bac52fd7957dfa2feb033384c2b">https://github.com/tnguyenquy/eShopOnWeb/commit/116ee293f79b7bac52fd7957dfa2feb033384c2b</a>.</p>
<h2 id="learn-more">Learn More</h2>
<ul>
<li><a href="https://martinfowler.com/articles/feature-toggles.html">Blog post</a> from Pete Hodgson at MartinFowler.com is a good read.</li>
<li><a href="https://docs.unlaunch.io/docs/getting-started/">Getting Started with Feature Flags</a></li>
<li><a href="https://docs.unlaunch.io/docs/sdks/dotnet-sdk">Unlaunch .NET SDK</a></li>
</ul>
The Complete Guide to Feature Flags2021-02-27T00:00:00+00:00https://codeahoy.com/2021/02/27/feature-flags<p>The term feature flags refers to a set of <em>techniques</em> that allow software developers and teams to change the behavior of their system in production without modifying or even deploying code.</p>
<p>Because of their ability to modify system behavior on the fly, feature flags are very <em>powerful</em> and <em>versatile</em>. They facilitate many use cases that boost developer productivity and make the user experience better and faster: gradually rolling out new features to users vs big-bang releases, testing in production, canary launches, experimentation and many more.</p>
<p>We’ll explore feature flag use cases later. But first, let’s examine feature flags in depth and see how they work.</p>
<h2 id="what-is-a-feature-flag">What is a feature flag?</h2>
<p>At the core of this amazing concept, lies a dead simple and basic foundation that uses conditional code (<code class="language-plaintext highlighter-rouge">if</code> statement) to determine whether to perform an action or not.</p>
<p>This is best explained with an example. I’ll use a real one that I worked on not too long ago. We wanted to allow our users to sign-in to our site using their Google account.</p>
<p>To do this, you first create a developer account with Google and get the OAuth code. The application must be approved by Google before you can use it on your site. Until then, it can only be used in <em>test</em> mode.</p>
<p>Traditionally, the code for Google sign-in will be kept in a separate Git branch. When the application is approved by Google, the branch can be merged into <code class="language-plaintext highlighter-rouge">develop</code> or <code class="language-plaintext highlighter-rouge">main</code> (aka <code class="language-plaintext highlighter-rouge">master</code>) so it can be released to users.</p>
<p>The challenge was that the verification process may take several weeks. I tested everything locally to make sure it was all working. But I couldn’t release it yet, even for internal users. If I merged into <code class="language-plaintext highlighter-rouge">develop</code> to deploy the feature on the development environment, it would also release it to production (since <code class="language-plaintext highlighter-rouge">release</code> branches were automatically cut off of <code class="language-plaintext highlighter-rouge">develop</code>.)</p>
<p>To summarize the issue: I wanted to release an <em>unfinished</em> feature to the internal team for feedback and <em>dogfooding</em>. I had no way to show it to the team without merging it to the <code class="language-plaintext highlighter-rouge">develop</code> branch.</p>
<p>I’d ask you to pause here and think how you’d solve this problem? Scroll up and read again if you just skimmed through.</p>
<p>The first solution that may come to mind is quite simple. Introduce a <strong>new</strong> <strong>variable</strong>, a config parameter e.g. <code class="language-plaintext highlighter-rouge">showSignInButton</code> or an environment variable e.g. <code class="language-plaintext highlighter-rouge">IS_PRODUCTION</code>. Use this in an <strong>conditional</strong> or <code class="language-plaintext highlighter-rouge">if-else</code> statement in the code to disable the feature in production but keep showing it on the development environment.</p>
<p><img src="/assets/images/featureflags/feature-flag-variations.png" alt="Feature Flag variations" /></p>
<p>And voila! You have arrived at the (most basic) <strong>definition</strong> of feature flag.</p>
<p>To use feature flags in our code, there are three core things that we need:</p>
<ol>
<li>Find the <strong>seam</strong> where you could hide or disable the feature. In the example I shared above,the <em>seam</em> was the login form container in React. I used feature flag to decide whether to render the sign-in button component or not.</li>
<li>Determine feature flag variation. This is usually done by calling a function called <code class="language-plaintext highlighter-rouge">evaluate()</code>, we’ll explore later, but for now all we need to know is that this function can return “on/off” or “true/false” using some logic.</li>
<li>Use the variation as a condition in the <code class="language-plaintext highlighter-rouge">if-then-else</code> statement to determine which block of code to execute e.g. the <code class="language-plaintext highlighter-rouge">if</code>, or the <code class="language-plaintext highlighter-rouge">else</code> block. In our example, “on” or “true” would call the sign in button component.</li>
</ol>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">variation</span> <span class="o">=</span> <span class="nx">evaluate</span><span class="p">(</span><span class="dl">"</span><span class="s2">google-sign-in-btn</span><span class="dl">"</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">variation</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">on</span><span class="dl">"</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// code to render Google sign-in component</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c1">// don’t show it</span>
<span class="p">}</span>
</code></pre></div></div>
<p>While this is how feature flags generally work, a complete feature flag platform like <a href="https://www.unlaunch.io/product.html">Unlaunch</a> is much more comprehensive, as we’ll see later.</p>
<p>Let’s take a deep dive into the internals of feature flags and how it all works, including the evaluation logic.</p>
<h2 id="how-feature-flags-work">How feature flags work?</h2>
<h3 id="anatomy-of-a-feature-flag">Anatomy of a feature flag</h3>
<p>A feature flag itself is nothing but an object or a container that contains the following key properties:</p>
<ul>
<li><strong>Name</strong> or unique identifier: The name of the feature flag must be unique from all other flags in the scope.</li>
<li><strong>Variations</strong>: A feature flag can be 2 or more variations (multivariate.) Variations are simply strings e.g. “on” or “off”. These are supposed to be used by developers in “if” statements to select the code path. In some feature flagging systems, this could also be “true” or “false”.</li>
<li><strong>(Targeting) Rules</strong>: These take in context such as the user attributes from HTTP request to determine which variation to return. For example, you can set targeting rules to enable a feature by returning “on” variation for new users only.</li>
</ul>
<p><img src="/assets/images/featureflags/feature-flag-internals.png" alt="Internals of feature flag" /></p>
<p>There can be many more properties of a feature flag such as whether it is enabled or disabled, dynamic configuration etc.</p>
<p>To recap: A feature flag is an object that contains a bunch of properties such as variations, targeting rules etc.</p>
<p>You can think of a feature flag JSON object describing its properties and state.</p>
<p>The second central component of a feature flagging system is called the evaluator.</p>
<h3 id="evaluator">Evaluator</h3>
<p>Evaluator is a method that takes as input the:</p>
<ul>
<li><strong>feature flag</strong> object (which as we learned in the last section contains all the properties,) and,</li>
<li><strong>context</strong> such as the user id and attributes from the HTTP request</li>
</ul>
<p>The output of the evaluator is always the variation that developers should use in their code to determine the code path. As we learned, variations are strings such as “on” or “off” that are used as conditions in an <code class="language-plaintext highlighter-rouge">if</code> statement.</p>
<p>The evaluator uses these two arguments to choose which variation to return. It knows how to evaluate the targeting rules.</p>
<p><img src="/assets/images/featureflags/feature-flag-management-mechanics.png" alt="Feature flag evaluation and mechanics" /></p>
<p>Side note: If you’re a Java developer, read this <a href="https://codeahoy.com/2020/11/22/feature-flags-with-java/">post</a> on how to integrate and use feature flags in Spring Boot applications.</p>
<h2 id="history-of-feature-flags">History of Feature Flags</h2>
<p>Facebook engineering and research have built a number of amazing products. Apache Cassandra, React, GraphQL are great examples.</p>
<p>While they didn’t invent this technique, Facebook certainly pioneered and made heavy use of feature flags internally.</p>
<p>Facebook is a massive system of interconnected backend services, databases, frontend, interfaces and more. As they grew big, they faced a challenge. How to release new features, changes and updates without breaking things.</p>
<p>In other words, how can the massive social networking site enable thousands of engineers to get their code out quickly to users in safe, small and incremental steps.</p>
<p>While a lot has gone into powering continuous delivery at scale at Facebook, pushing out an endless stream of changes every single hour, it was made possible by an internal tool called Gatekeeper.</p>
<p>What is Gatekeeper? You might have already guessed it but it’s Facebook’s internal system for managing feature flags. Jack Lindamood, former Facebook engineer, <a href="https://www.quora.com/How-does-Facebooks-Gatekeeper-service-work" rel="nofollow">described</a> it as:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="nf">gatekeeper_allowed</span><span class="p">(</span><span class="s1">'my_feature_name'</span><span class="p">,</span> <span class="nv">$viewing_user_or_application</span><span class="p">))</span> <span class="p">{</span>
<span class="nf">run_this_tested_code</span><span class="p">();</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nf">run_this_old_code</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>From Facebook engineering <a href="https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale" rel="nofollow">blog:</a></p>
<blockquote>
<p>If we do find a problem, we can simply switch the gatekeeper off rather than revert back to a previous version or fix forward.</p>
<p>This quasi-continuous release cycle comes with several advantages:</p>
<p><strong>It eliminates the need for hotfixes.</strong>…
<strong>It allows better support for a global engineering team.</strong>… all engineers everywhere in the world can develop and deliver their code when it makes sense for them.
<strong>It makes the user experience better, faster.</strong>… when it takes days or weeks to see how code will behave, engineers may have already moved on to something new. With continuous delivery, engineers don’t have to wait a week or longer to get feedback about a change they made. They can learn more quickly what doesn’t work, and deliver small enhancements as soon as they are ready instead of waiting for the next big release. …</p>
</blockquote>
<p>While Facebook adopted many practices to power continuous delivery at its massive scale, feature flags were <em>fundamental</em> to their approach.</p>
<p>Today, feature flags are widely used at large and medium size companies.</p>
<p>But are they useful just to super large organizations like Facebook or can small companies and startups can benefit from them as well? We’ll discuss this a little later in this post, but the short answer is <em>yes</em>.</p>
<h4 id="whats-in-a-name-that-which-we-call-a-rose">What’s in a name? That which we call a rose…</h4>
<p>Feature flags are known by several other names in the development community: <strong>feature toggles</strong> or <strong>feature flippers</strong> and perhaps a few more. For the remainder of this post, we’d stick to the term feature flag as it’s the most popular.</p>
<h2 id="feature-flag-use-cases">Feature Flag Use Cases</h2>
<p>Earlier I mentioned that feature flags are very versatile and can be used to achieve a variety of tasks. They are useful not just for engineering teams, but equally useful to QA, Operations and Product teams.</p>
<p>Let’s recap the key point that we have learned about feature flags so far:</p>
<blockquote>
<p>Feature flags allow developers to modify the behavior of their code at runtime. In other words, they give the ability to ship multiple code paths and choose between them at runtime.</p>
</blockquote>
<p>That’s incredibly powerful and can be used in a variety of contexts to achieve many different goals.</p>
<h3 id="canary-releases-and-gradual-roll-outs-rapid-releases-at-scale">Canary Releases and Gradual Roll outs: Rapid Releases at Scale</h3>
<p>This is perhaps the most common use case of feature flags that I have encountered.</p>
<p>Traditionally, product features were launched as all-or-nothing. Also known as big-bang releases, this meant making new features available to all users at some cut-off point or the release date. These types of releases require a lot of testing prior to launch and if something goes wrong later, the developers have to patch or get hotfixes out to resolve the bug.</p>
<p>In other words, when the feature is ready, you let it rip.</p>
<p>In canary releases, instead of launching the feature to all your users at the same time, it is released to a small number of users initially. This allows reviewing results such as user sentiment and more concrete metrics like system load, performance etc. Once satisfied, the feature can be launched to a wider group of users.</p>
<p>Canary releases limit the <strong>blast radius</strong>. If issues are discovered, only a small number of users are impacted.
Canary releases have been in use for a long time. Traditionally, they are achieved by having a separate cluster of servers or a replica of production environment. The load balancer sends a small number of users to the cluster where the new version of the application is deployed. For example 2% of the traffic goes to new code, rest goes to the existing production environment.</p>
<p><img src="/assets/images/featureflags/blue-green-deployments.png" alt="Canary releases using Blue Green Deployments" /></p>
<p>You can also achieve canary launches using feature flags. In fact, feature flags bring several improvements over traditional canary releases.</p>
<ul>
<li>Canary clusters are typically controlled by DevOps or Operations teams. It requires jumping through extra hops and coordination if a developer needs to roll back something.</li>
<li>If you do need to roll back a feature, all features in the canary cluster will be rolled back. Including good features that other teams may have deployed.</li>
<li>While canary releases are better than all-or-nothing releases, there are still discrete jumps: 0% to 2% to 100% that may not be ideal for all use cases.</li>
<li>Not all teams have the budget or resources to manage a separate canary cluster.</li>
</ul>
<p>Feature flags put canary releases on steroids. If I compare traditional canary clusters to monoliths, feature flag based ‘canary’ releases are like microservices. They are small, decentralized and put control back in the hands of teams and developers.</p>
<p>Feature flags put control right back into the hands of developers. Developers can launch their features when they want it, disable just the feature that is misbehaving without impacting other features.
And best of all, feature flags do not require building a complex and expensive infrastructure to do canary releases.</p>
<p>They also enabled a wonderful use case that I personally love: Gradual or Percentage based roll outs. This allows teams to slowly ramp up traffic to their feature in any ‘continuous’ increments they want.</p>
<p>For example, when we switched over to a new search backend (ElasticSearch,) we initially sent 1% of the traffic to it. Later we increased it to 5% and monitored for several days. We kept increasing it gradually. At 60% traffic, we discovered several issues. We instantly disabled the feature with a click of a button, fixed the issue and resumed again.</p>
<p>I have also seen some teams let their PMs release (low risk) features such as a new blog post on their site, awards lists, etc. in coordination with marketing.</p>
<h3 id="alpha-testing-and-dogfooding">Alpha testing and dogfooding</h3>
<p><a href="https://blog.unlaunch.io/2021-02-15-dogfodding-at-unlaunch/">Dogfooding</a>, aka eating your own dog food or drinking your own champagne, is the practice of using your own products within your company. It’s a great way to not only test products in practical, real-world situations for finding bugs, but also to first hand experience your product like your users.</p>
<p>While dogfooding should be an <strong>on-going</strong> endeavor, it is especially important in the context of new features or major changes and provides an early opportunity for internal users to use the feature and provide meaningful feedback before opening the floodgates.</p>
<p>Feature flags are great for establishing a culture of dogfooding within the company. Release features to production but only for internal teams and allow them to be the first users.</p>
<h3 id="continuous-delivery">Continuous Delivery</h3>
<p>Continuous delivery is not a new concept. Over time, our field has evolved and created many methodologies such as agile software developers, continuous integration, continuous delivery to name a few, with the goal of delivering new changes to their users faster, safer and higher quality.</p>
<p>The typical git-flow goes like this:</p>
<ol>
<li>Developers create separate feature branches to work on their features</li>
<li>When the feature is complete, it is merged into the develop branch. This usually results in the feature being deployed to the central development or QA environment.</li>
<li>The code is then merged into the master or release branch and released to production.</li>
</ol>
<p><img src="/assets/images/featureflags/git-flow-feature-flags.png" alt="Git flow" /></p>
<p>While this was a great approach, it has a few challenges:</p>
<ol>
<li>The feature branches can be long running. Even if the feature is complete, the team may have to wait for an external approval or another feature to launch before they can release their code.I have seen cases where feature branches have sit in isolation for weeks before they are merged into <code class="language-plaintext highlighter-rouge">develop</code> because they were dependent on the results of an AB Tests that hasn’t completed.</li>
<li>The feature cannot be released even internally until the feature branch is merged into the <code class="language-plaintext highlighter-rouge">develop</code> branch.</li>
<li>I have always held the belief that the best test environment is the production environment. I’m not saying everyone should test only on production, but it is the most accurate. It’s hard to emulate production traffic and patterns in test environments. Following git flow, because we can’t merge our features until ready, we can’t get them to production, even just for ourselves or the team.</li>
</ol>
<p>Using feature flags, you can merge in-progress feature branches into <code class="language-plaintext highlighter-rouge">develop</code> or even <code class="language-plaintext highlighter-rouge">main</code> (release) branches. They allow unfinished features to be released to production. Not only this gets rid of the long running git branches, it also great for <a href="https://blog.unlaunch.io/2021-02-15-dogfodding-at-unlaunch/">dogfooding</a>, or showing off features to internal teams for testing or feedback.</p>
<p>For example, we used to regularly push almost-done features to production, enabling for internal users only by IP or email domain e.g. ‘@company.com’, and share links around (including the CEO and relevant stakeholders.) Everyone has access to the production environment and we instantly received feedback.</p>
<p>If you think about it, in git-flow, merging code is tightly coupled with releasing it to your users. When you merge to develop, it is released to everyone within the company. When you merge it to master, it is released to all users.</p>
<p>Feature flags provide a nice way of separating the two concerns: merging code is separated from when you release it. In other words, you can merge unfinished features all day long and only allow users to see it when ready.</p>
<h3 id="ab-testing-and-experimentation">AB Testing and Experimentation</h3>
<p>Feature flags are great for running AB tests. You can define buckets (variations) and assign percentages to variations.</p>
<p>Product and marketing teams have been running AB tests for a long time. They are great for vetting new ideas quickly, and eliminating bad ones.</p>
<p><img src="/assets/images/featureflags/A-B_testing_example.png" alt="AB Testing using feature flags" /></p>
<p>While the situation is improving, I don’t see engineers running a lot of experiments for technical features that they implement. AB tests are usually pitched by product teams.</p>
<p>I don’t blame engineers entirely - AB testing is a complex business especially for big product related features and requires deep analysis (usually by analytics or data science teams across many KPIs.)</p>
<p>But not everything requires that level of in-depth analysis. Sometimes, all that’s needed to call an experiment is the count of errors across the test buckets and controls.</p>
<p>At my last two jobs, we successfully ran many light-weight, engineering focused experiments. Things like response time impact across variants, error counts, etc. were commonly experimented upon. Feature flags made it easy to run these and quickly analyze the results.</p>
<h3 id="kill-switches-and-performance-knobs">Kill switches and performance knobs</h3>
<p>Feature flags can be used to implement kill-switches. These allow, usually DevOps or Operation teams, to gracefully degrade non-essential functionality when under load or in the event of outages. For example, a website may disable upload functionality for some or all users if there’s too much load on the backend systems or the system is under attack.
Kill switches are related to another concept called the circuit breaker which enables or disables a certain code path automatically. It can be thought of as a manual circuit breaker.</p>
<p>Performance knobs are similar to kill switches but instead of completely disabling a feature, they are used for throttling or rate limiting a feature or API for a certain class of users. For example, under extreme load, you could limit access to new users but allow old users to continue as normal.
Migrations especially those that require coordination</p>
<p>Feature flags can be very useful when performing migrations especially those that require coordination.</p>
<h3 id="feature-flags-lifespan-and-dynamism">Feature Flags Lifespan and Dynamism</h3>
<p>We explored many use cases of feature flags. In his extensive post on <a href="https://martinfowler.com/articles/feature-toggles.html" rel="nofollow">feature flags</a>, Pete Hodgson, created a nice graph of <em>types</em> of feature flags that he described vs their <em>lifespan</em>.</p>
<p>Feature flags fall in two categories with respect to their lifespan: <strong>temporary</strong> vs long-living or even <strong>permanent</strong>. A feature flag you’re using to roll out a new feature is temporary. When the feature is 100% rolled out, the flag can be deleted. On the other hand, kill switch to control whether to show or hide a heavy widget on your website is a permanent flag that will always be there.</p>
<p>The graph below shows lifespan - how long feature flags stay active in code vs dynamism. <strong>Dynamism</strong> describes how dynamic a feature flag is. At one end of the spectrum, you have flags like kill switches that don’t care about any context. At the other end, you have feature flags that depend on per-request parameters like type of user and user attributes.</p>
<p><img src="/assets/images/featureflags/feature-flags-temporary-vs-permanent.png" alt="Feature flags lifespan. Temporary vs permanent flags" /></p>
<h2 id="how-to-feature-flags">How to feature flags?</h2>
<p>There are many ways to implement and use feature flags.</p>
<p>The easiest way is to use hard-coded <code class="language-plaintext highlighter-rouge">if-else</code> statements in your code. This approach is not very flexible and defeats the purpose of feature flags (changes to feature state such as enable, disable or gradual rollout need code deploys.)</p>
<p>Taking it one step further, you can make the flag condition configurable. Such as adding it to a database where you can change it on the fly.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">enableLatestWidget</span> <span class="o">=</span> <span class="nx">db</span><span class="p">.</span><span class="nx">read</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">enableLatestWidget</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// show widget</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c1">// don’t show it</span>
<span class="p">}</span>
</code></pre></div></div>
<p>While this is better than the first approach and certainly works, it’s very brittle.</p>
<p>You can’t do things like gradual roll outs, targeting users based on their IDs or attributes, experiment or look at KPIs. There’s no way to look at all the feature flags laid out on a nice dashboard and collaborate with your team in one place.</p>
<p>At least not without writing a lot of code yourself.</p>
<p>Taking it even a step further brings us to complete <strong>feature flag management tools</strong> like <a href="https://www.unlaunch.io/">Unlaunch</a>.</p>
<p>Comparing managing feature flags using a database vs feature flag tools is like comparing Excel spreadsheet to Jira (or Trello.) Sure, you can track tasks in Excel, but it’s much more efficient to use Jira.</p>
<p>Arguably the biggest benefit these mature feature flag tools provide is encouraging feature flags and experimentation best practices and establishing a <strong>culture</strong> organization wise.</p>
<p>These tools have low overhead. For server-side applications, flags are fetched when application starts (and then refreshed periodically.) All evaluations occur in-memory, introducing no additional latency.</p>
<div class="alert alert-light" role="alert">
<p class="text-danger lead text-center mt-3">
Unlaunch is a complete feature flag management tool that I have been working on since last year. It is free for solo developers and small teams.</p>
<p class="lead text-center">
<a href="https://app.unlaunch.io/signup" class="text-primary">Sign up for a free account today</a>
</p>
</div>
<p><a href="https://app.unlaunch.io/signup">
<img src="/assets/images/featureflags/feature-flags-dashboard.png" class="img-fluid" alt="Feature Flags Dashboard" />
</a></p>
<h3 id="cleaning-up-after-yourself">Cleaning up after yourself</h3>
<p>This is important enough to warrant its own section.</p>
<p>You must remove feature flags from your code once they are no longer needed. If you launched a new feature using feature flags, after it is rolled out at 100% and there’s no risk (or you can’t disable it anymore,) go remove the feature flag code from your code. Search by the name of the flag in your code and delete the flag code and code path that is not used anymore.</p>
<h3 id="feature-flags--startups">Feature Flags & Startups</h3>
<p>Feature flags are very common in large to medium sized organizations and have been around for many years.</p>
<p>But I have not seen a whole lot feature flags use at smaller companies or startups.</p>
<p>Which raises the <em>question</em>: can startups and smaller companies reap from using feature flags or is it just needless overhead?</p>
<p>The answer is that startups could and <em>absolutely should use feature flags</em>.</p>
<p>Startups operate at a much faster pace than large organizations, by their nature. Feature flags are an ideal vehicle to safely and reliably keep releasing new features to your users. By adopting feature flags early, startups can:</p>
<ul>
<li>build a culture of safe and fast releases.</li>
<li>use feature flags for Alpha testing or <em>dogfooding</em> their products. This is great for teams who don’t have the budget or resources to maintain a separate QA or development environment to test or stage their features.</li>
</ul>
<p>Last year I was consulting for a small startup that was building Chatbots. They didn’t have enough budget to build and maintain a separate QA environment to test things out before they release to their users. So we built a simple feature flags based system that allowed its developers to release features to production but only allow access to internal teams (by phone number.)</p>
<div class="alert alert-light" role="alert">
<p class="text-danger lead text-center mt-3">
Are you using feature flags in your organization release new features faster and with confidence? Please comment below to share any tips, ideas or best practices. </p>
</div>
Tutorial on using Feature Flags in Java [Complete Example]2020-11-22T00:00:00+00:00https://codeahoy.com/2020/11/22/feature-flags-with-java<p>Feature flags (or feature toggles) is a powerful technique used by modern software teams to manage the behavior of their code in production. Gone are the days when production releases happened occasionally and were a big event. These days, agile teams deliver changes to production continuously, sometimes releasing changes several times a day without any fanfare. Feature flags let teams release their changes to production and stay in full control over who gets to see the changes and when. Using feature flags, you can:</p>
<ul>
<li>deploy changes to production but keep them hidden from everyone but internal users.</li>
<li>gradually roll out features to build confidence.</li>
<li>if things don’t look good (you discover errors), roll back instantly without any code changes to redeploys.</li>
<li>test your changes into production.</li>
</ul>
<p>I’ve been using feature flags for the last 5 years and find them very useful in shipping changes quickly and confidently to millions of users, from infrastructure changes like DB upgrades to UI changes.</p>
<p>In this post, we’ll see how to use feature flags in Java using Spring Boot. Let’s get started.</p>
<p>But first, head over to Unlaunch and <strong><a href="https://app.unlaunch.io/signup" target="_blank" rel="noopener noreferrer">create a free account</a></strong>. After you have created a new account, come back and resume reading.</p>
<h2 id="challenge">Challenge</h2>
<p>Suppose you’re a software developer working on a backend service: User Profile. There is an API that returns user information from the database, combining it with information retrieved from some other services and sources. One day, you were reviewing the performance of the API and noticed it is very slow and takes up to a second to return response.</p>
<p>After some investigation and find the root cause. The delay is being introduced due to the way the service calls other services to get user’s status, open orders and pending cancelations: the calls are being made sequentially, that is, one after the other. You think you can improve the performance by <em>parallelizing</em> the calls.</p>
<p>In traditional software development, you’d create a new feature branch e.g. <code class="language-plaintext highlighter-rouge">feature/JIRA-101-async-calls-in-user-profile</code>. You’d make your code changes and keep committing to this branch. When everything looks good, you’d open a pull request to merge into the development branch. You’d do some more QA and performance testing the QA environment. Finally, your code will be merged into the main branch and your feature will go to staging and then to production. If you find any issues in production, you’d have to create a hotfix into main directly to rollback your changes.</p>
<p>Sounds stressful. If you think about the change you’ve identified, i.e. making asynchronous calls, it is not complicated. But the way software releases are done in large enterprises and the risk of breaking something even with a small change and impacting millions of users, there’s a lot of added friction and stress.</p>
<p>Enter feature flags. Let’s see how we’d utilize feature flags to solve performance issues and release with the peace of mind.</p>
<ol>
<li>Create a new feature flag. Turn it off for everyone except yourself. You can do by user id, by environment, or any other criteria.</li>
<li>Create a new method which makes asynchronous calls to other services.</li>
<li>In the main method, check if the feature flag is <strong>on</strong>.
<em>3a:</em> If the feature flag is <em>on</em>: call the new method which makes async calls
<em>3b:</em> If the feature flag is <em>off</em>: Call the old method that you’re trying to replace.</li>
<li>Enable the feature flag in the QA environment. This will allow everyone using the QA environment to see your changes and test them.</li>
<li>Merge into the main branch if everything looks good. Enable the feature on your staging environment. Enable it for all internal teams in production but keep it hidden from your users.</li>
<li>Compare the performance (API response time) of your change to the old API. If everything looks great, gradually roll out the feature to real users.</li>
</ol>
<p>Let’s see how we can do these steps and use feature flags to release our code to production quickly and confidently.</p>
<h2 id="solution">Solution</h2>
<h3 id="1-set-up-feature-flag">1. Set up Feature Flag</h3>
<p>If you haven’t already done so, head over to <a href="https://app.unlaunch.io/signup">https://app.unlaunch.io/signup</a> and create a new account. <a href="https://unlaunch.io">Unlaunch is a free feature flag service</a>. As part of account creation, you’d be asked to create a new <strong>project</strong>. A project is a collection of related feature flags. In this example, we’ll create a new project called “User Profile” as part of registration. By default, each project gets two environments: <em>Production</em> and <em>Test</em>. To keep things simple for this example, we’ll do all work in the Production environment. But separate environments allow you to test changes to feature flags independently.</p>
<p>Make sure that the <em>Production</em> environment is selected using the dropdown on the top left corner of the sidebar.</p>
<p><img src="https://codeahoy.com/img/feature-flags/initial.png" alt="Unlaunch Console" class="center-image" />
Figure 1 - Unlaunch Console</p>
<p>Next up, create a new feature. Follow the screenshot below. Here, we create a feature flag with two variations: <em>on</em> and <em>off</em>. In our code, we’ll check this flag. If <em>on</em> variation is returned, we’ll show the new feature (i.e. call the new method we just implemented.) Otherwise, we’ll hide the feature (i.e. use the old code.)</p>
<p><img src="https://codeahoy.com/img/feature-flags/createflag.png" alt="new feature flag" class="center-image" />
Figure 2 - Create a new flag</p>
<p>Next, we’ll enable the feature flag, that is, serve the “on” variation to only a single user for now. For everyone, else, we’ll serve the “off” variation. Please make sure your setup looks similar to the screenshot below. Also, don’t forget to enable the flag by clicking the green “Enable Flag” button.</p>
<p><img src="https://codeahoy.com/img/feature-flags/flagsetup.png" alt="Turn the feature flag on for one user id" class="center-image" />
Figure 3 - Turn the feature flag on for user id <code class="language-plaintext highlighter-rouge">umer@gmail.com</code> only</p>
<p>At this point, the flag setup is complete. Let’s take a look at how we’d set up the SDK to call this flag and show the new feature.</p>
<h3 id="2-call-feature-flag-in-your-application">2. Call Feature Flag in Your Application</h3>
<p>Now, let’s use this feature flag in our Spring Boot service to determine whether to use new or old algorithm.</p>
<p>Note: If you are looking for a plain Java example, please see to <a href="https://github.com/unlaunch/hello-java">this repo</a>.</p>
<p>First, add the Unlaunch Java SDK dependency in your <code class="language-plaintext highlighter-rouge">pom.xml</code> (or <code class="language-plaintext highlighter-rouge">build.gradle</code>)</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>io.unlaunch.sdk<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>unlaunch-java-sdk<span class="nt"></artifactId></span>
<span class="nt"><version></span>0.0.8<span class="nt"></version></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<p>Next, you’ll need to provide the SDK key to connect to your Unlaunch project and environment. You can find the SDK key for your environment by clicking <em>Settings</em> in the sidebar. Choose the <strong>Server Key</strong> for the <em>Production</em> environment.</p>
<p><img src="https://codeahoy.com/img/feature-flags/key.png" alt="new feature flag" class="center-image" /></p>
<p>You can paste the SDK key in your <code class="language-plaintext highlighter-rouge">application.yaml</code> or make it available as an environment variable <code class="language-plaintext highlighter-rouge">export UNLAUNCH_SERVER_KEY=<paste server key here></code>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">unlaunch</span><span class="pi">:</span>
<span class="na">server</span><span class="pi">:</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">PASTE_SERVER_KEY_HERE_FROM_SETTINGS</span>
</code></pre></div></div>
<h4 id="2a-initialize-the-unlaunch-client">2a. Initialize the Unlaunch Client</h4>
<p>Next, we’ll create and initialize the <a href="https://javadoc.io/doc/io.unlaunch.sdk/unlaunch-java-sdk/latest/io/unlaunch/UnlaunchClient.html">UnlaunchClient</a> as a bean. When you initialize <a href="https://javadoc.io/doc/io.unlaunch.sdk/unlaunch-java-sdk/latest/io/unlaunch/UnlaunchClient.html">UnlaunchClient</a>, it will download all feature flags in memory. All evaluations will be done against in memory data store (e.g. HashMap) and are really fast.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@org</span><span class="o">.</span><span class="na">springframework</span><span class="o">.</span><span class="na">context</span><span class="o">.</span><span class="na">annotation</span><span class="o">.</span><span class="na">Configuration</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Configuration</span> <span class="o">{</span>
<span class="nd">@Value</span><span class="o">(</span><span class="s">"${unlaunch.server.key}"</span><span class="o">)</span>
<span class="kd">private</span> <span class="nc">String</span> <span class="n">sdkKey</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Logger</span> <span class="n">logger</span> <span class="o">=</span> <span class="nc">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="nc">Configuration</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="nd">@Bean</span>
<span class="kd">public</span> <span class="nc">UnlaunchClient</span> <span class="nf">unlaunchClient</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">UnlaunchClient</span> <span class="n">client</span> <span class="o">=</span> <span class="nc">UnlaunchClient</span><span class="o">.</span><span class="na">builder</span><span class="o">().</span>
<span class="n">sdkKey</span><span class="o">(</span><span class="n">sdkKey</span><span class="o">).</span>
<span class="n">pollingInterval</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">).</span>
<span class="n">eventsFlushInterval</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">).</span>
<span class="n">eventsQueueSize</span><span class="o">(</span><span class="mi">500</span><span class="o">).</span>
<span class="n">metricsFlushInterval</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="nc">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">).</span>
<span class="n">metricsQueueSize</span><span class="o">(</span><span class="mi">100</span><span class="o">).</span>
<span class="n">build</span><span class="o">();</span>
<span class="k">try</span> <span class="o">{</span>
<span class="n">client</span><span class="o">.</span><span class="na">awaitUntilReady</span><span class="o">(</span><span class="mi">2</span><span class="o">,</span> <span class="nc">TimeUnit</span><span class="o">.</span><span class="na">SECONDS</span><span class="o">);</span>
<span class="n">logger</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"unlaunch client is ready"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">InterruptedException</span> <span class="o">|</span> <span class="nc">TimeoutException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">"client wasn't ready "</span> <span class="o">+</span> <span class="n">e</span><span class="o">.</span><span class="na">getMessage</span><span class="o">());</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">client</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h4 id="2b-evaluate-the-flag-to-choose-new-or-old-algorithm">2b. Evaluate the Flag to Choose New or Old Algorithm</h4>
<p>Then in the <code class="language-plaintext highlighter-rouge">UserService</code> class, let’s decide whether to use the new algorithm or not.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Service</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">UserService</span> <span class="o">{</span>
<span class="nd">@Autowired</span>
<span class="kd">private</span> <span class="nc">UnlaunchClient</span> <span class="n">unlaunchClient</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">User</span> <span class="nf">getUserById</span><span class="o">(</span><span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Evaluate the feature flag</span>
<span class="nc">String</span> <span class="n">variation</span><span class="o">=</span> <span class="n">unlaunchClient</span><span class="o">.</span><span class="na">getVariation</span><span class="o">(</span><span class="s">"implement-async-calls"</span><span class="o">,</span> <span class="n">id</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">variation</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"on"</span><span class="o">))</span> <span class="o">{</span>
<span class="c1">// Call the New algorithm if the flag returns: on</span>
<span class="k">return</span> <span class="nf">newAlgorithm</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="c1">// Call the Old algorithm if flag returns: off</span>
<span class="k">return</span> <span class="nf">oldAlgorithm</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>One note: The <code class="language-plaintext highlighter-rouge">getVariation(...)</code> method will <em>never</em> throw an exception or return <code class="language-plaintext highlighter-rouge">null</code>. If there is an error such as if the flag is not found or the initial sync fails, it will return a special String value: <strong>control</strong>. So you don’t have to worry about adding any additional error or checks around this method and can safely use it everywhere.</p>
<h3 id="3-try-it-out">3. Try it out</h3>
<p>You can find the complete source code for this project on <a href="https://github.com/unlaunch/springboot-example">GitHub</a>. You can clone and run it locally using <code class="language-plaintext highlighter-rouge">mvn spring-boot:run</code> or (<code class="language-plaintext highlighter-rouge">gradlew bootRun</code>).</p>
<p>To test the feature flag, call the API with any user id e.g. <code class="language-plaintext highlighter-rouge">kelly@gmail.com</code>. Because we disabled the flag for all users except <code class="language-plaintext highlighter-rouge">umer@gmail.com</code>, the old algorithm will return. <code class="language-plaintext highlighter-rouge">http://localhost:8085/api/v1/users/kelly@gmail.com</code></p>
<p><img src="https://codeahoy.com/img/feature-flags/output-kelly.png" alt="new feature flag" class="center-image" /></p>
<p>Now let’s call the same API with <code class="language-plaintext highlighter-rouge">umer@gmail.com</code> which should return the <em>on</em> variation and hence the new algorithm. (See Figure 3.) <code class="language-plaintext highlighter-rouge">http://localhost:8085/api/v1/users/umer@codeahoy.com</code></p>
<p><img src="https://codeahoy.com/img/feature-flags/output-umer.png" alt="new feature flag" class="center-image" /></p>
<p>That’s all. If you enjoyed this post, please share it.</p>
<ul>
<li><a href="https://github.com/unlaunch/springboot-example">Complete Source Code on GitHub - Spring Boot</a></li>
<li><a href="https://github.com/unlaunch/hello-java">Source Code on GitHub - Plain Java</a></li>
</ul>
COVID-19 - Remote Work Policy by Companies2020-03-15T00:00:00+00:00https://codeahoy.com/2020/03/15/cvoid-19-wfh-companies<div class="d-flex ca_margin_t_xl justify-content-between">
<!-- Share buttons -->
<div class="sharethis-inline-share-buttons"></div>
</div>
<p><br />
In response to Coronavirus outbreak (COVID-19), many companies, including Amazon, Apple, Facebook, Google, Salesforce, and more, have been encouraging those workers who can do so to work from home. Some companies have even made it mandatory for employees to work remotely, shutting down their offices. A surprising number of companies have no plans to follow suit.</p>
<p>We have created these trackers to track remote work policy by employer and to call those employers out who aren’t allowing their employees remote work despite their employees being capable of performing their job duties remotely.</p>
<h3 id="list-of-remote-work-policy-by-country">List of remote work policy by country</h3>
<h4 id="1-usa"><a href="/general/cvoid-19-wfh-companies-list-usa.html">1. USA</a></h4>
<h4 id="2-canada"><a href="/general/cvoid-19-wfh-companies-list-canada.html">2. Canada</a></h4>
<h4 id="3-india"><a href="/general/cvoid-19-wfh-companies-list-india.html">3. India</a></h4>
<p><em>Date: March 15 - March 17, 2020</em></p>
<p>You can help by filling out a completely <strong><a href="https://forms.gle/vtYCjuspqX8N3PEN7">anonymous survey</a></strong> to track companies, not just in the US but around the world, and their remote work policies.</p>
<h3 id="about-this-data">About this data</h3>
<p>The list is generated using reviews submitted by CodeAhoy readers. You’re free to use this data
under the terms of <strong><a href="https://creativecommons.org/licenses/by-sa/4.0/">cc-by-sa 4.0</a></strong> with attribution required. A link to codeahoy.com will be sufficient.</p>
<p>If you’re an official from the company listed below and the information isn’t accurate or the policy has been updated, please email us at: <em>name_of_this_blog</em>@gmail.com or message us on <a href="https://twitter.com/codeahoy">Twitter</a>.</p>
<p>(Fyi, <em>name_of_this_blog</em> = codeahoy)</p>
<div class="d-flex ca_margin_t_xl justify-content-between">
<!-- Share buttons -->
<div class="sharethis-inline-share-buttons"></div>
</div>
COVID-19 Hiring Freeze - Company List2020-03-15T00:00:00+00:00https://codeahoy.com/2020/03/15/covid-19-hiring-freeze<div class="d-flex ca_margin_t_xl justify-content-between">
<!-- Share buttons -->
<div class="sharethis-inline-share-buttons"></div>
</div>
<p><br /></p>
<p>Coronavirus disease (COVID-19) is adding a lot of uncertainty and anxiety for millions of people. Events, left, right, and center and being cancelled. Markets are being very volatile and stocks are dropping sharply.</p>
<p>Several companies have initiated hiring freezes. Some have frozen hiring for all positions while others have put freeze in place only for non-essential positions. Millions of people especially those who are on the job market will be affected by this. However, we at CodeAhoy believe that once this is over, there is a strong possibility that markets will jump back up and there will be a <strong>hiring frenzy</strong>.</p>
<p>To track hiring freezes across companies all over the world, we are asking our users to help build a list of companies that have put a freeze on hiring.</p>
<h4 id="contribute-to-this-list-httpsformsglexeog7lvysd2h7wqa9">Contribute to this list: <a href="https://forms.gle/Xeog7LvySd2H7WQA9">https://forms.gle/Xeog7LvySd2H7WQA9</a></h4>
<p>This list was generated by reviews submitted by the users of CodeAhoy.com</p>
<script>
function myFunction(rand_int1 ) {
var dots = document.getElementById("dots-" + rand_int1 );
var moreText = document.getElementById("more-"+ rand_int1);
var btnText = document.getElementById("myBtn-"+ rand_int1);
if (dots.style.display === "none") {
dots.style.display = "inline";
btnText.innerHTML = "Show More >";
moreText.style.display = "none";
} else {
dots.style.display = "none";
btnText.innerHTML = "Show Less";
moreText.style.display = "inline";
}
}
</script>
<table class="table table-bordered table-hover table-condensed">
<colgroup>
<col span="1" style="width: 23%;" />
<col span="1" style="width: 25%;" />
<col span="1" style="width: 12%;" />
<col span="1" style="width: 37%;" />
</colgroup>
<thead><tr>
<th title="Field #1">Company Name</th>
<th title="Field #2">Official Policy</th>
<th title="Field #3">Country</th>
<th title="Field #6">Comments</th>
</tr></thead>
<tbody><tr><td> Amazon</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Apple</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">Apple is still hiring. No communication on freeze</div></div></td></tr>
<tr><td> Atlassian</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">Not an employee but just got booked for an interview</div></div></td></tr>
<tr><td> Boeing</td> <td> Total hiring freeze </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Capital One</td> <td> No </td> <td>US</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">Summer intern 2020</div></div></td></tr>
<tr><td> Citizen App</td> <td> Partial freeze on some roles </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Google</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Heap</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> IBM</td> <td> No </td> <td>Canada</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Informa</td> <td> Partial freeze on some roles </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">All hiring needs to be approved by ceo and a special committee</div></div></td></tr>
<tr><td> Microsoft</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Nordstrom</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Reonomy</td> <td> Total hiring freeze </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Sezzle</td> <td> Total hiring freeze </td> <td>Usa</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Skyscanner</td> <td> Total hiring freeze </td> <td>Austria</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> SpotOn</td> <td> Total hiring freeze </td> <td>usa</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Starbucks</td> <td> Total hiring freeze </td> <td>Canada</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">Last week I was offered a position and set up a time to begin orientation. Last night I received a <span id="dots-84354417">...</span><span class="more" id="more-84354417">call from that same store manager advising that while the position is still mine they are freezing the hiring process for the time being as per instruction from Starbucks corporate.</span><small><span class="ca_btn_link_no_padding" onclick="myFunction(84354417)" id="myBtn-84354417">Show More ></span></small></div></div></td></tr>
<tr><td> TripAdvisor</td> <td> Partial freeze on some roles </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Value Village</td> <td> Partial freeze on some roles </td> <td>Canada</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
<tr><td> Walgreens</td> <td> No </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">Walgreens is currently hiring full time, part time, and temporary CSAs, Pharmacy Techs, Shift Leads, and Beauty Consultants</div></div></td></tr>
<tr><td> Yelp</td> <td> Partial freeze on some roles </td> <td>USA</td> <td><div class="d-flex flex-column"><div class="ca_empty"><small class="text-muted">Current Employee - </small> </div><div class="mt-1">-</div></div></td></tr>
</tbody></table>
<div class="d-flex ca_margin_t_xl justify-content-between">
<!-- Share buttons -->
<div class="sharethis-inline-share-buttons"></div>
</div>
Tech Debt Developer Survey Results 2020 - Impact on Retention2020-02-17T00:00:00+00:00https://codeahoy.com/2020/02/17/technical-debt-survey<p>Last month, I wrote a blog post called <a href="https://codeahoy.com/2020/01/25/technical-debt/">Technical Debt is Soul-crushing</a>. In it, I discussed the effects of tech debt on software developers and how it makes them unhappy. I wrote it because, being in management for several years now, I have seen how people discuss tech debt as a product or engineering problem. They completely overlook or ignore its impact on people.</p>
<p>For software developers, it is very frustrating to work on a codebase that has a high amount of tech debt. They feel unproductive and handicapped. This creates an atmosphere where people start thinking about leaving and they do soon as they find a better option.</p>
<p>Because the blog post was getting good amount of traffic (10k views in a week), I decided to add a survey. The survey is closed and the results are in.</p>
<h3 id="68-of-developers-said-they-work-on-products-with-high-or-very-high-amounts-of-tech-debt">68% of Developers Said They Work on Products with High or Very High Amounts of Tech Debt</h3>
<p>No large software is tech debt free. At least, I haven’t come across one in my life. Some have more, others have less.</p>
<p>Not a single person said that their product contains ‘No tech debt’.</p>
<p><img src="https://codeahoy.com/img/tech-debt-survey/howmuch.jpeg" alt="tech_debt_amount_in_software_systems" class="center-image" /></p>
<hr />
<h3 id="50-of-developers-are-likely-or-very-likely-to-leave-their-jobs-because-of-tech-debt">50% of Developers Are Likely or ‘Very Likely’ to Leave Their Jobs Because of Tech Debt</h3>
<p>27% percent indicated that they think about it, but aren’t sure.</p>
<p><img src="https://codeahoy.com/img/tech-debt-survey/leave_employer.jpeg" alt="Tech_Debt_Leave_Employer" class="center-image" /></p>
<hr />
<h3 id="question-is-your-management-aware-of-tech-debt-and-are-they-taking-action-to-pay-it-off">Question: Is Your Management Aware of Tech Debt and Are They Taking Action to Pay It Off?</h3>
<p>I asked this question to see if there’s a correlation between developer dissatisfaction and management or leadership being aware of the problem and taking action to pay it off. Here are the results.</p>
<p><img src="https://codeahoy.com/img/tech-debt-survey/managementaware.jpeg" alt="Tech_Debt_Management_Awareness" class="center-image" /></p>
<hr />
<h3 id="question-how-long-have-you-been-working-on-the-product-with-tech-debt">Question: How Long Have You Been Working on the Product with Tech Debt?</h3>
<p>Here are the results.</p>
<p><img src="https://codeahoy.com/img/tech-debt-survey/howlong.jpeg" alt="Tech_Debt_Years_Retention" class="center-image" /></p>
<hr />
<h3 id="survey-methodology">Survey Methodology</h3>
<p><strong>117 software developers</strong> from all over the world took the survey. The majority were from the USA, followed by Canada, Australia, Germany, India, Russia , UK and other European countries.</p>
<p>The survey ran in February 2020.</p>
<p>91 software developers took the web survey. 26 respondents were from my personal contacts. Senior or Lead software developers mainly from the USA and Canada who use Java. I limited my contacts as not to introduce sample selection bias.</p>
<hr />
<p>Before I continue further and look at some comments from the survey, if you enjoyed this post, please <strong>share</strong> it on Twitter, Facebook and other sites. It helps me grow. The links to share are at the bottom.</p>
<h3 id="types-of-tech-debt">Types of Tech Debt</h3>
<p>I asked survey participants to identify the type of tech debt present in their project. Here are some of their <strong>responses</strong>.</p>
<ul>
<li>Poor code, outdated libraries, lack of documentation</li>
<li>Outdated libraries, a mix of design patterns, poorly designed code, obviously-slower-than-it-should-be performance</li>
<li>bloated monolith - lack of processes, no time allocated for reducing tech debt - lack of documentation, onboarding is not trivial - low amounts of automated testing - compiler warnings mostly ignored - static analysis tools not included in the dev process. running such tools on the codebase produce non-trivial warnings (that are not related to readability)</li>
<li>Poor code, bad design choices, lack of unit tests, no documentation, haphazard architecture</li>
<li>Outdated</li>
<li>no design no docs no tests no separation to concerns no competence</li>
<li>legacy code base</li>
<li>Poor design choices as the product evolved, mostly from a technical perspective dealing with the divide/chasm between frontend & backend, where and how is the data being transmitted.</li>
<li>poor coding practices, outdated libraries (largely from silly backwards compatibility requirements)</li>
<li>Monolithic processes, lack of documentation, inconsistencies across similar parts of the product, lack of shared code leading to mass code duplication</li>
<li>tech debt helps me keep my job</li>
<li>Poor code & design choices, outdated libraries, barely maintained logging & monitoring solutions, lack of documentation, terrible deployment strategy, lack of tests (especially integration tests)</li>
<li>Large monolith, lack of continuous delivery, vague processes and no central authority for organizing deployments of the monolith, consistent failure to investment in hiring enough people to support the work</li>
<li>Design docs written without background research, some design decisions made without a design doc</li>
<li>Zero tests, no documentation, massive code duplication, hardcoded IDs in code as hacks</li>
<li>Inconsistent use of designs/patterns across code base. Lacking alignment to broader architectural goals.</li>
<li>monolith, not implemented design principles (especially SRP and OCP from SOLID), outdated libraries, lack of process, lack of documentation</li>
<li>Bloated monolith, cowboy programming methods, hard-coded stuff everywhere, outdated programming languages and frameworks, lack of process</li>
<li>outdate languages and frameworks, monolithic structure, coupling between components that should not be connected</li>
<li>Building an in-house UI framework built on another external framework that’s not supported anymore.</li>
<li>bloated monolith, poor code or design choices, outdated libraries - terraform, lack of processes or documentation, Left over AWS infrastructure for services not used.</li>
<li>in-house frameworks flaky tests and code (sic!) unstable test hardware (the setup is done poorly), often redesigning code that wasn’t even updated to the previous design</li>
<li>Bloated monolith, poor code and design choices, far too much “duct tape” accumulated over decades.</li>
<li>lack of documentation, lots of invalid inconsistent data in the database, lots of code which is (presumably) not used anymore with no easy way to be sure.</li>
<li>Framework built 10 years ago in house</li>
<li>Poor abstractions. Architecture requiring substantial boilerplate.</li>
<li>Complicated event & callback system, varying programming styles, lack or processes.</li>
<li>lack of processes or documentation, poor SE practices, no code review</li>
</ul>
<hr />
<h3 id="other-comments-from-the-survey">Other Comments from the Survey</h3>
<p>I asked readers to leave comments on the survey. Here are some of their comments.</p>
<ul>
<li>Management actively acknowledges it and we even prioritize among technical debt, but <strong>the technical debt items never make it into work streams</strong> because it’s never seen as important enough.</li>
<li>I actually left because of their inabiiity to make a choice</li>
<li>Senior developers are the problem. They added tech debt because it is their <em>job security</em></li>
<li>it’s hard to tackle the tech debt when most of the engineers are not very good. they simply don’t know how to write better code, and don’t always see the virtue in better designs when they’re presented. they also don’t know their tools very well (git rebasing and automated refactoring are foreign to them).</li>
<li>One of the curious things I’ve noticed is that our company is struggling financially, which means addressing tech debt is de-prioritized over new features that might bring in more revenue. However, I think a lot of our issues are related to our tech debt (such as systems being degraded over the weekend without us noticing due to poor monitoring). For whatever reason, new features keeps winning over fixing the issues that cost us users.</li>
<li>Old services and endpoints are shut down when no longer needed; logging integration APIs record actual production usage.</li>
<li>Automated tools help find dead code and refactor, but doesn’t write documentation</li>
<li>I consider tech debt a challenge - I have not been in the project since the beginning, but I do believe we can solve (maybe without going too much in the direction of your other article “overhours”)</li>
<li>Upper management is very aware of the crippling technical debt load our primary product contains, as it leads directly to numerous bugs – typically found by customers – that are very difficult to track down and fix. However, the immediate dev management doesn’t see that there’s a problem, so nothing is being done to rectify the issue.</li>
<li>Product features take priority and management is clueless how to fix it</li>
</ul>
<p>Here are some comments I liked from other sites.</p>
<blockquote>
<p>The system I’m working on is indeed <strong>soul-crushingly</strong> burdened. Everyone knows this: there’s a survey every year and “crushing technical debt” is always the <strong>top complaint</strong>. I mention it to the <strong>bosses</strong>, and they go “yeah, we know.” I say <strong>we never seem to schedule time to fix it</strong>, and I’m told “you’re not in the same meetings we are.” Yeah, but I’m the one that would be fixing it, if we were fixing it, eh?</p>
<p>Now I’ve started actually dealing with the management by treating the debt like it’s impactful.</p>
<p>“How long will it take to do X?” “I can’t predict. The thing that should have taken a day last time took three weeks.”</p>
<p>“Why did the system break?” “Shitty architecture from the start mandated making changes that are impossible to test, and our tests are so flakey it’s impossible to tell whether anything’s broken, so we pushed it out to prod and waited to see who complained about which feature.”</p>
<p>Without something like that, management never actually feels the pain, never understands that it’s not just something hard they’re asking for, but an interlocking tangle of shortcuts that make things 100x as hard to accomplish as they should be.</p>
</blockquote>
<blockquote>
<p>Oh, I understand this so much. . I am A data analyst who does a lot of programming. We automate Excel spreadsheets and databases tools and stuff using VBA and Python and SQL. The guy before me was not a programmer at all as is often the case in people who develop automated Excel spreadsheets with VBA. Every tool that the client has asked me to update is so horrible that I can’t even Trace through the code To figure out where I can insert a feature. It’s like a puzzle that has no logical solution.</p>
<p>Some of the code has variables that are sprinkled all over that have the same name and then he just repurposes them with new names. I swear all over the code There are blocks that look like this:</p>
<p>A=1</p>
<p>B=A</p>
<p>C=B</p>
<p>Instead of using modular functions and subroutines, every single function is repeated over and over with small modifications made. Because of that the code base is probably 30 times longer than it should be.</p>
<p>The worst part is throughout the code there are comments saying this should be fixed or that should be made better, but it was never done.</p>
<p>I like my job for several reasons, but this code I have to work with causes me to think about <strong>quitting</strong> probably every month or two.</p>
</blockquote>
Portal Theme and Blog Redesign2020-02-15T00:00:00+00:00https://codeahoy.com/2020/02/15/portal-jekyll-theme<p>CodeAhoy website has been redesigned<em>!</em></p>
<p>When I launched CodeAhoy in 2016, I wanted to keep things simple and clean. My previous blog on Wordpress had become a cluttered mess and was really slow, especially on mobile devices. I was also bored with Wordpress and wanted to try something different.</p>
<p>Then I stumbled upon <a href="https://jekyllrb.com/" rel="nofollow">Jekyll</a>. For those of you who aren’t familiar with it, Jekyll is a super easy static site generator for blogs. It takes your markdown files (and some configuration) and spits out a complete HTML website. The icing on the cake was that GitHub Pages supported Jekyll as one of the blog engines, which makes sense because Jekyll is written by one of the <a href="https://en.wikipedia.org/wiki/Tom_Preston-Werner" rel="nofollow">co-founders</a> of GitHub.</p>
<p>So it was a no brainer. I chose Jekyll.</p>
<p>The next thing I needed was a <strong>theme</strong>. Jekyll didn’t have the same number and variety of themes and plugins available like Wordpress. But I did end up finding a <a href="https://github.com/poole/hyde" rel="nofollow">theme that I liked</a>. The theme itself was pretty simple, <strong>two-column</strong> layout. One column for the left sidebar, which always stayed in place, and the second displayed the main content, i.e., the blog post. It was clean, and did I already mention simple?</p>
<p>After applying the theme and releasing my blog, this was the end result.</p>
<p><img src="https://codeahoy.com/img/screenshot-old-blog.jpg" alt="old_blog_screenshot" class="center-image" /></p>
<p>I liked it because of the focus on readability. But over time, my affinity for the theme was reduced to a glimmer. What did I not like about the design? Well for starters, it <strong>wasn’t mobile-friendly</strong>. (Or may, it got so after my tweaks to the CSS… I don’t remember.) On mobile devices, the sidebar moves to the top of the page, taking up half the screen real-estate.</p>
<p>The second challenge I had once I had a good number of posts on my blog was that there was no room to display meta content like new or related blog posts and ads on the page. I was not too fond of the idea of embedding these things in the blog post itself. What I wanted was a thin right sidebar to display these things. I made up my mind that I’ll change the layout but got busy with other things so the idea collected dust.</p>
<p>Finally, I decided it is time to revamp the design finally. It was beginning to look pretty ancient and mobile usability was just horrible. So on the President’s Day long weekend, I made it my goal to get it done finally. My requirements from the new design were:</p>
<ul>
<li>mobile-first</li>
<li>keep the focus on readability. Put content first.</li>
<li>Less cruft for improved page load speeds.</li>
<li>For blog posts, I wanted a layout that has: a top navbar, and two-column layout for the main area. The left column will be much bigger and I will use it to show blog posts. The smaller right column is where I’d display meta content like features posts etc.</li>
</ul>
<p>With the new requirements in mind, I set off to find the right Jekyll theme. I imagined that after so many years and improved Jekyll adoption courtesy of GitHub, I’d have better luck at finding the right theme quickly. I was wrong.</p>
<p>I searched for many hours. I looked for both paid and unpaid themes. I couldn’t find anything that met my criteria. I was contemplating just giving up and to build my own theme from scratch (Bootstrap,) I decided to go through my shortlist of candidates. On second look, a Medium-styled theme called <a href="https://github.com/wowthemesnet/mundana-theme-jekyll" rel="nofollow">Mundana</a> really stood out. The design was clean and very close to what I wanted. It was missing a right sidebar that I wanted, plus it had some cruft that I wanted to get it out. I also didn’t like animations and transitions so much. I checked its license, and since it was MIT, I made up my mind to tweak it to meet my requirements.</p>
<p>Plug. Mundana is built by <a href="https://www.wowthemes.net/" rel="nofollow">WowThemes</a>. These guys have some really cool Jekyll (and Wordpress) theme. Check them out.</p>
<p>Back to the redesign story. Actually, I’ll just give you a summary: it has been a while since I used Bootstrap, which I didn’t realize when I thought I’ll be able to finish the redesign in a day. It ended up taking twice as long and I’m still not done. Thankfully, I remembered how <a href="https://developer.mozilla.org/en-US/docs/Learn/CSS/CSS_layout/Flexbox" rel="nofollow">flexbox</a>es work and that was my saving grace. This is what the final result looked like:</p>
<p><img src="https://codeahoy.com/img/blog-post-2-col-layout.jpg" alt="new_blog_screenshot" class="center-image" /></p>
<p>Am I happy with it? Not really. It doesn’t look bad but it can be improved. The Jumbotron displaying the blog post title, author and featured image doesn’t feel right and it is something I plan on fixing next. I also want to improve the theme’s performance by removing cruft like JQuery, and other CSS/JS files I don’t need. I also want to use <code class="language-plaintext highlighter-rouge">async</code> and <code class="language-plaintext highlighter-rouge">defer</code> for JS file to improve first component paint times. I want the navbar to be stick on mobile. A lot of work but I’ll chip away.</p>
<p>I’m making my changes available for free for anyone to use. If you want to contribute, you’re welcome to do so! If you’ve any questions about the theme, feel free to message me (<a href="https://twitter.com/codeahoy">Twitter</a>).</p>
<p>World, meet <strong>Portal</strong>.</p>
<h2 id="portal-jekyll-theme">Portal Jekyll Theme</h2>
<p>Portal is a mobile-first Jekyll theme for technical blogs and websites. The source code is available at: <a href="https://github.com/umermansoor/Portal-Jekyll-Theme">https://github.com/umermansoor/Portal-Jekyll-Theme</a></p>
<h3 id="features">Features</h3>
<ul>
<li>Modified Boostrap 4.1 theme</li>
<li>Complete Jekyll integration</li>
<li>About Me and Author pages</li>
<li>Sitemap, Feed and Atom</li>
<li>robots.txt</li>
<li>Easy mailchimp integration for newsletter sign-ups</li>
<li>Easy Disqus integration</li>
</ul>
<h3 id="how-to-use">How to use</h3>
<p>These instructions assume you have Jekyll installed on your machine.</p>
<ol>
<li>Clone the git repo</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/umermansoor/Portal-Jekyll-Theme.git
</code></pre></div></div>
<ol>
<li>Start the theme with sample blog contents</li>
</ol>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jekyll serve --watch
</code></pre></div></div>
<p>If everything works as expected, you should be able to see it by visiting <code class="language-plaintext highlighter-rouge">http://127.0.0.1:4000/index.html</code>. I have added some sample blog posts and content to illustrate the design.</p>
Technical Debt is Soul-crushing2020-01-25T00:00:00+00:00https://codeahoy.com/2020/01/25/technical-debt<p>Technical debt is incurred when the software or system designers take <strong>shortcuts</strong> to ship a feature faster, increasing the overall complexity of the system. The goal is to optimize the present rather than the future. In other words, it’s the easier path that takes us to the end-goal faster, but the resulting code (or design) is messy and complicated. It will require extra time in the future to add new features or to fix bugs.</p>
<p>The most common reason why companies take on technical debt is to meet the time to market demands. “We must release this feature by February, or our revenue will take a big hit. Just hack it for now, and we’ll fix it later.” Other reasons for incurring tech debt include lousy design choices, poor programming, changing requirements, or the presence of outdated libraries or frameworks that made sense in the past but have become a liability now.</p>
<p>Technical debt is <em>not</em> always a bad thing. It can help companies ship a critical feature fast and acquire users more quickly than its competition. My first job was at a startup. We intentionally took on tech debt because a) what we were doing was risky, b) we had a tight deadline to meet or the company would run out of money - no point in writing perfect code if it wasn’t going to be ever released. Our tech debt wasn’t the opposite of <a href="https://codeahoy.com/2017/08/19/yagni-cargo-cult-and-overengineering-the-planes-wont-land-just-because-you-built-a-runway-in-your-backyard/">over-engineering</a>. It was an intentional compromise to get the product out of the door on time. We understood that we’d have to pay the debt off or it will make it difficult to maintain and grow the system in the future.</p>
<!--more-->
<h2 id="tech-debt-is-demoralizing-for-software-developers">Tech Debt is Demoralizing for Software Developers</h2>
<p>The problem starts when companies forget to pay off the debt and let it creep and pile up for an extended period. The past comes to haunt the present. For good software developers, it is totally <strong>demoralizing</strong> to work on products that have high tech debt. This aspect isn’t often talked about, but its effects are very real. Simple things like changing a title tag of a webpage page take up a whole day because the logic was scattered in five different files. At the end of the day, it’s not a great feeling that it took so much time for a small task. It’s even more upsetting when they have to explain it to their managers, colleagues, or the product team why it took so long. Troubleshooting a bug is not just difficult but also painful. Jeff Atwood called it a <strong><a href="https://blog.codinghorror.com/paying-down-your-technical-debt/">major disincentive</a></strong> to work on a project:</p>
<blockquote>
<p>Beyond what Steve describes here, I’d also argue that accumulated technical debt becomes a major disincentive to work on a project. It’s a collection of small but annoying things that you have to deal with every time you sit down to write code. But it’s exactly these small annoyances, this sand grinding away in the gears of your workday, that eventually causes you to stop enjoying the project.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/tech-debt-dev.jpeg" alt="tech_debt_human" class="center-image" /></p>
<p>I’ll add that it is even more frustrating if it is a large project with many other developers working on it. Imagine finally finishing a feature only to find that it is not working on the QA or Stage environment because another developer from a different team changed something which somehow broke the feature you just completed and you both have no idea how to get out of the mess. It can also fester and promote intra-team conflicts and dissatisfaction.</p>
<p>In addition to just being frustrating, good software developers have a burning desire to keep their skills up-to-date. Systems with high tech debt are very difficult to work with, much less keep up to date. As a result, developers are stuck with libraries and frameworks that are many years old. They fear they will break something, so they avoid upgrading it altogether.</p>
<h2 id="cost-of-tech-debt-employee-turnover">Cost of Tech Debt: Employee Turnover</h2>
<p>The real cost of all this is turnover. Good developers leave when they believe it is going nowhere. There is no sense of accomplishment, just frustration. There is worrying about the future since they aren’t learning anything new or acquiring new skills. They leave when they find a better opportunity, leaving behind those who are at peace with taking shortcuts or stuck working on the crummy project.</p>
<p>The people who are left behind do not voice their concerns as vocally as those who left and they continue increasing the tech debt. They have learned to survive with the beast and are at peace, taking the <em>quick-and-dirty</em> path all the time. One could also argue that the messy codebase encourages further poor practices and software rot owing to the <strong><a href="https://codeahoy.com/2016/05/02/software-rot-entropy-and-the-broken-window-theory/">broken window theory</a></strong>.</p>
<p>I’d argue that for organizations with high tech debt, it is more expensive to replace software developers than those with a lower amount of debt. When someone leave, they end up taking a chunk of tribal knowledge of code and processes with them. New hires take a long time to understand and get up to speed. It takes them a long time before they become productive.</p>
<h2 id="how-to-fix-it">How to fix it?</h2>
<p>Good leaders and managers must evaluate tech debt regularly and take proactive steps to keep it under control. The worst thing senior technical leadership can do is deny the existence or effects of tech debt if it is present. The second worst thing to do is to <em>ignore</em> it and take no action if an action is needed. Those who do this are being short-sighted because they will soon be drowning in problems (if they stay.)</p>
<p>Let’s look at some <strong>strategies to paying off the debt</strong>. This is in no way a comprehensive list and based on my personal experiences.</p>
<p>A common strategy that I have seen work is setting aside a percentage of time each quarter or sprint to gradually pay off tech debt along with new feature development. Anywhere from 15% to 30% is reasonable, depending on how bad it is. This strategy works great if the tech debt isn’t out of control.</p>
<p>If the debt is out of control, then it must be attacked head-on and prioritized. A good strategy is to acknowledge the problem and let the software team know that it will be addressed. For large or medium organizations, it is important that all senior leaders are on board. For example, if the product or the marketing team don’t see the value or believe in it, they’ll continue pushing for new features that will result in a poor dynamic and make it fail. The debt metaphor works really well on non-technical people and helps them understand the problem. After the buy-in across the board, invite senior engineers to come up with a solution.</p>
<ol>
<li>Start by identifying tech debt in your system or code. Resist the urge to define solutions at this stage. Create a shared spreadsheet identifying classes, methods, entire modules or services, use cases, dead feature flags or A/B tests, etc. in your code.</li>
<li>Go through the items that are identified above and extract common themes. Prioritize these themes such that most painful ones are taken care of first. Define clear goals. The goal shouldn’t be to pay off all tech debt but to achieve a reasonable ratio. You can choose to leave some of it in there or until the next phase.</li>
<li>Discuss steps but avoid the urge to <strong><a href="https://codeahoy.com/2016/04/21/when-to-rewrite-from-scratch-autopsy-of-a-failed-software/">rewrite from scratch</a></strong>. Inexperienced developers love total rewrites. Resist the urge and instead identify incremental improvements.</li>
<li>Be open to retiring systems that aren’t useful or needed anymore. When a system is retired, all of its technical debt is retired with it. At my previous (mobile gaming) company, we had a few very old central services (backend) that had a ton of tech debt and troubleshooting them every now and then was very time-consuming. These services were used by older game clients and had become irrelevant for business purposes. In theory, we could shut these down without any impact. But it wasn’t technically possible because doing so would break the API calls and impact the game and the user (forced updates of game clients wasn’t an option), so we came up with a creative way. We retired these services and replaced them with a new ‘mock’ service that always returned a mock (successful) response.</li>
<li>Assign action items to engineers and teams. Make sure they are empowered and that there is a clear execution plan.</li>
<li>Changing existing habits. This might be a little harder, especially with engineers who have been with the company for a long time and are used to just hacking it in even when it is not required. Coaching helps. Re-iterate the debt minimization vision and promote best practices like writing unit tests etc. for all new features.</li>
<li>Devise a strategy to <em>track</em> new tech debt. An easy and common technique is placing <code class="language-plaintext highlighter-rouge">TODO</code> comments with a description e.g. <code class="language-plaintext highlighter-rouge">TODO - move this logic to XYZ</code> or <code class="language-plaintext highlighter-rouge">TODO: setup and use internationalized string instead of hardcoded</code></li>
<li>Make it visible and a priority. Follow up regularly in check-ins and re-adjust as required.</li>
</ol>
<p>This process is an art rather than science. The most important thing is to identify a path that allows tech debt to be paid off gradually, and also allows new feature development or bug fixes to happen in parallel (albeit with some compromises.) Even the most complex systems can be gradually retired. I’ve found reverse-proxy’ing to be an effective technique in which some requests or parts of requests are redirected to newer services and older services are retired slowly.</p>
<p>If you’re a leader, you must assess the tech debt situation in your organization. Early detection and prevention will go a long way. Don’t ignore it if it has got too big and your developers are complaining and stressed out. It the situation isn’t too bad, schedule time to pay it off periodically. If it is out of control, make it an immediate priority. You might get away with it in the short term, but it will come back to bite you and feature development will slow down to snail’s pace.</p>
<p>I ran a survey to understand the impact of tech debt on employee retention. Please visit <a href="https://codeahoy.com/2020/02/17/technical-debt-survey/">this link to see the results</a>.</p>
Do Software Developers Normally Code on Weekends? Work-life Balance and Overtime in the Tech Industry2019-10-19T00:00:00+00:00https://codeahoy.com/2019/10/19/do-software-developers-work-weekends-work-life-tech<p>Despite all the perks tech companies offer to employees these days (unlimited PTOs, catered lunches, refreshments, beer on tap, dog friendly offices, gyms, yoga classes), there is often an indirect or hidden pressure on software engineers to put in more hours or to code on weekends. Managers say things like:</p>
<ul>
<li>“You can call me on the weekend or even at night if you have any questions. I will answer your phone because I’ll be working anyways.”</li>
<li>“Sam is an amazing engineer. So dedicated and such a hardworker. He couldn’t finish his tasks because he was blocked for two days by the other team so he spent all weekend coding.”</li>
<li>“I’m so proud of my team. They regularly stayed late and worked weekends to deliver on our roadmap.”</li>
<li>“We’ll be using the new technology XYZ for this project. Why don’t you start learning XYZ in your ‘free time’ (aka the weekend)?”</li>
</ul>
<p>Subtle hints like these leave many developers wondering whether working on weekends a normal thing that is expected from us as software engineers.</p>
<p>Which brings us to the <em>question</em>: <strong>“Is it OK to not code for work on the weekends?”</strong></p>
<!--more-->
<p>The <em>answer</em> is: <strong>it is ABSOLUTELY OK to not work on weekends</strong>. You have a life and it’s pefectly OK to enjoy it.</p>
<p>One exception to this rule is if there is a <strong>real emergency</strong>. No, not finishing a story in time for the Sprint Retro doesn’t constitute an emergency. It was probably not estimated correctly and besides, <a href="https://codeahoy.com/2016/05/14/software-estimates-are-not-targets/">estimates are not targets</a>. Emergencies should be <em>rare</em> or there are bigger issues with the team or the company. For example, a while ago, I was handling the launch of a product (a mobile game) that was attached to PR events. It was supposed to go live on Monday. On Friday afternoon before the launch, we discovered a major issue that was going to seriously jeopardize the launch if not fixed. This was a real emergency. The team came into work both on Saturday and Sunday. We fixed the issue just in time for launch and it went super smooth. Next week, the entire team took 2 extra days off and got a 4-day weekend. It was followed by a <a href="https://codeahoy.com/2016/06/20/blameless-postmortems-examining-failure-without-blame/">post-mortem analysis</a> to understand the root cause so we don’t have make the same mistake again.</p>
<h3 id="isitoktolearnnewtechnologiesmyjobrequiresat-work">Is It OK to Learn New Technologies My Job Requires at Work?</h3>
<p>If your work requires you to learn a new technology, it’s quite ok to learn it at work. If you are required to <em>learn</em> it in order to apply it at your job, then by <strong>definition it is work</strong>. This is where most managers either turn a blind eye or encourage employees to learn on their <strong>‘free time’</strong>. If you’re in a situation like this, work with your manager to carve out dedicated time each sprint (or a week, whatever) where you can spend time learning the new technology you’ve been asked to learn. This is not a black or white rule. If you’re really interested in the learning the new technology and want to put in your time because it is good for your personal development and growth, then by all means you should to invest your free time in learning. The point I’m trying to make is that you <strong>should not be pressured</strong> into doing it. Similarly, if you really enjoy coding and want to take your skills to the next level by doing work on weekends, that’s totally fine as long as it is your choice.</p>
<h3 id="how-to-be-effective-at-work">How to be Effective at Work?</h3>
<p>If you don’t want to put overtime, you must become <strong>efficient while at work</strong> and avoid distractions. As an engineer, if you are attending a lot of useless meetings a week, talk to your manager or the person who owns the meeting to see if you really need to be in those meetings. Try to avoid wasting time with people who come to work to <strong>socialize</strong> with others and then stay late to complete their tasks. <strong><a href="https://codeahoy.com/2016/05/28/why-do-developers-love-music-so-much/">Use headphones</a></strong>. If the socializers still don’t get the signal, walk away and tell them you’re busy. Work from home once a day every week where you can focus distraction free. Talk to your management team to implement <strong>no meeting days</strong> where other than daily stand-ups, no other meetings are allowed (this will usually require support from executives if you are working for a large company because it needs to be understood and accepted company wide.)</p>
<h3 id="the-myth-of-overtime">The Myth of Overtime</h3>
<p>Putting in extra hours <strong>does</strong> work in the short term and will get the job done. But it’s not a viable long term strategy. In fact, the effects of overtime either cancel out the progress made in the longer term or leave teams worse off than they were before. <a href="https://www.goodreads.com/book/show/67825.Peopleware">Peopleware</a>, one of my favourite management books of all time, covers this issue very nicely and uses the sprinting analogy: <strong>it’s great for the final stretch but if you do it for too long, you won’t be able to finish the race</strong>. Wise managers use it judiciously and think in terms of benefits divided by costs. The benefit could be that project is delivered on time. The cost will be downtime and decreased productivity or worse, replacing smart people who quit (which is almost never worth it!)</p>
<h3 id="what-about-workaholics">What About Workaholics?</h3>
<p>It’s totally true that some people are workaholics. If you ask them about Work-Life balance, don’t be surprised if you hear <em>‘What’s that?</em> They’ll consistently put in very long hours even when there’s no pressure on them. Under pressure, they’d even sleep at the office. It might be hard to believe for some, but I know regular employees who worked on Christmas holidays and were at the office every single day because they wanted to deliver on their commitments for the quarter.</p>
<p><img src="https://codeahoy.com/img/worked_my_butt_off.jpeg" alt="Workaholic_Software_Engineer" class="center-image" /></p>
<p>But the productivity of workaholics decline after a while. Sooner or later, they’ll realize that life is going by while they were sacrificing it for work. Other people were enjoying it. They’ll become grumpy. Develop health issues. Burn out. Mentally check out or quit.</p>
<p>Chapter 3 of Peopleware, “Vienna waits for you” starts with the following tale:</p>
<blockquote>
<p>Some years ago I was swapping war stories with the manager of a large project in southern California. He began to refate the effect that his project and its crazy hours had had on his staff. There were two divorces that he could trace directly to the overtime his people were putting in, and one of his worker’s kids had gotten into some kind of trouble with drugs, probably because his father had been too busy for parenting during the past year. Finally there had been the nervous breakdown of the test team leader.</p>
<p>As he continued through these horrors, I began to realize that in his own strange way, <strong>the man was bragging</strong>. You might suspect that with another divorce or two and a suicide, the project would have been a complete success, at least in his eyes.</p>
</blockquote>
<p>If you’re an employee and your teammate is a workaholic, don’t feel bad. It doesn’t mean you have to do the same.</p>
<p>If you’re a manager, you need to keep an eye to detect problems early and course-correct. I was talking to a friend who quit his job a couple of months ago. His manager would spend time at the office socializing and then regularly send emails on weekend cc’ing higher ups asking for updates from the team. None of the manager’s bosses stepped in to say “what are you doing?” I’m glad he quit and now is in a place with better culture and working conditions. As a manager, I try to keep a close eye on overtime. My priority is to prevent burn out. There are times when we are running behind it gets tempting to push the team to put in extra hours. But there are options 98% of the time: work with the product team to negotiate the scope to reduce it so it can be released on time. Find other creative solutions. And if nothing else is possible, find out a way to make roadmap and OKR adjustments if necessary and push things out. 98% of the time, it’s not worth doing at the cost of burnout.</p>
<h2 id="a-case-against-workoholics">A Case Against Workoholics?</h2>
<p>I’m in no way suggesting that worholics are bad people. There are times when companies need workoholics to succeed. Especially startups in their infancy. But it’s not a scalable approach. <strong>If your company can only be run by workholics, you’re in trouble.</strong></p>
<p>So it’s not so black and white. If you’re starting out or learning a new technology, it’s totally OK to put in extra hours initially so you could have it easy down the road. I’m not against it at all; This post is about companies that try to promote this culture and pressure employees into following it. That is not OK!</p>
<hr />
<p>If you manager tries to indirectly pressure you into working weekend regularly by saying things like “<em>you can call me on Saturday (or at midnight) if you have any questions, I’ll be there to answer since I’ll be working anyways</em>” you can reply “<em>No, thanks. I don’t work on weekends as a principle.</em>” If that doesn’t work and your manager or peers continue to pressure you, start looking for a new job which offers better work-life balance. Seriously, it is not worth it. Don’t listen to the BS <strong><a href="https://www.businessinsider.com/jeff-bezo-advice-to-amazon-employees-dont-aim-for-work-life-balance-its-a-circle-2018-4">work-life harmony</a></strong> crap and <strong>don’t let corporations control and define your quality of life</strong>.</p>
<p>See you next time.</p>
GraphQL - A Practical Overview and Hands-On Tutorial2019-10-13T00:00:00+00:00https://codeahoy.com/2019/10/13/graphql-practical-tutorial<p>GraphQL is a <strong>Q</strong>uery <strong>L</strong>anguage for APIs. It provides a fresh and <em>modern</em> approach to fetching (and manipulating) data from APIs when compared to traditional methods such as REST. Its power comes from the ability to let clients talk to a single endpoint and specify precisely what data they need. That’s very powerful indeed.</p>
<p>This blog post is a hands-on introduction to GraphQL and its important features. When I first encountered GraphQL (we’re switching public REST APIs to GraphQL,) coming from REST background, I was baffled. ‘Where’s the list of API end-points?’, ‘Is the list of fields documented somewhere?’ So I decided to write this article not as a comprehensive overview that deep-dives into internals, but rather to give an understanding of what GraphQL is and <em>how to use it</em> using real examples. It assumes no previous knowledge of GraphQL. Let’s get started.</p>
<!--more-->
<h2 id="graphql---an-overview">GraphQL - An Overview</h2>
<p>GraphQL - or more specifically - the data query language specification and the runtime for it - was developed by Facebook. It was <em>opened sourced</em> it in <strong>2015</strong>, after a few years of internal use at Facebook. What inspired the need for GraphQL? Here’s an excerpt from the <a href="https://engineering.fb.com/core-data/graphql-a-data-query-language/">Facebook engineering blog</a>:</p>
<blockquote>
<p>As we transitioned to natively implemented models and views, we found ourselves for the first time needing an API data version of News Feed — which up until that point had only been delivered as HTML. We evaluated our options for delivering News Feed data to our mobile apps, including RESTful server resources <em>…</em> We were frustrated with the <strong>differences between the data we wanted to use in our apps and the server queries they required</strong>.</p>
</blockquote>
<p>Once you build and expose a REST API, it’s pretty rigid. For example, suppose we have built a News Feed RESTful API which is returning 10 attributes for each item in the news feed. Down the road, you are building a mobile app for low-tier devices that don’t have the screen real-estate to show the news feed in all its glory. So instead of showing all 10 attributes for each news feed item, you resort to showing just 5. This is <strong>wasteful</strong> because you’re still fetching all 10 attributes. The typical solution would be to go to the RESTful API development (backend) team and ask them to make the fields you are not using optional. But this is not straight-forward because there are other clients out there and you might introduce a breaking change. They could introduce a new version or do something else. But the point I’m trying to illustrate is that the process is not free of friction and is inefficient.</p>
<p>GraphQL solves this exact problem by putting a lot of power in the hands of client developers. The basic premise is that the clients can always describe what data they need. Continuing the Facebook engineering blog post:</p>
<blockquote>
<p>There was also a considerable amount of code to write on both the server to prepare the data and on the client to parse it. This frustration inspired a few of us to start the project that ultimately became GraphQL. GraphQL was our opportunity to rethink mobile app data-fetching from the perspective of product designers and developers. It moved the focus of development to the client apps, where designers and developers spend their time and attention.</p>
</blockquote>
<p>GraphQL has a strong community behind it that supports many languages including Java, Javascript, C#, Python, etc. It’s used by many companies and teams of all sizes including Facebook, Pinterest, GitHub, Yelp, and many others.</p>
<h3 id="playing-with-graphql-in-browser-graphiql">Playing with GraphQL in Browser: GraphiQL</h3>
<p>Let’s get hands-on and learn GraphQL by trying it out, thanks to the nice folks at GitHub who have made public their GraphQL APIs. What’s even better is that we don’t need to install any command line tools or do anything special. All you need is a web-browser and a free GitHub account. To run queries on GitHub’s GraphQL, we’ll be using a tool called <a href="https://github.com/graphql/graphiql">GraphiQL</a> (<a href="http://www.howtopronounce.cc/graphiql">pronounciation</a>). Think of it like <a href="https://swagger.io">Swagger</a> for RESTful APIs.</p>
<blockquote>
<p>A graphical interactive in-browser GraphQL IDE.</p>
</blockquote>
<p>Graph<em>i</em>QL is easy to integrate and is used a lot by teams when working with GraphQL in development and pre-production environments. I use it a lot to explore queries, find issues, etc. before moving on to implementation. Goes without saying, but you shouldn’t enable it on your public servers unless you’re exposing a public API like GitHub.</p>
<p>Alright, without further ado, let’s head over GitHub’s Graph<em>i</em>QL endpoint and open it in a <strong>new tab</strong> so you can try out the examples. Here’s the link to <strong><a href="https://developer.github.com/v4/explorer/">GitHub GraphQL Explorer</a></strong>. (They have customized the UI a bit but it’s Graph<em>i</em>QL under the hood.)</p>
<h3 id="first-graphql-query">First GraphQL Query</h3>
<p>Type the following query and hit the play button in the top right corner or press <code class="language-plaintext highlighter-rouge">Ctrl-Enter</code> to execute the query.</p>
<pre class="prettyprint">
{
viewer {
name
isEmployee
location
}
}
</pre>
<p><img src="https://codeahoy.com/img/graphql/graphiql-overview.png" alt="GitHub_GraphiQL" /></p>
<p>Congratulations! You just ran our first GraphQL query. The data GraphQL returned is in <strong>JSON format</strong> and has the same <strong>shape</strong> as the query. This is an important concept in GraphQL: the shape of the response (query result) closely matches the result. The syntax of the query is custom to GraphQL and corresponds to its schema which we’ll explore in the next section.</p>
<p>In the query we just ran, we asked the GraphQL server to return three fields (<code class="language-plaintext highlighter-rouge">name</code> etc.) for an object called <code class="language-plaintext highlighter-rouge">viewer</code> which represents your GitHub account. We could easily ask for more fields as shown in the example below where I added three new fields to the same query:</p>
<p><img src="https://codeahoy.com/img/graphql/graphiql-more-parameters.png" alt="GitHub_GraphiQL_Additional_Fields" /></p>
<p>Hint: Graph<em>i</em>QL will <strong>autocomplete</strong> fields as you start typing. Try out other parameters that are available.</p>
<h2 id="graphql-schema">GraphQL Schema</h2>
<p>The schema is the main part of any GraphQL implementation. Schemas are written using what’s known as the <strong>Schema Definition Language</strong> or SDL for short. SDL is human readable and while it might look like Javascript, it’s not. The syntax doesn’t correspond to any one programming language which makes it language-agnostic for good reasons. SDL describes all the fields, arguments, and functionality that is available to clients.</p>
<h4 id="fields">Fields</h4>
<p>Fields are the basic unit of data in GraphQL and the center of its universe. According to the <a href="https://graphql.github.io/learn/schema/">official documentation</a>:</p>
<blockquote>
<p>the GraphQL query language is basically about selecting fields on objects.</p>
</blockquote>
<p>Reviewig the query we just wrote:</p>
<pre class="prettyprint">
{
viewer {
name
isEmployee
location
}
}
</pre>
<ol>
<li>We start with the special “root” object.</li>
<li>We select an object <code class="language-plaintext highlighter-rouge">viewer</code> on the query which represents your Github account.</li>
<li>For the <em>object</em> returned by <code class="language-plaintext highlighter-rouge">viewer</code>, we select the <code class="language-plaintext highlighter-rouge">name</code>, <code class="language-plaintext highlighter-rouge">isEmployee</code> and <code class="language-plaintext highlighter-rouge">location</code> fields to be returned.</li>
</ol>
<p>Fields have <strong>types</strong>. The following types are currently supported:</p>
<ul>
<li>Integer</li>
<li>Float</li>
<li>Boolean</li>
<li>String</li>
<li>Enum</li>
<li>List</li>
<li>Object (custom types)</li>
</ul>
<p>GitHub GraphQL implementation makes available the following <strong>objects</strong> that are commonly associated with GitHub. We can query these objects and ask GitHub to return fields that we need, as we did for the <code class="language-plaintext highlighter-rouge">viewer</code> object in the last example.</p>
<ul>
<li>Viewer</li>
<li>Repositories</li>
<li>Users</li>
<li>Organizations</li>
<li>Issues</li>
<li>etc.</li>
</ul>
<p>For more information on schema and the Schema Definition Language (SDL,) you can go to this <a href="https://www.apollographql.com/docs/apollo-server/schema/schema/">tutorial</a>.</p>
<h3 id="viewing-api-documentation">Viewing API Documentation</h3>
<p>Graph<em>i</em>QL makes it very easy to explore the schema to get information on what queries it support. It comes with a built-in documentation tool called <strong>Documentation Explorer</strong> which shows you all the available types, fields, arguments, available fields and more. It’s a pretty kick-ass feature and I like it a lot. To open the <strong>Document Explorer</strong>, <em>click</em> the <code class="language-plaintext highlighter-rouge">< Docs</code> icon typically on the top left side. This will open a new pane. <em>Type</em> <code class="language-plaintext highlighter-rouge">repository</code> in the search field to explore what arguments it takes and which fields it returns.</p>
<p><img src="https://codeahoy.com/img/graphql/graphiql-schema.png" alt="GitHub_GraphiQL_Schema_Documentation" /></p>
<p>In addition to using <strong>Document Explorer</strong> to explore the schema, you could also query it schema directly. This is useful if you are not using the Graph<em>i</em>QL interface and need an alternate way to look at the schema. <code class="language-plaintext highlighter-rouge">__schema</code> and <code class="language-plaintext highlighter-rouge">__type</code> fields are used for this purpose. This is called <strong>introspection</strong> and you can read more about it <a href="https://graphql.org/learn/introspection/">here</a> if you are interested.</p>
<h2 id="graphql-query-arguments">GraphQL Query Arguments</h2>
<p>In the last couple of queries we ran, we just asked GraphQL to return as some fields e.g. <code class="language-plaintext highlighter-rouge">name</code> or <code class="language-plaintext highlighter-rouge">location</code> without passing any arguments. It’s possible to pass arguments to a query. Let’s take a look at another query, one that requires arguments.</p>
<pre class="prettyprint">
{
repository(owner: "umermansoor", name: "microservices") {
name
nameWithOwner
description
# Nested field
stargazers {
totalCount
}
}
}
</pre>
<p><img src="https://codeahoy.com/img/graphql/graphiql-query-with-arguments.png" alt="GitHub_GraphiQL_With_Arguments" /></p>
<p>In the above query, we passed two arguments: <code class="language-plaintext highlighter-rouge">owner</code> and <code class="language-plaintext highlighter-rouge">name</code> to the <code class="language-plaintext highlighter-rouge">repository</code> object of type <code class="language-plaintext highlighter-rouge">Repository</code>. This object returns information about a GitHub repository. In this example, I passed my Github username <code class="language-plaintext highlighter-rouge">umansoor</code> as the owner and <code class="language-plaintext highlighter-rouge">microservices</code> as the name of a repository which I own. (Fun fact: This repository is an example project I created when I was experimenting with Flask. You can read more about it <a href="https://codeahoy.com/2016/07/10/writing-microservices-in-python-using-flask/.">here</a>)</p>
<p>GraphQL arguments can be either required or optional. This is controlled in the schema. You can easily see which arguments are required vs optional. An argument ending with an exclamation point (<code class="language-plaintext highlighter-rouge">!</code>) means that is required. One’s that do not end with an <code class="language-plaintext highlighter-rouge">!</code> are optional.</p>
<p><img src="https://codeahoy.com/img/graphql/graphql-arguments-required-optional.png" alt="Required_vs_Optional_Arguments" /></p>
<h2 id="multiple-queries-and-aliases">Multiple Queries and Aliases</h2>
<p>Let’s revisit our <code class="language-plaintext highlighter-rouge">repository</code> example and suppose we want to fetch information about <strong>two separate</strong> repositories at the same time. A naive approach will be to repeat the query twice. Let’s see what happens when we do that:</p>
<p><img src="https://codeahoy.com/img/graphql/graphql-multiple-query-error.png" alt="GitHub_GraphiQL_Multiple_Query_Error" /></p>
<p>It threw an error basically saying that there’s an argument conflict. In GraphQL, we can’t query the same field with different arguments, which is what we did the in the last example: we called the <code class="language-plaintext highlighter-rouge">repository</code> object (of type <code class="language-plaintext highlighter-rouge">Repository</code>) with two different sets of arguments:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">repository(owner: "umermansoor", name: "microservices")...</code></li>
<li><code class="language-plaintext highlighter-rouge">repository(owner: "graphql", name: "graphiql") ...</code></li>
</ul>
<p>This REST like approach didn’t work. Good news is that there’s a really easy way to fetch both fields in one call. This is where <strong>aliases</strong> come in. They allow us to attach a custom name to each query. Here’s the last query that failed with aliases which nows works:</p>
<pre class="prettyprint">
{
pythonMicroservicesRepository: repository(owner: "umermansoor", name: "microservices") {
name
nameWithOwner
description
forkCount
}
graphqlRepository:repository(owner: "graphql", name: "graphiql") {
name
nameWithOwner
description
forkCount
}
}
</pre>
<p>This is the result that I get. Notice that even with aliases, the shape of the response closely resembles the query. The name <code class="language-plaintext highlighter-rouge">repository</code> in the response is replaced by the alias that we provided in the query:</p>
<pre class="prettyprint">
{
"data": {
"pythonMicroservicesRepository": {
"name": "microservices",
"nameWithOwner": "umermansoor/microservices",
"description": "Example of Microservices written using Flask.",
"forkCount": 199
},
"graphqlRepository": {
"name": "graphiql",
"nameWithOwner": "graphql/graphiql",
"description": "An in-browser IDE for exploring GraphQL.",
"forkCount": 890
}
}
}
</pre>
<h3 id="fetching-related-objects-using-connections-edges-and-nodes">Fetching Related Objects using Connections, Edges and Nodes</h3>
<p>Suppose you want to fetch repositories that belong to a user and the commits that the user has done. Think about it for a moment about how you’d normally do it using REST API.</p>
<p>In REST, it will typically take multiple calls. In GraphQL, you can fetch related objects easily in the same query. The first concept we’ll look at is called <strong>Connection</strong>. A connection allows fetching related objects in the same query. Objects are connected to other objects using <strong>edges</strong>. In other words, when you query for a connection, you’re traversing connection’s edges to get its nodes. A <strong>node</strong> is a generic term for an object that is accessible via an edge. You can read more about Connections <a href="https://blog.apollographql.com/explaining-graphql-connections-c48b7c3d6976">here</a>. I’m borrowing a diagram and an example below to complete the definition.</p>
<p><img src="https://codeahoy.com/img/graphql/graphql_connections_edges_nodes.png" alt="GraphQL_Connections_Edges_Node" /></p>
<p>If we want to get first three friends (also users) that a user say <em>caesar</em> is connected to, we can run this query:</p>
<pre class="prettyprint">
// Made up example. Will NOT work in GitHub GraphQL Explorer
{
user(id: "caesar") {
id
name
friendsConnection(first: 3) {
edges {
cursor
node {
name
}
}
}
}
}
</pre>
<p>You can try out a similar query in GitHub GraphQL Explorer. The following query will return first 5 public repositories for a <em>alexcrichton</em> (btw, who is an excellent developer!)</p>
<pre class="prettyprint">
{
user(login: "alexcrichton") {
bio
repositories(last: 5, privacy: PUBLIC) {
edges {
node {
name
}
}
}
}
}
</pre>
<p>The <code class="language-plaintext highlighter-rouge">first</code> parameter that we passed to <code class="language-plaintext highlighter-rouge">repositories</code> connection in the last example allowed us to <strong>paginate</strong> results. There are several other pagination options available such as <code class="language-plaintext highlighter-rouge">last</code>, <code class="language-plaintext highlighter-rouge">offset</code>, <code class="language-plaintext highlighter-rouge">after</code> etc. You can read more <a href="https://graphql.org/learn/pagination/">here</a>.</p>
<h2 id="operation-names-and-variable-arguments">Operation Names and Variable Arguments</h2>
<p>In our last example, we passed the argument (username) directly into the query. What if we want to make it dynamic and control it easily without editing the string in the query for each time we want to lookup a different user?</p>
<p>Up until now, we have been using a shorthand syntax to query GitHub GraphQL by omitting the optional <code class="language-plaintext highlighter-rouge">query</code> keyword. Imagine if we have a lot of queries. It will be nice to have a way to name the queries to make them easy to find. It turns out there’s a way in GraphQL to do this. We can have multiple queries and give them different names so we can choose at runtime which query we want to run.</p>
<p><img src="https://codeahoy.com/img/graphql/graphiql_named_queries.png" alt="GraphQL_Named_Query" /></p>
<p>In the image above, we have defined two different queries and gave them unique names: <code class="language-plaintext highlighter-rouge">UserDetails1</code> and <code class="language-plaintext highlighter-rouge">UserDetails2</code> to get details for two different GitHub accounts. It works but doesn’t look right (duplication). We are running the same query twice with different hardcoded usernames. Let’s make it dynamic by passing the username as a parameter at run-time. We can this by following these steps:</p>
<ol>
<li>Define a named query e.g. <code class="language-plaintext highlighter-rouge">GitHubUserDetails</code></li>
<li>Specify a query argument and make it required by adding exclamation point e.g. <code class="language-plaintext highlighter-rouge">query GitHubUserDetails($username:String!)</code></li>
<li>Provide the value to the argument at run-time before running the query e.g. <code class="language-plaintext highlighter-rouge">{"username": "alexcrichton"}</code></li>
</ol>
<p>Putting all these together, we get:</p>
<pre class="prettyprint">
query GitHubUserDetails($username:String!) {
user(login: $username) {
bio
repositories(last: 5, privacy: PUBLIC) {
edges {
node {
name
}
}
}
}
}
</pre>
<p>And in query parameters section at the bottom, set the value of the <code class="language-plaintext highlighter-rouge">username</code> parameter:</p>
<pre class="prettyprint">
{
"username": "alexcrichton"
}
</pre>
<p><img src="https://codeahoy.com/img/graphql/graphql_namedquery_variable.png" alt="GraphQL_Named_Query_Variable" /></p>
<h2 id="mutations">Mutations</h2>
<p>So far, we’ve been just fetching data. Just like in REST APIs (e.g. <a href="https://codeahoy.com/2016/07/04/rest-design-choosing-the-right-http-method/">PUT method in REST</a>,) there are ways in GraphQL to update or modify data on the server. This is achieved using what’s known as <strong>mutations</strong>. You can see a list of available mutations and more details in the Document Explorer. Let’s see how to use mutations using an example. In this example, we’ll add a new star to an existing repository.</p>
<p>Please note that GitHub has blocked mutations to third-party repositories. So you’ll need to follow these for <strong>one of your own repositories</strong>.</p>
<p>Before you can run a mutation to add a star to one of your repositories, you’d need to get the unique id called for your repository. It’s simple enough to get this information using a query like the one below. Please replace <code class="language-plaintext highlighter-rouge">owner</code> with your GitHub username (it has to be yours!) and <code class="language-plaintext highlighter-rouge">name</code> with the name of one of your repositories.</p>
<pre class="prettyprint">
query FindRepositoryId {
repository(owner: "umermansoor", name:"hadoop-java-example") {
id
}
}
</pre>
<p>You should see a response like this:</p>
<pre class="prettyprint">
{
"data": {
"repository": {
"id": "MDEwOlJlcG9zaXRvcnk3NTkxMzY1"
}
}
}
</pre>
<p>Now that we have <code class="language-plaintext highlighter-rouge">id</code>, we can run our mutation to add star to the repository.</p>
<pre class="prettyprint">
mutation AddStarMutation($input: AddStarInput!) {
addStar(input: $input) {
clientMutationId
}
}
</pre>
<p>Then in the Query Parameters section at the bottom, add the following code, replacing <code class="language-plaintext highlighter-rouge">starrableId</code> with the <code class="language-plaintext highlighter-rouge">id</code> you retrieved in the query above. <code class="language-plaintext highlighter-rouge">clientMutationId</code> could be anything and you can leave it as is.</p>
<pre class="prettyprint">
{
"input": {
"starrableId": "MDEwOlJlcG9zaXRvcnk3NTkxMzY1",
"clientMutationId": "12345"
}
}
</pre>
<p><img src="https://codeahoy.com/img/graphql/graphql-mutation-example.png" alt="GraphQL_Mutation" /></p>
<p>Run the mutation now. If everything was done right, you should see that you’ve starred the repository.</p>
<p>That’s all. I hope this gave you a quick overview of GraphQL and its capabilities.</p>
<p>Note: While I have compared GraphQL and REST in this article, please keep in mind that GraphQL is not a replacement of REST. There are many cases where REST makes perfect sense (e.g. for internal APIs).</p>
<p>If you’re interested in learning more about GraphQL, I would highly recommend the video course below. It’s free and shows how to build a GraphQL server using Node.js and allow React frontend of query it using Apollo Client. Until next time!</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/ed8SzALpx1Q" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
== vs === in Javascript and Which Should be Used When2019-10-12T00:00:00+00:00https://codeahoy.com/javascript/2019/10/12/==-vs-===-in-javascript<p>In Javascript, we have couple of options for checking equality:</p>
<ul>
<li><strong>==</strong> (Double equals operator): Known as the equality or <em>abstract comparison</em> operator</li>
<li><strong>===</strong> (Triple equals operator): Known as the identity or <em>strict comparison</em> operator</li>
</ul>
<p>In this post, we’ll explore the similarities and differences between these operators.</p>
<!--more-->
<p>Let’s declare two variables <code class="language-plaintext highlighter-rouge">foo</code> and <code class="language-plaintext highlighter-rouge">bar</code> and compare them using both operators.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">foo</span> <span class="o">=</span> <span class="mi">13</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">bar</span> <span class="o">=</span> <span class="mi">13</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">foo</span> <span class="o">==</span> <span class="nx">bar</span><span class="p">);</span> <span class="c1">// true</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">foo</span> <span class="o">===</span> <span class="nx">bar</span><span class="p">);</span> <span class="c1">// also true</span>
</code></pre></div></div>
<p>In the above example, both operators returned the same answer i.e. <code class="language-plaintext highlighter-rouge">true</code>. So what’s the difference?</p>
<h2 id="the-difference-between--and-">The Difference between <code class="language-plaintext highlighter-rouge">==</code> and <code class="language-plaintext highlighter-rouge">===</code></h2>
<p>The difference between <code class="language-plaintext highlighter-rouge">==</code> and <code class="language-plaintext highlighter-rouge">===</code> is that:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">==</code> <strong>converts</strong> the variable values to the <strong>same</strong> type before performing comparison. This is called <a href="https://developer.mozilla.org/en-US/docs/Glossary/Type_coercion">type coercion</a>.</li>
<li><code class="language-plaintext highlighter-rouge">===</code> does <strong>not</strong> do any type conversion (coercion) and returns <em>true</em> only <strong>if</strong> both values <strong>and</strong> types are identical for the two variables being compared.</li>
</ul>
<p>Let’s take a look at another example:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">one</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">one_again</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">one_string</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">;</span> <span class="c1">// note: this is string</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">==</span> <span class="nx">one_again</span><span class="p">);</span> <span class="c1">// true</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">===</span> <span class="nx">one_again</span><span class="p">);</span> <span class="c1">// true</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">==</span> <span class="nx">one_string</span><span class="p">);</span> <span class="c1">// true. See below for explanation.</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">===</span> <span class="nx">one_string</span><span class="p">);</span> <span class="c1">// false. See below for explanation.</span>
</code></pre></div></div>
<ul>
<li>Line 7: <code class="language-plaintext highlighter-rouge">console.log(one == one_string)</code> returns <em>true</em> because both variables, <code class="language-plaintext highlighter-rouge">one</code> and <code class="language-plaintext highlighter-rouge">one_string</code> contain the same value even though they have <em>different types</em>: <code class="language-plaintext highlighter-rouge">one</code> is of type <code class="language-plaintext highlighter-rouge">Number</code> whereas <code class="language-plaintext highlighter-rouge">one_string</code> is <code class="language-plaintext highlighter-rouge">String</code>. But since the <code class="language-plaintext highlighter-rouge">==</code> operator does type coercion, the result is true.</li>
<li>Line 8: <code class="language-plaintext highlighter-rouge">console.log(one === one_string)</code> returns <em>false</em> because the types of variables are different.</li>
</ul>
<h3 id="is--faster-than--a-quick-look-at-the-performance-of-the-two-operators">Is <code class="language-plaintext highlighter-rouge">===</code> Faster than <code class="language-plaintext highlighter-rouge">==</code>? A Quick Look at the Performance of the Two Operators</h3>
<p><strong>In theory</strong>, when comparing variables with identical types, the performance should be similar across both operators because they use the same algorithm. When the types are different, triple equals operator (<code class="language-plaintext highlighter-rouge">===</code>) should perform better than double equals (<code class="language-plaintext highlighter-rouge">==</code>) because it doesn’t have to do the extra step of type coercion. But does it? Here are some performance tests you could try yourself to see for yourself.</p>
<ul>
<li><a href="https://web.archive.org/web/20160312115113/https://jsperf.com/triple-equals-vs-twice-equals/18">jsperf Test 2</a></li>
</ul>
<p>If you look at the graph at the bottom of the tests, you’d see performance varies across different browser implementations and the gains in performance are almost negligible.</p>
<p>But if you think about it, <strong>performance is totally irrelevant</strong> and shouldn’t play a role in deciding when to use one operator over the other. Either you need type coercion or you don’t. If you don’t need it, don’t use double equals operator (<code class="language-plaintext highlighter-rouge">==</code>) because you might get unexpected results. Most <a href="https://www.jslint.com/">linters</a> will complain if you use <code class="language-plaintext highlighter-rouge">==</code>. To further scare you away from <code class="language-plaintext highlighter-rouge">==</code>: it’s pretty <strong>confusing</strong> and has odd rules. For example, <code class="language-plaintext highlighter-rouge">"1" == true</code> or <code class="language-plaintext highlighter-rouge">"" == 0</code> will return <code class="language-plaintext highlighter-rouge">true</code>. For more peculiarities, take a look at the <a href="https://dorey.github.io/JavaScript-Equality-Table/">Javascript Equality Table</a>.</p>
<p>In short, always use <code class="language-plaintext highlighter-rouge">===</code> everywhere except when you need type coercion (in that case, use <code class="language-plaintext highlighter-rouge">==</code>.)</p>
<h3 id="inequality-operators--and-">Inequality Operators: <code class="language-plaintext highlighter-rouge">!=</code> and <code class="language-plaintext highlighter-rouge">!==</code></h3>
<p><code class="language-plaintext highlighter-rouge">==</code> and <code class="language-plaintext highlighter-rouge">===</code> have their counterparts when it comes to checking for inequality:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">!=</code>: Converts values if variables are different types before checking for inequality</li>
<li><code class="language-plaintext highlighter-rouge">!==</code>: Checks both type and value for the two variables being compared</li>
</ul>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">one</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">one_again</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">one_string</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">;</span> <span class="c1">// note: this is a string</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">!=</span> <span class="nx">one_again</span><span class="p">);</span> <span class="c1">// false</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">!=</span> <span class="nx">one_string</span><span class="p">);</span> <span class="c1">// false</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">one</span> <span class="o">!==</span> <span class="nx">one_string</span><span class="p">);</span><span class="c1">// true. Types are different</span>
</code></pre></div></div>
<h3 id="equality-operators-and-objects-and-other-reference-types">Equality Operators and Objects (and other reference types)</h3>
<p>So far, we have been exploring equality or inequality operators using primitive types. What about reference types like <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array">Arrays</a> or <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object">Objects</a>. If we create two arrays that have identical contents, can we compare them using equalty operators the same way we do it for primitives? The answer is <em>no</em>, you can’t. Let’s take a look at an example:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">a1</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="kd">var</span> <span class="nx">a2</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">a1</span> <span class="o">==</span> <span class="nx">a2</span><span class="p">);</span> <span class="c1">// false</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">a1</span> <span class="o">===</span> <span class="nx">a2</span><span class="p">);</span> <span class="c1">// false</span>
</code></pre></div></div>
<p>Here, both the <code class="language-plaintext highlighter-rouge">==</code> and <code class="language-plaintext highlighter-rouge">===</code> return the same answer: <code class="language-plaintext highlighter-rouge">false</code>. What’s happening here is that both <code class="language-plaintext highlighter-rouge">a1</code> and <code class="language-plaintext highlighter-rouge">a2</code> are pointing to different objects in memory. Even though the array contents are the same, these essentially have different values. Same applies to objects and other reference types.</p>
<h3 id="ecmascript-6-objectis">ECMAScript 6: Object.is()</h3>
<p>I said that the beginning of the article that are couple of options for checking equality in Javascript. That isn’t true anymore. ECMA Script 6 introduced a third method for comparing values:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Object.is()</code></li>
</ul>
<p>Triple equals operator (<code class="language-plaintext highlighter-rouge">===</code>) is the recommended way for value comparison, but it’s not perfect. Here’s couple of examples where its behavior is confusing:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="o">+</span><span class="mi">0</span> <span class="o">===</span> <span class="o">-</span><span class="mi">0</span><span class="p">);</span> <span class="c1">// true </span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="kc">NaN</span> <span class="o">===</span> <span class="kc">NaN</span><span class="p">);</span> <span class="c1">// false</span>
</code></pre></div></div>
<p>To make comparisons less confusing, <a href="https://en.wikipedia.org/wiki/ECMAScript#6th_Edition_-_ECMAScript_2015">ECMAScript 6</a> introduced a new method: <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/is">Object.is()</a>. It takes two arguments and returns <code class="language-plaintext highlighter-rouge">true</code> if both the values and types are equal. Essentially, its identical to the <code class="language-plaintext highlighter-rouge">===</code> operator, but without its quirks. Let’s take a look at some examples:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">is</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">));</span> <span class="c1">// true</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">is</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="dl">"</span><span class="s2">2</span><span class="dl">"</span><span class="p">));</span> <span class="c1">// false. Different types</span>
<span class="c1">// And it fixes the quirks of ===</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">is</span><span class="p">(</span><span class="o">+</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">0</span><span class="p">));</span> <span class="c1">// false</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">is</span><span class="p">(</span><span class="kc">NaN</span><span class="p">,</span> <span class="kc">NaN</span><span class="p">));</span><span class="c1">// true</span>
</code></pre></div></div>
<p>Next up, read our <strong><a href="/learn/html/ch8/">Free Primer on JavaScript</a></strong>.</p>
What are -Xms and -Xms parameters in Java/JVM (Updated up to Java 13)2019-09-02T00:00:00+00:00https://codeahoy.com/2019/09/02/java-xmx-vs-xms<p>In short,</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Xmx</code> specifies the <strong>maximum</strong> heap size available to an application</li>
<li><code class="language-plaintext highlighter-rouge">Xms</code> specifies the <strong>minimum</strong> heap size available to an application</li>
</ul>
<p>These are Java Virtual Machine (JVM) parameters that are used to specify memory boundaries for Java applications. They are often used when troubleshooting performance issues or <a href="https://docs.oracle.com/javase/9/docs/api/java/lang/OutOfMemoryError.html">OutOfMemoryError</a>s. They <strong>control the amount of memory</strong> that is available to a Java application. The <code class="language-plaintext highlighter-rouge">Xmx</code> parameter specifies the <strong>maximum memory</strong> an app can use, where as <code class="language-plaintext highlighter-rouge">Xms</code> specifies the <strong>minimum</strong> or the initial memory pool. If your application exceeds the maximum memory (allocated using the <code class="language-plaintext highlighter-rouge">Xmx</code>) and the garbage collector cannot free up memory, the JVM will crash with a <a href="https://docs.oracle.com/javase/9/docs/api/java/lang/OutOfMemoryError.html">OutOfMemoryError</a>. If you’re interested, I wrote an <strong>article</strong> explaining with examples <a href="ttps://codeahoy.com/2017/08/06/basics-of-java-garbage-collection/">how garbage collection works and its generations</a>.</p>
<!--more-->
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>java <span class="nt">-Xmx256m</span> Xmx1024m <span class="nt">-jar</span> yourapp.jar
</code></pre></div></div>
<p>In the example above, the application <code class="language-plaintext highlighter-rouge">yourapp.jar</code> will get an initial memory pool of 256 megabytes and a maximum up to 1024 megabytes. In <code class="language-plaintext highlighter-rouge">256m</code>, the <code class="language-plaintext highlighter-rouge">m</code> stands for megabytes. You can use <code class="language-plaintext highlighter-rouge">g</code> or <code class="language-plaintext highlighter-rouge">G</code> to indicate gigabytes.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Xmx1g</code> or <code class="language-plaintext highlighter-rouge">Xmx1G</code>: Set the maximum memory size to 1 gigabytes.</li>
<li><code class="language-plaintext highlighter-rouge">Xmx1024m</code> or <code class="language-plaintext highlighter-rouge">Xmx1024M</code>: Set the maximum memory size to 1024 megabytes.</li>
<li><code class="language-plaintext highlighter-rouge">Xmx1024000k</code> or <code class="language-plaintext highlighter-rouge">Xmx1024000K</code>: Sets the maximum memory size to 1024000 kilobytes.</li>
</ul>
<p>It’s important to note that both <code class="language-plaintext highlighter-rouge">Xmx</code> and <code class="language-plaintext highlighter-rouge">Xms</code> are optional. If these are not provided, the Java Virtual Machine (JVM) will use <strong>default values</strong> for them.</p>
<h2 id="default-java-xmx-and-xms-values">Default Java Xmx and Xms Values</h2>
<p>The default values vary and depend on different factors. It depends on the amount of physical memory of the system, JVM mode (e.g. <code class="language-plaintext highlighter-rouge">-server vs -client</code>) and other factors like JVM implementation and version.
Typically, the default values are <strong><a href="https://docs.oracle.com/javase/9/gctuning/ergonomics.htm#JSGCT-GUID-83551BA5-ADEA-4E2E-B60A-3A953DA8FD02">calculated</a></strong> as follows:</p>
<ul>
<li>Initial heap size of 1/64 of physical memory (for <code class="language-plaintext highlighter-rouge">Xms</code>)</li>
<li>Maximum heap size of 1/4 of physical memory (for <code class="language-plaintext highlighter-rouge">Xmx</code>)</li>
</ul>
<p>An easy way to determine the default settings is to use the Print Flags option. It will show xms (InitialHeapSize) and xmx (MaxHeapSize) in bytes. You’ll need to convert manually to MB or GB.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-XX</span>:+PrintCommandLineFlags <span class="nt">-version</span>
</code></pre></div></div>
<p>On my machine (Macbook Pro with 8 GB of memory) I got the following output:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">-XX</span>:InitialHeapSize<span class="o">=</span>134217728 <span class="nt">-XX</span>:MaxHeapSize<span class="o">=</span>2147483648 <span class="nt">-XX</span>:+PrintCommandLineFlags
<span class="nt">-XX</span>:+UseCompressedClassPointers <span class="nt">-XX</span>:+UseCompressedOops <span class="nt">-XX</span>:+UseParallelGC
</code></pre></div></div>
<p>So on my machine with <strong>8 GB</strong> of total physical memory, I get:</p>
<ul>
<li>Xms (InitialHeapSize): 134217728 bytes or <strong>134 MB</strong> (~ roughly 1/64th of 8 GB)</li>
<li>Xmx (MaxHeapSize): 2147483648 bytes or <strong>2 GB</strong> (~ roughly 1/4th of 8 GB)</li>
</ul>
<p>You can specify either Xms, Xmx or both. If you don’t specify either one of them, the default value will be used. In the example below, the maximum memory will be limited to 1024 megabytes. The initial memory will use the default value.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
java <span class="nt">-Xmx1024m</span> <span class="nt">-jar</span> yourapp.jar
</code></pre></div></div>
<p>Here’s a <strong>good YouTube video</strong> that walks through the process of troubleshooting memory related errors and shows to fix them using examples.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/kQpkjCUQvEc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<h4 id="java-13-and-the-z-garbage-collector">Java 13 and the Z Garbage Collector</h4>
<p>Java 13 introduced a new garbage collector called ZGC. One of its features includes an optimization to return un-used memory to the operating system. This feature is enabled by default and it will not return memory such that heap size shrinks below <code class="language-plaintext highlighter-rouge">Xms</code>. So if you’re setting <code class="language-plaintext highlighter-rouge">Xms</code> to equal <code class="language-plaintext highlighter-rouge">Xmx</code> (as many developers do,) it will essentially disable the feature.</p>
<p>If you want to see all available JVM parameters, you can use the <code class="language-plaintext highlighter-rouge">java -X</code> switch e.g.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>java <span class="nt">-X</span>
<span class="nt">-Xmixed</span> mixed mode execution <span class="o">(</span>default<span class="o">)</span>
<span class="nt">-Xint</span> interpreted mode execution only
<span class="nt">-Xbootclasspath</span>:<directories and zip/jar files separated by :>
<span class="nb">set </span>search path <span class="k">for </span>bootstrap classes and resources
<span class="nt">-Xbootclasspath</span>/a:<directories and zip/jar files separated by :>
append to end of bootstrap class path
<span class="nt">-Xbootclasspath</span>/p:<directories and zip/jar files separated by :>
prepend <span class="k">in </span>front of bootstrap class path
<span class="nt">-Xdiag</span> show additional diagnostic messages
<span class="nt">-Xnoclassgc</span> disable class garbage collection
<span class="nt">-Xincgc</span> <span class="nb">enable </span>incremental garbage collection
<span class="nt">-Xloggc</span>:<file> log GC status to a file with <span class="nb">time </span>stamps
<span class="nt">-Xbatch</span> disable background compilation
<span class="nt">-Xms</span><size> <span class="nb">set </span>initial Java heap size
<span class="nt">-Xmx</span><size> <span class="nb">set </span>maximum Java heap size
<span class="nt">-Xss</span><size> <span class="nb">set </span>java thread stack size
<span class="nt">-Xprof</span> output cpu profiling data
<span class="nt">-Xfuture</span> <span class="nb">enable </span>strictest checks, anticipating future default
<span class="nt">-Xrs</span> reduce use of OS signals by Java/VM <span class="o">(</span>see documentation<span class="o">)</span>
<span class="nt">-Xcheck</span>:jni perform additional checks <span class="k">for </span>JNI functions
<span class="nt">-Xshare</span>:off <span class="k">do </span>not attempt to use shared class data
<span class="nt">-Xshare</span>:auto use shared class data <span class="k">if </span>possible <span class="o">(</span>default<span class="o">)</span>
<span class="nt">-Xshare</span>:on require using shared class data, otherwise fail.
<span class="nt">-XshowSettings</span> show all settings and <span class="k">continue</span>
<span class="nt">-XshowSettings</span>:all
show all settings and <span class="k">continue</span>
<span class="nt">-XshowSettings</span>:vm show all vm related settings and <span class="k">continue</span>
<span class="nt">-XshowSettings</span>:properties
show all property settings and <span class="k">continue</span>
<span class="nt">-XshowSettings</span>:locale
show all locale related settings and <span class="k">continue
</span>The <span class="nt">-X</span> options are non-standard and subject to change without notice.
</code></pre></div></div>
Spring Boot - Replace Tomcat With Jetty As the Embedded Server2019-09-01T00:00:00+00:00https://codeahoy.com/java/springboot/tutorial/2019/09/01/spring-boot-replace-tomcat-with-jetty-as-the-embedded-server<p><a href="http://tomcat.apache.org/">Apache Tomcat</a> and <a href="https://www.eclipse.org/jetty/">Eclipse Jetty</a> are two of the most popular web servers and Java Servlet containers. Tomcat is more widely used compared to Jetty and has significantly more market share. On the other hand, Jetty is light-weight, more compact and has a smaller CPU and memory footprint. For this reason, it is easier to work with it in development than Tomcat. This is not to suggest that Jetty isn’t good for production - in my experience, it is as performant as Tomcat if not more.</p>
<p>Many developers prefer Jetty over Tomcat during the development stage when they want to rapidly launch and test web apps on their local machines. <strong>Spring Boot <a href="https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web">web starter</a></strong> uses Tomcat as the <em>default</em> embedded server. Let’s take a look at how to change it to Jetty.</p>
<p><!--more--></p>
<p>If you’d like to change the embedded web server to Jetty in a new Spring Boot web starter project, you’ll have to:</p>
<ol>
<li>Exclude Tomcat from web starter dependency, since it is added by default</li>
<li>Add the Jetty dependency</li>
</ol>
<h3 id="step-1-exclude-tomcat">Step 1: Exclude Tomcat</h3>
<p><strong>Find</strong> the following dependency in <code class="language-plaintext highlighter-rouge">pom.xml</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.springframework.boot<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>spring-boot-starter-web<span class="nt"></artifactId></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<p><strong>Replace</strong> it with:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.springframework.boot<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>spring-boot-starter-web<span class="nt"></artifactId></span>
<span class="nt"><exclusions></span>
<span class="nt"><exclusion></span>
<span class="nt"><groupId></span>org.springframework.boot<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>spring-boot-starter-tomcat<span class="nt"></artifactId></span>
<span class="nt"></exclusion></span>
<span class="nt"></exclusions></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<h3 id="step-2-add-jetty">Step 2: Add Jetty</h3>
<p>Add the following dependency to your <code class="language-plaintext highlighter-rouge">pom.xml</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.springframework.boot<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>spring-boot-starter-jetty<span class="nt"></artifactId></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<p>Please note that there are some other starters e.g. <a href="https://www.thymeleaf.org/">ThymeLeaf templating engine</a>, etc. that might pull in Tomcat by default. If you’re using one of these, you’ll have to manually <em>exclude</em> Tomcat from all such dependencies. When I’m running into issues, I look at maven dependencies to check if Tomcat is being pulled in. You could check for this by either inspecting the dependencies on command line e.g. <code class="language-plaintext highlighter-rouge">mvn dependency:tree -Dverbose</code> or using the inspector in your favourite IDE e.g. IntelliJ or Eclipse.</p>
<h3 id="github-project">GitHub Project</h3>
<p>I created a <a href="https://github.com/codeahoy/BootWebJetty" rel="nofollow">SpringBoot Jetty Starter project</a> on GitHub which excludes Tomcat and uses Jetty as the web server. Here’s the complete <a href="https://github.com/codeahoy/BootWebJetty/blob/master/pom.xml" rel="nofollow">pom file</a> which shows how this is done.</p>
<p>To run the project, you’ll need to clone it first.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/codeahoy/BootWebJetty.git
</code></pre></div></div>
<p>To run it,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean package spring-boot:run
</code></pre></div></div>
<p>In the output, you should see Jetty running on port 8080.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Jetty started on port<span class="o">(</span>s<span class="o">)</span> 8080 <span class="o">(</span>http/1.1<span class="o">)</span> with context path <span class="s1">'/'</span>
</code></pre></div></div>
<p>I created a sample controller called greeting, so you should be able to open http://127.0.0.1:8080/greeting and see a message printed to the browser window.</p>
<p>The project uses SpringBoot 2.2.4 and Java 8. You can change this in the pom file if you want to use different versions.</p>
<p>That’s all. If you found this post useful, please share it using the sharing buttons below. It will help us grow. Thank you.</p>
How Docker Works? Under the Hood Look at How Containers Work on Linux2019-04-12T00:00:00+00:00https://codeahoy.com/2019/04/12/what-are-containers-a-simple-guide-to-containerization-and-how-docker-works<p>Docker is awesome. It enables software developers to package, ship and run their applications <em>anywhere</em> without having to worry about setup or dependencies. Combined with Kubernetes, it becomes even more powerful for streamlining cluster deployments and management. I digress. Back to Docker. Docker is loved by software developers and its adoption rate has been remarkable.</p>
<p>In this post, we’ll look at how Docker works under the hood. Docker uses a technology called “Containerization” to do its magic and that’s what we are going to explore next.</p>
<h2 id="why-do-we-need-containers">Why Do We Need Containers?</h2>
<p>Let’s say you want to run a software <em>Foo</em> on your computer. <em>Foo</em> requires Node.js version <em>10</em> (assume <em>Foo</em> is incompatible with newer Node.js versions), so you install Node <em>10</em> on your machine. Later, you want to run another software, <em>Bar</em>, which requires Node.js version <em>15</em>.</p>
<p>This has created a problem. (Assume we can’t use <code class="language-plaintext highlighter-rouge">nvm</code> to switch between Node versions easily.)</p>
<p>One way we could solve this problem is by using Virtual Machines or VMs to create isolated environments for running <em>Foo</em> and <em>Bar</em>. You can create <em>one</em> VM with <em>Foo</em> and Node 10 and <em>another</em> with <em>Bar</em> and Node 15. Voila, we are back in business.</p>
<p>However, there’s an issue with this approach: it’s very <strong>inefficient</strong>. Each VM requires its own <em>operating system</em>. We are now running two separate guest operating systems on our computer just to run two different processes.</p>
<p>What if there was a way to run <em>Foo</em> and <em>Bar</em> and your machine <strong>without</strong> running two extra operating systems?</p>
<p>Let’s review the <em>interaction</em> between processes and operating system. Whenever a process wants to do anything, it asks the operating system. Processes like <em>Foo</em> and <em>Bar</em> would ask the OS questions like, “which Node version do you have installed?” or “how much memory is available to me?” or “what other processes are running?”</p>
<p>What if we could <strong>intercept</strong> and <strong>control</strong> the communication between processes and operating systems and send customized responses to processes to control their behavior? For example, when <em>Foo</em> and <em>Bar</em> ask “what Node version do you have installed”, we tell Foo that we have Node v10 and send its executable location to Foo. When Bar asks the same question, we respond with Node v16. In other words, what if we create a <em>virtual OS</em> for <em>Foo</em> and <em>Bar</em>?</p>
<p>This is exactly what containers do. They allow us to run different processes, <em>Foo</em> and <em>Bar</em>, on the same OS. A <strong>container runtime</strong> intercepts these questions from processes and gives a customized response. It will tell <em>Foo</em> that Node 10 is installed, and <em>Bar</em> will receive a response that the machine only has Node 15.</p>
<figure class="figure d-block">
<p><img src="/img/dockercontainers/containers.png" class="rounded mx-auto d-block" alt="Containers provide a virtualized runtime to run processes in isolated environments." /></p>
</figure>
<p>By strategically modifying responses to <em>Foo</em> and <em>Bar</em>, we have essentially isolated their environments without running separate operating systems. This strategy is called “containerization”.</p>
<p>Docker is a set of tools <em>and</em> a widely popular <em>container runtime</em>. It’s a <strong>complete platform</strong> for building, testing, deploying and publishing <strong>containerized</strong> applications. I say platform because Docker is a <strong>set of tools</strong> for managing all-things related to containers.</p>
<p>Now that we understand the use case for containers and what they are, let’s take a more in-depth look and how they actually work to achieve isolation.</p>
<h2 id="what-are-containers">What are Containers?</h2>
<p>Containers provide a way to install and run your applications in <strong>isolated environments</strong> on a machine. Applications running inside a container are limited to resources (CPU, memory, disk, process space, users, networking, volumes) allocated for that container. Their <strong>visibility</strong> is limited to container’s resources and doesn’t conflict with other containers. You can think of containers as isolated sandboxes on a single machine for applications to run in.</p>
<p>As we have discussed, this concept is very similar to <strong>virtual machines</strong>. The key differentiator is that containers use a <em>light-weight</em> technique to achieve resource isolation. The technique used by containers exploits features of the underlying <strong>Linux kernel</strong> as opposed to <strong><a href="https://en.wikipedia.org/wiki/Hypervisor">hypervisor</a></strong> based approach taken by virtual machines. In other words, containers call Linux commands to allocate and isolate a set of resources and then runs your application in this space. Let’s take a quick look at two such features:</p>
<h3 id="1-namespaces">1. namespaces</h3>
<p>I’m over simplifying but <a href="http://man7.org/linux/man-pages/man7/namespaces.7.html">Linux namespaces</a> basically allow users to isolate resources like CPU, between independent processes. A process’ access and visibility are limited to its namespace. So users can run processes in one namespace without ever having to worry about conflicting with processes running inside another namespace. Processes can even have the same PID on the same machine within different containers. Likewise, applications in two different containers can use port same ports (e.g. port 80).</p>
<h3 id="2-cgroups">2. cgroups</h3>
<p><a href="http://man7.org/linux/man-pages/man7/cgroups.7.html">croups</a> allow putting <strong>limits</strong> and constraints on available resources. For example, you can create a namespace and limit available memory for processes inside it to 1 GB on a machine that has say 16 GB of memory available.</p>
<p>By now, you’ve probably guessed how Docker works. Behind the scenes, when you ask Docker to run a container, it <strong>sets up a resource isolated environment</strong> on your machine. Then it <strong>copies over your packaged application</strong> and associated files to the filesystem inside the namespace. At this point, the environment setup is complete. Docker then executes the command that you specified and hands over the control.</p>
<p>In short, Docker <strong>orchestrates</strong> by setting up containers using Linux’s namespace and cgroups (and few other) commands, copying your application files to disk allocated for the container and then running the startup command. It also comes with a bunch of other tools for managing containers like the ability to list running containers, stopping containers, publishing container images, and many others.</p>
<figure class="figure d-block">
<p><img src="/img/dockercontainers/containers-on-box.png" class="rounded mx-auto d-block" alt="Containers running on linux kernel to provide isolated environments to processes" /></p>
</figure>
<p>Compared to virtual machines, containers are <strong>light weight and faster</strong> because they make use of the underlying Linux OS to run natively in <strong>loosely</strong> isolated environments. A virtual machine hypervisor creates a very strong boundary to prevent applications from breaking out of it, where as <a href="https://sysdig.com/blog/container-isolation-gone-wrong/">containers’ boundaries are not as strong</a>. Another difference is that since <em>namespace</em> and <em>cgroups</em> features are only available on Linux, <strong>containers can not run on other operating systems</strong>. At this point you might be wondering how Docker runs on macOS or Windows? Docker actually uses a little trick and installs a <strong>Linux virtual machines</strong> on non-Linux operating systems. It then runs containers inside the virtual machine.</p>
<p>Let’s put everything that we have learned so far and create and run a Docker container from scratch. If you don’t already have Docker installed on your machine, head over <a href="https://docs.docker.com/install/">here</a> to install. In our super made up example, we’ll create a Docker container, download a web server written in C, compile it, run it and then connect to the web server from our web browser (in other words, from host machine that’s running the container.)</p>
<p>We’l start where all Docker projects start. By creating a file called <code class="language-plaintext highlighter-rouge">Dockerfile</code>. This file contains instructions that tell Docker how to create a <strong>docker image</strong> that’s used for creating and running containers. Since, we didn’t discuss images, let’s take a look at the <a href="https://docs.docker.com/get-started/#images-and-containers">official definition</a>:</p>
<blockquote>
<p>An image is an <strong>executable package that includes everything needed to run an application–the code, a runtime, libraries, environment variables, and configuration files</strong>. A container is a runtime instance of an image</p>
</blockquote>
<p>Put simply, when you ask Docker to run a container, you must give it an <strong>image</strong> which contains:</p>
<ol>
<li><strong>File system snapshot</strong> containing your application and all of its dependencies.</li>
<li><strong>A startup command</strong> to run when the container is launched.</li>
</ol>
<p>Back to creating our <code class="language-plaintext highlighter-rouge">Dockerfile</code> so we can build an image. It’s extremely common in the Docker world to create images based on other images. For example, the official reds Docker image is based on ‘Debian’ file system snapshot (rootfs tarball), and installs on configures Redis on top of it.</p>
<p>In our example, we’ll base our image on <a href="https://hub.docker.com/_/alpine">Alpine Linux</a>. When you see the term <em>alpine</em> in Docker, it usually means a stripped down, bare-essentials image. Alpine Linux image is about 5 MB in size!</p>
<p>Alright. Create a new folder (e.g. <code class="language-plaintext highlighter-rouge">dockerprj</code>) on your computer and then create a file called <code class="language-plaintext highlighter-rouge">Dockerfile</code>.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umermansoor:dockerprj<span class="nv">$ </span><span class="nb">touch </span>Dockerfile
</code></pre></div></div>
<p>Paste the following in the <code class="language-plaintext highlighter-rouge">Dockerfile</code>.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Use Alpine Linux rootfs tarball to base our image on</span>
FROM alpine:3.9
<span class="c"># Set the working directory to be '/home'</span>
WORKDIR <span class="s1">'/home'</span>
<span class="c"># Setup our application on container's file system</span>
RUN wget http://www.cs.cmu.edu/afs/cs/academic/class/15213-s00/www/class28/tiny.c <span class="se">\</span>
<span class="o">&&</span> apk add build-base <span class="se">\</span>
<span class="o">&&</span> gcc tiny.c <span class="nt">-o</span> tiny <span class="se">\</span>
<span class="o">&&</span> <span class="nb">echo</span> <span class="s1">'Hello World'</span> <span class="o">>></span> index.html
<span class="c"># Start the web server. This is container's entry point</span>
CMD <span class="o">[</span><span class="s2">"./tiny"</span>, <span class="s2">"8082"</span><span class="o">]</span>
<span class="c"># Expose port 8082</span>
EXPOSE 8082
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">Dockerfile</code> above contains instructions for Docker to create an <em>image</em>. Essentially, we base our image on Alpine Linux (<a href="https://web.archive.org/web/20190922045729/http://www.ethernetresearch.com/geekzone/building-linux-rootfs-from-scratch/">rootfs tarball</a>) and set our working directory to be <code class="language-plaintext highlighter-rouge">/home</code>. Next, we downloaded, compiled and created an executable of a simple web server written in C. After, that we specify the command to be executed when container is run and expose container’s port 8082 to the host machine.</p>
<p>Now, let’s create the image. Running <code class="language-plaintext highlighter-rouge">docker build</code> in the <strong>same directory</strong> where you created <code class="language-plaintext highlighter-rouge">Dockerfile</code> should do the trick.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umermansoor:dockerprj<span class="nv">$ </span>docker build <span class="nt">-t</span> codeahoydocker <span class="nb">.</span>
</code></pre></div></div>
<p>If the command is successful, you should see something similar:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Successfully tagged codeahoydocker:latest
</code></pre></div></div>
<p>At this point, our image is created. It essentially contains:</p>
<ol>
<li>Filesystem snapshot (Alpine Linux and the web server we installed)</li>
<li>Startup command (<code class="language-plaintext highlighter-rouge">./tiny 8092</code>)</li>
</ol>
<p><img src="https://codeahoy.com/img/dockercontainers/image.png" alt="image" /></p>
<p>Now that we’ve created the image, we can build and run a container from this image. To do so, run the following command:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umermansoor:dockerprj<span class="nv">$ </span>docker run <span class="nt">-p</span> 8082:8082 codeahoydocker:latest
</code></pre></div></div>
<p>Let’s understand what’s going on here.</p>
<p>With <code class="language-plaintext highlighter-rouge">docker run</code>, we asked Docker to create and start a container from the <code class="language-plaintext highlighter-rouge">codeahoydocker:latest</code> image. <code class="language-plaintext highlighter-rouge">-p 8082:8082</code> maps port 8082 of our local machine to port 8082 inside the container. (Remember, our web server inside the container is listening for connections on port 8082.) You’ll not see any output after this command which is totally fine. Switch to your web browser and navigate to <a href="localhost:8082/index.html">localhost:8082/index.html</a>. You should see <em>Hello World</em> message. (Instructions on how to delete the image and container to clean up will be in comments.)</p>
<p><img src="https://codeahoy.com/img/dockercontainers/tiny-container.png" alt="tiny-container" /></p>
<p>In the end, I’d like to add that while Docker is awesome and it’s a good choice for most projects, <strong>I don’t use it everywhere</strong>. In our case, Docker combined with Kubernetes makes it really easy to deploy and manage backend <strong>microservices</strong>. We don’t have to worry about provisioning a new environment for each service, configurations, etc. On the other hand, for performance intensive applications, Docker may not be the best choice. One of the projects I worked on had to handle long-living TCP connections from mobile game clients (1000s per machine.) <strong>Docker networking</strong> presented a lot of issues and I just couldn’t get the performance out of it and didn’t use it for the project.</p>
<p>Hope this was helpful. Until next time.</p>
YAGNI, Cargo Cult and Overengineering - the Planes Won't Land Just Because You Built a Runway in Your Backyard2017-08-19T00:00:00+00:00https://codeahoy.com/2017/08/19/yagni-cargo-cult-and-overengineering-the-planes-wont-land-just-because-you-built-a-runway-in-your-backyard<p>It was April. Year was probably was 2010. The cold, snowy winter was finally coming to an end and the spring was almost in the air. I was preparing for my <strong>final exams</strong>. The review lectures were going on for the <strong>RDBMS course</strong> that I was enrolled in at my university.</p>
<p><img src="https://codeahoy.com/img/blogs/uofc_winter_campus-0006.jpg" alt="uofc" /></p>
<!--more-->
<p>Around the same time, I had started hearing and reading about the shiny, new technology that was going to change the way we use databases. The <em>NoSQL</em> movement was gaining momentum. I was reading blogs about how MongoDB is big time outperforming ancient, <a href="https://www.youtube.com/watch?v=b2F-DItXtZs" rel="nofollow">non web scale</a> relational databases.</p>
<p>After the lecture, I asked my professor:</p>
<blockquote>
<p><em>Me:</em> So, between RDBMS and <em>NoSQL</em> databases, which one do you think is the best?</p>
<p><em>Professor:</em> Well, it depends.</p>
<p><em>Me:</em> Depends on what?</p>
<p><em>Professor:</em> Depends on what you are trying to achieve. Both have their pros and cons. <strong>You pick the right tool for the job</strong>.</p>
<p><em>Me:</em> But MySQL can’t really scale.</p>
<p><em>Professor:</em> How do you think we got this far? Send me an email and I’ll send you some papers and practical uses in the industry.</p>
</blockquote>
<p>SQL was hard for my brain, especially the joins. I loved NoSQL. Simple <em>key->value</em> model without any joins! RDBMS systems that were designed in 1960’s were simply not enough to keep up with modern demands. I had lost all interest in RDBMS and predicted they’ll just fade off in the next few years.</p>
<hr />
<p>It’s 2012. <a href="https://codeahoy.com/2016/04/21/when-to-rewrite-from-scratch-autopsy-of-a-failed-software/">We’re <em>redesigning</em> my employer’s flagship product</a>. The first version was a <strong>monolith</strong> that used the boring <strong>MySQL</strong>. Spending too much time reading blogs and <em>Hacker News</em> comments section, we convinced ourselves that we need to go big and modern:</p>
<ul>
<li>Break monolith into service-oriented architecture, aka, the SOA.</li>
<li>Replace MySQL with Cassandra (MySQL to Redis to Cassandra)</li>
</ul>
<p>And we built it.</p>
<p>There was nothing wrong with the new system… except <strong>one major flaw</strong>. It was too complex for a small startup team to maintain. We had built a <strong>Formula One</strong> race car, that makes frequent pit-stops and requires very specialized maintenance, when we needed a <strong>Toyota Corolla</strong> that goes on for years and years on just the oil change.</p>
<hr />
<p>Fast forward to 2017. It feels like almost all software developers I interview these days, have hands-on experience with microservices architecture and many have even actually used it in production.</p>
<p>A grey San Francisco afternoon. I’m conducting an on-site interview. Masters degree and 3 years of experience at a startup that looked like it didn’t make it. I asked him to tell me about the system he built.</p>
<blockquote>
<p><em>Guy:</em> We built the system using the <em>microservices</em> architecture. We had lots of small services which made our life really easy…</p>
<p><em>Me:</em> Tell me more about it</p>
<p><em>Guy:</em> Data was written to the BI system through a <em>Kafka</em> cluster. <em>Hadoop</em> and <em>MapReduce</em> system was built to process data for analytics. The system was super scalable.</p>
</blockquote>
<p>I pressed him to tell me drawbacks of microservices architecture and problems it introduces. The guy tried to hand-wave his way through and was convinced, just like I was in 2010 about NoSQL databases, that there are absolutely no issues with microservices architecture.</p>
<p>I’m not saying microservices architecture is bad. It has its pros and cons. Organizations who have very complex systems that justify the operational burden, <strong>the overhead</strong>, it introduces. Martin Fowler, who coined the term microservices, warns us of the “<a href="https://martinfowler.com/bliki/MicroservicePremium.html" rel="nofollow">microservices premium</a>”:</p>
<blockquote>
<p>The fulcrum of whether or not to use microservices is the complexity of the system you’re contemplating. <strong>The microservices approach is all about handling a complex system, but in order to do so the approach introduces its own set of complexities</strong>. When you use microservices you have to work on automated deployment, monitoring, dealing with failure, eventual consistency, and other factors that a distributed system introduces. There are well-known ways to cope with all this, but it’s extra effort, and nobody I know in software development seems to have acres of free time.</p>
<p>So my primary guideline would be <strong>don’t even consider microservices unless you have a system that’s too complex to manage as a monolith</strong>. The majority of software systems should be built as a single monolithic application. Do pay attention to good modularity within that monolith, but don’t try to separate it into separate services.</p>
</blockquote>
<p>Netflix is a great example. Their system grew too large and too complex to justify switching to microservices.</p>
<p>In almost all cases, you can’t go wrong by building a monolith first. You break your architecture into services-oriented or… microservices when the benefits outweigh the complexity.</p>
<p>The guy also mentioned <strong>Kafka</strong>. System that handles <a href="https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines" rel="nofollow">2 million writes a second</a> at LinkedIn:</p>
<blockquote>
<p><em>Me:</em> How much data do you stream to Kafka roughly?</p>
<p><em>Guy:</em> We <em>could</em> stream gigabytes of logs…</p>
<p><em>Me:</em> How much data are you streaming <em>right now</em>?</p>
<p><em>Guy:</em> Not a whole lot right now because we only have 3 customers. But the system could scale up to support millions and millions of users. Also, with both Kafka and Hadoop clusters, we get fool-proof fault-tolerance</p>
<p><em>Me:</em> How big is the team and company?</p>
<p><em>Guy:</em> Overall, I guess there were (less than 10) people in the company. Engineering was about 5.</p>
</blockquote>
<p>At this point, I was tempted to ask if he had ever heard of <em><a href="https://martinfowler.com/bliki/Yagni.html" rel="nofollow">YAGNI</a></em>:</p>
<blockquote>
<p>Yagni … stands for “You Aren’t Gonna Need It”. It is a mantra from Extreme Programming … It’s a statement that some capability we <strong>presume our software needs in the future should not be built now because “you aren’t gonna need it”</strong>.</p>
</blockquote>
<p>People sometimes have honest intentions and they don’t introduce new tools, libraries, frameworks, just for the sake of <em>enhancing their resumes</em>. Sometimes they simply speculate enormous growth and try to do everything up front to not have to do this work later.</p>
<blockquote>
<p>The common reason why people build presumptive features is because they think it will be cheaper to build it now rather than build it later. But that cost comparison has to be made at least against the <strong>cost of delay</strong>, preferably factoring in the probability that you’re building an unnecessary feature, for which your odds are at least ⅔.</p>
</blockquote>
<hr />
<p>People know complexity is bad. No one likes to see bugs filed on JIRA or get PagerDuty alerts at 3 a.m. that something is wrong with Cassandra cluster. But why do software developers still do it? Why do they choose to build complex systems without proper investigation?</p>
<p>Are there other reasons for building complex features besides preparing for a hypothetical future?</p>
<p>I’m sure majority of you are familiar with the term <strong>cargo cult software development</strong>. Teams who slavishly and blindly follow techniques of large companies they idolize like Google, Apple or Amazon, in the hopes that they’ll achieve similar success by emulating their idols.</p>
<p>Just like the South Sea natives who built primitive runways, prayed and performed rituals in the hopes that planes would come in and <strong>bring cargo</strong>, the food and supplies. <strong>Richard Feynman</strong> warned graduates at the California Institute of Technology to not fall victim to the cargo cult thinking:</p>
<blockquote>
<p>In the South Seas there is a cargo cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas — he’s the controller — and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes landed.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/cargo-cult.jpg" alt="cargo-cult" /></p>
<p>It reminded me of the time when we took a perfectly good monolith and created service-oriented architecture (SOA). <strong>It didn’t work. The planes didn’t land</strong>.</p>
<p>Whatever the reasons:</p>
<ul>
<li><em>not wanting to be left out</em>: the new Javascript framework from this week will be the next hottest thing.</li>
<li><em>enhancing resume with buzzwords</em>: Monolith won’t impress recruiters / interviewers.</li>
<li><em>imitating heroes</em>: Facebook does it, Google does it, Twitter does it.</li>
<li>technology that could keep up with the projected 5000% YoY company growth: YAGNI.</li>
<li>latest tools to convince people that you’re a <em>proper</em> SF bay area tech company.</li>
</ul>
<p>Cargo-cult engineering just doesn’t work. You are not Google. What works for them will most likely not work for your much, much smaller company. Google actually needed MapReduce because they wanted to regenerate indexes for the entire World Wide Web, or something like that. They needed fault tolerance from thousands of commodity servers. They had <strong><a href="https://www.usenix.org/legacy/event/osdi08/tech/full_papers/zaharia/zaharia_html/index.html">20 petabytes of data</a></strong> to process.</p>
<p>20 petabytes is just enormous. In terms of the number of disk drives, here’s what half of that, 10 petabytes, would <a href="https://www.backblaze.com/blog/10-petabytes-visualized/" rel="nofollow">look</a> like:</p>
<p><img src="https://codeahoy.com/img/blogs/10petabytes.jpg" alt="10petabytes" /></p>
<p>To avoid falling in the cargo-cult trap, I have learned to do the following:</p>
<ul>
<li>Focus on the problem first, not the solution. Don’t pick any tool until you have fully understood what you are trying to achieve or solve. Don’t give up solving the actual problem and make it all about learning and using the shiny new tech.</li>
<li>Keep it simple. It’s an over-used term, but software developers <em>still</em> just don’t get it. Keep. It. Simple.</li>
<li>If you are leaning towards something that Twitter or Google uses, do your homework and understand the real reasons why they picked that technology.</li>
<li>When thinking of growth, understand that the chances of your startup growing to be the size of Facebook are slim to none. Even if your odds are huge, is it really worth all this effort to set-up a ‘world-class foundation’ now vs doing it later?</li>
<li>Weigh operational burden and complexity. Do you really need multi-region replication for fault-tolerance in return of making your DevOps life 2x more difficult?</li>
<li>Be selfish. Do you want to be woken up in the middle of night because something, somewhere stopped working? There is nothing wrong with learning the new JavaScript framework from last week. Create a new project and publish it on GitHub. Don’t build production systems just because you like it.</li>
<li>Think about people who’d have to live with your mess. Think about your <strong>legacy</strong>. Do people remember you as the guy who built rock-solid systems or someone who left a crazy mess behind?</li>
<li>Share your ideas with experts and veterans and let them <strong>criticize</strong>. Identify people in other teams who you respect and who’d disagree freely.</li>
<li>Don’t jump to conclusions on the results of quick experiments. <em>HelloWorld</em> prototypes of anything is easy. Real-life is very different from <em>HelloWorld</em>.</li>
</ul>
<hr />
<p>We discussed YAGNI in this post, but it is generally applied in the context of writing software, design patterns, frameworks, ORMs, etc. Things that are in control of one person.</p>
<blockquote>
<p>YAGNI is coding what you need, as you need, refactoring your way through.</p>
</blockquote>
<p>Back to the guy I interviewed. It’s highly unlikely, even for a small startup, that a software developer in the trenches was allowed to pick Hadoop, Kafka and microservices architecture. Cargo cult practices usually start from someone higher up the ranks. A tech leader who may be a very smart engineer, but very bad at making rational decisions. Someone who probably spends way too much time reading blogs and tries very hard keep up with the Amazon or Google way of building software.</p>
<hr />
<p>2017 is almost over. <strong>NoSQL has matured</strong>. MongoDB is on its way out. DynamoDB is actually a very solid product and is maturing really well. <strong>RDBMS systems didn’t die</strong>. One can argue they are actually doing pretty good. StackOverflow is powered by <em>just</em> <a href="https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/" rel="nofollow">4 Microsoft SQL Servers</a>. <a href="https://eng.uber.com/mysql-migration/" rel="nofollow">Uber runs on MySQL</a>.</p>
<hr />
<p>You may have great reasons to use MapReduce or SOA. What matters is <em>how</em> you arrive at your decision. Whether by careful, sane thought or by jumping on the bandwagon and cargo-cult’ing.</p>
<p>As my professor said: “<em>Pick the right tool for the job</em>.” I’ll also add: <strong>don’t build Formula One cars when you need a Corolla</strong>.</p>
Caching Strategies and How to Choose the Right One2017-08-11T00:00:00+00:00https://codeahoy.com/2017/08/11/caching-strategies-and-how-to-choose-the-right-one<p><strong>👉 Read First: <a href="/2022/04/03/cache-invalidation/">A Brief Overview of Caching</a></strong></p>
<p>Caching is one of the easiest ways to increase system performance. Databases can be slow (yes even the NoSQL ones) and as you already know, speed is the name of the game.</p>
<p><img src="https://codeahoy.com/img/blogs/speed-matters.jpg" alt="speed-matters" /></p>
<!--more-->
<p>If done <em>right</em>, caches can reduce response times, decrease load on database, and save costs. There are several strategies and choosing the <em>right</em> one can make a big difference. Your caching strategy depends on the data and <strong>data access patterns</strong>. In other words, how the data is written and read. For example:</p>
<ul>
<li>is the system write heavy and reads less frequently? (e.g. time based logs)</li>
<li>is data written once and read multiple times? (e.g. User Profile)</li>
<li>is data returned always unique? (e.g. search queries)</li>
</ul>
<p>A caching strategy for Top-10 leaderboard system for mobile games will be very different than a service which aggregates and returns user profiles. Choosing the right caching strategy is the key to improving performance. Let’s take a quick look at various caching strategies.</p>
<h2 id="cache-aside">Cache-Aside</h2>
<p>This is perhaps the most commonly used caching approach, at least in the projects that I worked on. The cache sits on the <em>side</em> and the application <strong>directly</strong> talks to both the cache and the database. There is no connection between the cache and the primary database. All operations to cache and the database are handled by the application. This is shown in the figure below.</p>
<p><img src="https://codeahoy.com/img/cache-aside.png" alt="cache-aside" /></p>
<p>Here’s what’s happening:</p>
<ol>
<li>The application first checks the cache.</li>
<li>If the data is found in cache, we’ve <em>cache hit</em>. The data is read and returned to the client.</li>
<li>If the data is <strong>not found</strong> in cache, we’ve <em>cache miss</em>. The application has to do some <strong>extra work</strong>. It queries the database to read the data, returns it to the client and <strong>stores</strong> the data in cache so the subsequent reads for the same data results in a cache hit.</li>
</ol>
<h4 id="use-cases-pros-and-cons">Use Cases, Pros and Cons</h4>
<p>Cache-aside caches are usually general purpose and work best for <strong>read-heavy workloads</strong>. <em>Memcached</em> and <em>Redis</em> are widely used. Systems using cache-aside are <strong>resilient to cache failures</strong>. If the cache cluster goes down, the system can still operate by going directly to the database. (Although, it doesn’t help much if cache goes down during peak load. Response times can become terrible and in worst case, the database can stop working.)</p>
<p>Another benefit is that the data model in cache can be different than the data model in database. E.g. the response generated as a result of multiple queries can be stored against some request id.</p>
<p>When cache-aside is used, the most common write strategy is to write data to the database directly. When this happens, cache may become inconsistent with the database. To deal with this, developers generally use time to live (TTL) and continue serving stale data until TTL expires. If data freshness must be guaranteed, developers either <strong><a href="/2022/04/03/cache-invalidation/">invalidate the cache entry</a></strong> or use an appropriate write strategy, as we’ll explore later.</p>
<h2 id="read-through-cache">Read-Through Cache</h2>
<p>Read-through cache sits in-line with the database. When there is a cache miss, it loads missing data from database, populates the cache and returns it to the application.</p>
<p><img src="https://codeahoy.com/img/read-through.png" alt="read-through" /></p>
<p>Both cache-aside and read-through strategies load data <strong>lazily</strong>, that is, only when it is first read.</p>
<h4 id="use-cases-pros-and-cons-1">Use Cases, Pros and Cons</h4>
<p>While read-through and cache-aside are very similar, there are at least two key differences:</p>
<ol>
<li>In cache-aside, the application is responsible for fetching data from the database and populating the cache. In read-through, this logic is usually supported by the library or stand-alone cache provider.</li>
<li>Unlike cache-aside, the data model in read-through cache cannot be different than that of the database.</li>
</ol>
<p>Read-through caches work best for <strong>read-heavy</strong> workloads when the same data is requested many times. For example, a news story. The disadvantage is that when the data is requested the first time, it always results in cache miss and incurs the extra penalty of loading data to the cache. Developers deal with this by ‘<em>warming</em>’ or ‘pre-heating’ the cache by issuing queries manually. Just like cache-aside, it is also possible for data to become inconsistent between cache and the database, and solution lies in the write strategy, as we’ll see next.</p>
<h2 id="write-through-cache">Write-Through Cache</h2>
<p>In this write strategy, data is first written to the cache and then to the database. The cache sits in-line with the database and writes always go <em>through</em> the cache to the main database. This helps cache maintain consistency with the main database.</p>
<p><img src="https://codeahoy.com/img/write-through.png" alt="write-through" /></p>
<p>Here’s what happens when an application wants to write data or update a value:</p>
<ol>
<li>The application writes the data directly to the cache.</li>
<li>The cache updates the data in the main database. When the write is complete, both the cache and the database have the same value and the cache always remains consistent.</li>
</ol>
<h4 id="use-cases-pros-and-cons-2">Use Cases, Pros and Cons</h4>
<p>On its own, write-through caches don’t seem to do much, in fact, they introduce extra write <strong>latency</strong> because data is written to the cache first and then to the main database (two write operations.) But when paired with read-through caches, we get all the benefits of read-through and we also get data <strong>consistency</strong> guarantee, freeing us from using cache invalidation (assuming ALL writes to the database go through the cache.)</p>
<p><a href="https://aws.amazon.com/dynamodb/dax/">DynamoDB Accelerator (DAX)</a> is a good example of read-through / write-through cache. It sits inline with DynamoDB and your application. Reads and writes to DynamoDB can be done through DAX. (Side note: If you are planning to use DAX, please make sure you familiarize yourself with <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.consistency.html">its data consistency model</a> and how it interplays with DynamoDB.)</p>
<h2 id="write-around">Write-Around</h2>
<p>Here, data is written directly to the database and only the data that is read makes it way into the cache.</p>
<h4 id="use-cases-pros-and-cons-3">Use Cases, Pros and Cons</h4>
<p>Write-around can be combine with read-through and provides good performance in situations where data is written once and read less frequently or never. For example, real-time logs or chatroom messages. Likewise, this pattern can be combined with cache-aside as well.</p>
<h2 id="write-back-or-write-behind">Write-Back or Write-Behind</h2>
<p>Here, the application writes data to the cache which stores the data and acknowledges to the application <em>immediately</em>. Then later, the cache writes the data <em>back</em> to the database.</p>
<p>This is very similar to to Write-Through but there’s one crucial difference: In Write-Through, the data written to the cache is <strong>synchronously</strong> updated in the main database. In Write-Back, the data written to the cache is <strong>asynchronously</strong> updated in the main database. From the application perspective, writes to Write-Back caches are <em>faster</em> because only the cache needed to be updated before returning a response.</p>
<p><img src="https://codeahoy.com/img/write-back.png" alt="write-back" /></p>
<p>This is sometimes called write-behind as well.</p>
<h4 id="use-cases-pros-and-cons-4">Use Cases, Pros and Cons</h4>
<p>Write back caches improve the write performance and are good for <strong>write-heavy</strong> workloads. When combined with read-through, it works good for mixed workloads, where the most recently updated and accessed data is always available in cache.</p>
<p>It’s resilient to database failures and can tolerate some database downtime. If batching or coalescing is supported, it can reduce overall writes to the database, which decreases the load and <strong>reduces costs</strong>, if the database provider charges by number of requests e.g. DynamoDB. Keep in mind that <strong>DAX is write-through</strong> so you won’t see any reductions in costs if your application is write heavy. (When I first heard of DAX, this was my first question - DynamoDB can be very expensive, but damn you Amazon.)</p>
<p>Some developers use Redis for both cache-aside and write-back to better absorb spikes during peak load. The main disadvantage is that if there’s a cache failure, the data may be permanently lost.</p>
<p>Most relational databases storage engines (i.e. InnoDB) have write-back cache enabled by default in their internals. Queries are first written to memory and eventually flushed to the disk.</p>
<h3 id="summary">Summary</h3>
<p>In this post, we explored different caching strategies and their pros and cons. In practice, carefully evaluate your goals, understand data access (read/write) patterns and choose the best strategy or a combination.</p>
<p>What happens if you choose wrong? One that doesn’t match your goals or access patterns? You may introduce additional latency, or at the very least, not see the <em>full benefits</em>. For example, if you choose <em>write-through/read-through</em> when you actually should be using <em>write-around/read-through</em> (written data is accessed less frequently), you’ll have useless junk in your cache. Arguably, if the cache is big enough, it may be fine. But in many real-world, high-throughput systems, when memory is never big enough and server costs are a concern, the right strategy, matters.</p>
<p>I hope you enjoyed this post. Let me know in the comments section below which type of caching strategies you used in your projects. Until next time.</p>
Basics of Java Garbage Collection2017-08-06T00:00:00+00:00https://codeahoy.com/2017/08/06/basics-of-java-garbage-collection<blockquote>
<ul>
<li><strong>Knock, knock</strong>.</li>
<li>Who’s there?</li>
<li><em>…long GC pause…</em></li>
<li><strong>Java</strong>.</li>
</ul>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/wiseguy.jpg" alt="wiseguy-eh" class="center-image" /></p>
<p>It’s an old joke from the time when Java was <em>new</em> and <em>slow</em> compared to other languages. Over time, Java became a lot <a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/java-gcc.html"><strong>faster</strong></a>. Today it powers many real-time applications with hundreds of thousands of concurrent users. These days, the <strong>biggest impact on Java’s performance comes from its garbage collection</strong>. Fortunately, in many cases, it can be tweaked and optimized to improve performance.</p>
<!--more-->
<p>For most applications, the default settings of the JVM work fine. But when you start noticing performance issues caused by garbage collection and giving more heap memory isn’t possible, you need to tune and optimize the garbage collection. For most developers, it’s a <strong>chore</strong>. It requires patience, good knowledge of how garbage collection works and an understanding of application’s behavior. This post is a high-level overview of Java’s garbage collection with some examples of troubleshooting performance issues.</p>
<p>Let’s get started.</p>
<p>Java ships with <strong>several</strong> garbage collectors. More specifically, these are different <em>algorithms</em> that run in their own <em>threads</em>. Each works differently and has pros and cons. The most important thing to keep in mind is that all garbage collectors <strong>stop the world</strong>. That is, your application is put on hold or paused, as the garbage is collected and taken out. The main difference among the algorithms is <em>how</em> they stop the world. Some algorithms sit completely <strong>idle until the garbage collection is absolutely needed</strong> and then pause your application for a long period while others <strong>do most of their work concurrently</strong> with your application and thus need a <strong>shorter pause</strong> during stop the world phase. The best algorithm depends on your goals: are your <strong>optimizing for throughput</strong> where long pauses every now and then are tolerable or you are <strong>optimizing for low latency</strong> by spreading it out and having short pauses all along.</p>
<p>To <em>enhance</em> the garbage collection process, Java (HotSpot JVM, more accurately) divides up the heap memory into two <em>generations</em>: <strong>Young Generation</strong> and <strong>Old Generation</strong> (also called Tenured). There is also a <em>Permanent Generation</em>, but we won’t cover it in this post.</p>
<p><img src="https://codeahoy.com/img/blogs/hotspot-heap.png" alt="hotspot-heap" /></p>
<p><strong>Young generation</strong> is where <em>young</em> objects live. It’s further subdivided into the following areas:</p>
<ol>
<li>Eden Space</li>
<li>Survivor Space 1</li>
<li>Survivor Space 2</li>
</ol>
<p>By default, <strong>Eden is bigger</strong> than the two survivor spaces combined. On my Mac OS X with 64-bit HotSpot JVM, Eden takes about 76% of all the young generation space. All objects are first created here. When Eden is full, a <strong>minor</strong> garbage collection is triggered. All new objects are quickly inspected to check their eligibility for garbage collection. The ones that are dead, that is, aren’t referenced (ignoring reference strength for this discussion) from other objects are marked as dead and garbage collected. The <strong>surviving objects are moved to one of the empty ‘survivor spaces’</strong>. Which one of two survivor spaces? To answer this question, let’s discuss survivor spaces.</p>
<p>The reason for having two survivor spaces is to avoid <strong>memory fragmentation</strong>. Imagine if there was just one survivor space. While you are at it, also imagine survivor space as a contiguous array of memory. When young generation GC runs through the array, it identifies dead objects for removal. This would leave holes in memory where objects previously lived and <strong>compaction</strong> will be needed. To avoid compaction, HotSpot JVM just copies all surviving objects from the survivor space to the other (empty) survivor space so that there are no holes or empty spaces. While we are discussing compaction, please note that <em>old generation</em> garbage collectors (with the exception of CMS) perform compaction on the old generation section of the heap memory to defragment it.</p>
<p>In short, minor garbage collections (triggered when Eden is full) <strong>ping-pong</strong> live objects from Eden and one of the survivor space (known as the ‘from’ survivor space in logs) <em>to</em> the other (known as the ‘to’ survivor space). This happens until one of the following happens:</p>
<ol>
<li>Objects reach <em>maximum tenuring threshold</em>, in other words, have ping-pong’ed enough times that they aren’t young anymore,</li>
<li>There is no room in survivor space to receive newly birthed objects (We’ll revisit this later.)</li>
</ol>
<p>When this happens, objects are moved to the old generation. (There could be other conditions but I’m not aware of them.) Let’s try to understand with a real example. Suppose we have the following application that creates a few ‘long-lived objects’ during initialization and creates many short-lived during its operation. (E.g. a web server that allocates short-lived objects for each incoming request.)</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">createFewLongLivedAndManyShortLivedObjects</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">HashSet</span><span class="o"><</span><span class="nc">Double</span><span class="o">></span> <span class="n">set</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashSet</span><span class="o"><</span><span class="nc">Double</span><span class="o">>();</span>
<span class="kt">long</span> <span class="n">l</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">100</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="nc">Double</span> <span class="n">longLivedDouble</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Double</span><span class="o">(</span><span class="n">l</span><span class="o">++);</span>
<span class="n">set</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">longLivedDouble</span><span class="o">);</span> <span class="c1">// add to Set so the objects continue living outside the scope</span>
<span class="o">}</span>
<span class="k">while</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// Keep creating short-lived objects. Extreme but illustrates the point</span>
<span class="nc">Double</span> <span class="n">shortLivedDouble</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Double</span><span class="o">(</span><span class="n">l</span><span class="o">++);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Let’s enable garbage collection logs and other settings using the following JVM command line arguments:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">-Xmx100m</span> // Allow JVM 100 MB of heap memory
<span class="nt">-XX</span>:-PrintGC // Enable GC Logs
<span class="nt">-XX</span>:+PrintHeapAtGC // Enable GC logs
<span class="nt">-XX</span>:MaxTenuringThreshold<span class="o">=</span>15 // Allow objects to live <span class="k">in </span>the young space longer
<span class="nt">-XX</span>:+UseConcMarkSweepGC // Ignore <span class="k">for </span>now<span class="p">;</span> covered later
<span class="nt">-XX</span>:+UseParNewGC // Ignore <span class="k">for </span>now<span class="p">;</span> covered later
</code></pre></div></div>
<p>The application logs showing the state before and after garbage collection are as follows:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Heap <b>before</b> GC <span class="nv">invocations</span><span class="o">=</span>5 <span class="o">(</span>full 0<span class="o">)</span>:
par new <span class="o">(</span><u>young</u><span class="o">)</span> generation total 30720K, used 28680K
eden space 27328K, <b>100%</b> used
from space 3392K, <b>39%</b> used
to space 3392K, 0% used
concurrent mark-sweep <span class="o">(</span><u>old</u><span class="o">)</span> generation total 68288K, used <b>0K</b> <br/>
Heap <b>after</b> GC <span class="nv">invocations</span><span class="o">=</span>6 <span class="o">(</span>full 0<span class="o">)</span>:
par new generation <span class="o">(</span><u>young</u><span class="o">)</span> total 30720K, used 1751K
eden space 27328K, <b>0%</b> used
from space 3392K, <b>51%</b> used
to space 3392K, 0% used
concurrent mark-sweep <span class="o">(</span><u>old</u><span class="o">)</span> generation total 68288K, used <b>0K</b>
</code></pre></div></div>
<p>From the logs, we can see a few things. The first thing to notice is that there have been 5 minor garbage collections before this one (total of 6.) Eden was 100% used which triggered it. One of survivor space is 39% used and as such has some room available. After the garbage collection is over, we can see that Eden went back to 0% and survivor space increased to 59%. This means that <strong>live objects from Eden and survivor space were moved to second survivor space</strong> and dead one’s were garbage collected. How can we tell that some dead objects were collected? We can see that Eden is much larger than survivor space (27328K vs 3392K) and since survivor space size only slightly increased, a large number of objects must have been collected. The <strong>old generation space stayed completely empty</strong> before and after the garbage collection (Recall that the <em>tenuring threshold</em> was set to 15.)</p>
<p>Let’s try <strong>another</strong> experiment. Let’s run an application that is only creating <strong>short-lived</strong> objects in multiple threads. Based on what we’ve discussed so far, <strong>none of these objects should go to the old generation</strong>; minor garbage collection should be able to clean them up.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">createManyShortLivedObjects</span><span class="o">()</span> <span class="o">{</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="no">NUMBER_OF_THREADS</span> <span class="o">=</span> <span class="mi">100</span><span class="o">;</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="no">NUMBER_OF_OBJECTS_EACH_TIME</span> <span class="o">=</span> <span class="mi">1000000</span><span class="o">;</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">i</span><span class="o"><</span><span class="no">NUMBER_OF_THREADS</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-></span> <span class="o">{</span>
<span class="k">while</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="o">;</span> <span class="n">i</span><span class="o"><</span><span class="no">NUMBER_OF_OBJECTS_EACH_TIME</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="nc">Double</span> <span class="n">shortLivedDouble</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Double</span><span class="o">(</span><span class="mf">1.0d</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">sleepMillis</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}).</span><span class="na">start</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>For this example, I’m gave the JVM only <strong>10 MB</strong> of memory. Let’s look at the GC logs.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Heap <b>before</b> GC <span class="nv">invocations</span><span class="o">=</span>0 <span class="o">(</span>full 0<span class="o">)</span>:
par new <span class="o">(</span><u>young</u><span class="o">)</span> generation total 3072K, used 2751K
eden space 2752K, 99% used
from space 320K, 0% used
to space 320K, 0% used
concurrent mark-sweep <span class="o">(</span><u>old</u><span class="o">)</span> generation total 6848K, used <b>0K</b> <br/>
Heap <b>after</b> GC <span class="nv">invocations</span><span class="o">=</span>1 <span class="o">(</span>full 0<span class="o">)</span>:
par new generation <span class="o">(</span><u>young</u><span class="o">)</span> total 3072K, used 318K
eden space 2752K, 0% used
from space 320K, 99% used
to space 320K, 0% used
concurrent mark-sweep <span class="o">(</span><u>old</u><span class="o">)</span> generation total 6848K, used <b>76K</b>
</code></pre></div></div>
<p><strong>Not what we predicted</strong>. We can see that this time, the <strong>old generation received objects right after the first minor</strong> garbage collection. We know that these objects are short-lived and tenuring threshold is set to 15 and this is the first collection. What happened is the following: the application created a large number of objects which filled up Eden space. Minor garbage collection ran and tried to collect garbage. However, most of these short-lived objects were <strong>active</strong> during the GC, i.e. were being referenced from a live thread and being processed. The young generation garbage collector had <strong>no choice but to push these objects to the old generation</strong>. This is bad because the objects that got pushed to the old generation were <strong>prematurely aged</strong> and can only be cleaned up by old generation’s major garbage collection which usually takes more time. With a particular GC algorithm that we’ll cover later, <em>CMS</em>, major GC is triggered when the old generation memory is 70% full. This default value can be changed with the <code class="language-plaintext highlighter-rouge">-XX:CMSInitiatingOccupancyFraction=70</code> argument.</p>
<p>How to prevent premature aging of short-lived objects? There are several ways. One theoretical way is to estimate the number of active short-lived objects and size the young generation appropriately. Let us make the following changes:</p>
<ul>
<li>Young Generation by default is 1/3 of the total heap. Let’s change this using the -<code class="language-plaintext highlighter-rouge">XX:NewRatio=1</code> which gives young generation more memory (~3.4 MB compared to the 3.0 MB the last time.)</li>
<li>Also increase the survivor space ratio using the <code class="language-plaintext highlighter-rouge">-XX:SurvivorRatio=1</code> argument. (~1.6MB each compared to 0.3 MB the last time.)</li>
</ul>
<p>The problem was fixed. After 8 minor garbage collections, the old generation space was still empty.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Heap <b>before</b> GC <span class="nv">invocations</span><span class="o">=</span>7 <span class="o">(</span>full 0<span class="o">)</span>:
par new generation total 3456K, used 2352K
eden space 1792K, 99% used
from space 1664K, 33% used
to space 1664K, 0% used
concurrent mark-sweep generation total 5120K, used <b>0K</b> <br/>
Heap <b>after</b> GC <span class="nv">invocations</span><span class="o">=</span>8 <span class="o">(</span>full 0<span class="o">)</span>:
par new generation total 3456K, used 560K
eden space 1792K, 0% used
from space 1664K, 33% used
to space 1664K, 0% used <span class="o">[</span>
concurrent mark-sweep generation total 5120K, used <b>0K</b>
</code></pre></div></div>
<p>This is in no way an exhaustive method of tuning garbage collection. I’m simply trying to demonstrate the steps involved. For real applications, optimum settings are found as a result of trial and error with different settings. For example, we could have also fixed the problem by <strong>doubling</strong> the total heap memory size.</p>
<h2 id="garbage-collection-algorithms">Garbage Collection Algorithms</h2>
<p>Now that we have covered generations, let’s look at garbage collection algorithms. HotSpot JVM comes with several algorithms for young and old generations. At a high level, there are three general types of collection algorithms, each with its own <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/collectors.html">performance characteristic</a>:</p>
<blockquote>
<p><strong>serial collector</strong> uses a single thread to perform all garbage collection work, which makes it relatively efficient because there is no communication overhead between threads. It is best-suited to single processor machines -XX:+UseSerialGC.</p>
<p><strong>parallel collector</strong> (also known as the throughput collector) performs minor collections in parallel, which can significantly reduce garbage collection overhead. It is intended for applications with medium-sized to large-sized data sets that are run on multiprocessor or multithreaded hardware.</p>
<p><strong>concurrent collector</strong> performs most of its work concurrently (for example, while the application is still running) to keep garbage collection pauses short. It is designed for applications with medium-sized to large-sized data sets in which response time is more important than overall throughput because the techniques used to minimize pauses can reduce application performance.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/gc-compared.png" alt="gc-compared" /></p>
<p>HotSpot JVM allows you to configure separate GC algorithms for young and old generations. But you can only <strong>pair up compatible</strong> algorithms. For example, you cannot pair up <em>Parallel Scavenge</em> for young generation collector with <em>Concurrent Mark Sweep</em> for old generation collector because they are not compatible. To make it easier for you, I was going to make an infographic to show which garbage collectors are compatible, however, luckily I searched first and found one created by JVM engineer, <a href="https://blogs.oracle.com/jonthecollector/our-collectors">Jon Masamitsu</a>.</p>
<p><img src="https://codeahoy.com/img/blogs/gc-collectors-pairing.jpg" alt="gc-collectors-pairing" /></p>
<blockquote>
<ol>
<li>“Serial” is a stop-the-world, copying collector which uses a single GC thread.</li>
<li><strong>“Parallel Scavenge”</strong> is a stop-the-world, copying collector which uses multiple GC threads.</li>
<li><strong>“ParNew”</strong> is a stop-the-world, copying collector which uses multiple GC threads. It differs from “Parallel Scavenge” in that it has enhancements that make it usable with CMS. For example, “ParNew” does the synchronization needed so that it can run during the concurrent phases of CMS.</li>
<li>“Serial Old” is a stop-the-world, mark-sweep-compact collector that uses a single GC thread.</li>
<li><strong>“CMS”</strong> (Concurrent Mark Sweep) is a mostly concurrent, low-pause collector.</li>
<li><strong>“Parallel Old”</strong> is a compacting collector that uses multiple GC threads.</li>
</ol>
</blockquote>
<p>Concurrent Mark Sweep (<em>CMS</em>), paired with <em>ParNew</em>, works really well for server-side applications processing live requests from clients. I have been using it with ~ 10GB of heap memory and it keeps response times steady and GC pauses are short. Some developers I know use Parallel collectors (<em>Parallel Scavenge</em> + <em>Parallel Old</em>) and are happy with results.</p>
<p>One important thing to know about the CMS is that there have been <strong><a href="http://openjdk.java.net/jeps/291">calls to deprecate</a></strong> it and it will probably happen in Java 9 :’( Oracle recommends that the new concurrent collector, the <a href="http://docs.oracle.com/javase/7/docs/technotes/guides/vm/G1.html">Garbage-First</a> or the <strong>G1</strong>, introduced first with Java, be used instead:</p>
<blockquote>
<p>The G1 collector is a server-style garbage collector, targeted for multi-processor machines with large memories. It meets garbage collection (GC) pause time goals with high probability, while achieving high throughput.</p>
</blockquote>
<p><strong>G1</strong> works on both old and young generation. It is optimized for larger heap sizes (>10 GB). I’ve not experienced G1 collector first-hand and developers in my team are still using CMS, so I can’t yet compare the two. A quick online search reveals benchmarks showing <a href="http://blog.novatec-gmbh.de/g1-action-better-cms/">CMS outperforming</a> <a href="https://dzone.com/articles/g1-vs-cms-vs-parallel-gc">G1</a>. I’d tread carefully, but G1 should be fine. It can be enabled with:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">-XX</span>:+UseG1GC
</code></pre></div></div>
<p>Hope you found this post useful. Until next time.</p>
Message Batching to Increase Throughput and Reduce Costs2017-08-03T00:00:00+00:00https://codeahoy.com/2017/08/03/message-batching-to-increase-throughput-and-reduce-costs<p>A while ago, I was working on a backend system on the AWS cloud. Individual services in the system communicated by exchanging <em>asynchronous messages</em> with each other using <a href="https://aws.amazon.com/sqs/">Amazon SQS</a>. During early stages of development, we ran small load tests and found that CPU use was high and we would need more servers to handle the load. (Estimated peak load was 50,000 requests per second.) Services were generating lots of small messages every second and the profiler showed that threads responsible for handling and sending messages to SQS, <strong>one message at a time</strong>, were using CPU more than they should. This was affecting performance and throughput.</p>
<!--more-->
<p>To overcome this challenge, we took a page from Amazon’s <a href="http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/throughput.html">best practices advice</a>, and introduced <strong>batching and buffering</strong> on senders to group multiple messages and send as one batch. This simple fix <strong>increased the throughput</strong>, <strong>cut down SQS costs</strong> (SQS <a href="https://aws.amazon.com/sqs/pricing/">pricing</a> is by <em>number</em> of messages), and had <strong>very little impact on latency</strong>. <em>Win, win, win</em>. The <em>custom</em> batching algorithm was simple: batch up to 15 messages or wait up to a maximum of 50 milliseconds. If either 15 messages arrive quickly to form a batch or 50 milliseconds elapse, the batch is sent out. These figures (batch size and maximum time) were established after trial and error, tuned for the best throughput and latency. The application was multi-threaded with hundreds of active threads, so the system allowed <strong>multiple concurrent</strong> batches to be active at the same time to reduce thread blocking.</p>
<p>The only downside is that if the application crashes after retrieving batch from the queue but <strong>before processing</strong> messages in it, we can <strong>lose</strong> some or all of the messages. To deal with complete message loss, the system can implement <strong>retries</strong> and resend messages upon timeout. Clients will see 2x response times, but at least get their response back.</p>
<p>If you are building applications which <em>generate</em> and exchange a <strong>lot of messages</strong> in <strong>short periods</strong>, batching can increase <em>throughput</em>, <em>performance</em> and in the case of SQS, <em>reduce costs</em>. I recommend using message batching for SQS even if only for cost reasons.</p>
<p>Until next time.</p>
Amazon DynamoDB Auto Scaling2017-07-29T00:00:00+00:00https://codeahoy.com/2017/07/29/at-last-amazon-adds-dynamoDB-auto-scaling<p>Amazon DynamoDB supports <strong><a href="https://aws.amazon.com/about-aws/whats-new/2017/06/announcing-amazon-dynamodb-auto-scaling/">Auto Scaling</a></strong> which is a fantastic feature.</p>
<p>When enabled, Auto Scaling adjusts <strong>read and write capacities</strong> of DynamoDB tables (and global secondary indexes) automatically <em>based on the demand</em>. If you haven’t used DynamoDB before, you might be wondering why is this important? Before Auto Scaling, the users were required to provide <strong>fixed capacities</strong> for their tables. These capacities were static and didn’t respond to traffic demands. This was problematic because:</p>
<ol>
<li>The application performance and high-availability was compromised whenever the utilization exceeded the provisioned throughput. When this happened, DynamoDB throttled requests, which resulted in loss of data or poor user experience.</li>
<li>Cost control was poor at best. DynamoDB charges you by how much you provision. You end up paying the <strong>full cost</strong> by provisioned amounts, even if you <strong>use less</strong> than what you provisioned. In other words, if you <strong>overprovision</strong> for peak load, you’ll pay extra during the non-peak hours, when the capacity isn’t being full utilized. On the other hand, if you <strong>underprovision</strong>, the performance of your application will suffer due to <strong>throttling</strong> when the load exceeds the provisioned capacity.</li>
</ol>
<!--more-->
<p>Many real-world use cases are difficult to predict in advance and fluctuations are common. Speaking of traffic fluctuations, they are prevalent and hard to deal with. In mobile gaming, the traffic can increase suddenly if Apple or Google features the game, or the publisher runs a massive ad campaign. A website can suddenly see a large number of visitors if an article is picked up by a major newspaper or news aggregator site.</p>
<ul id="markdown-toc">
<li><a href="#what-is-dynamodb-auto-scaling" id="markdown-toc-what-is-dynamodb-auto-scaling">What is DynamoDB Auto Scaling?</a> <ul>
<li><a href="#historical-perspective" id="markdown-toc-historical-perspective">Historical Perspective</a></li>
<li><a href="#auto-scale-target-utilization" id="markdown-toc-auto-scale-target-utilization">Auto Scale Target Utilization</a></li>
</ul>
</li>
<li><a href="#dynamodb-auto-scaling-pricing" id="markdown-toc-dynamodb-auto-scaling-pricing">DynamoDB Auto Scaling Pricing</a></li>
<li><a href="#dynamodb-auto-scaling-vs-on-demand" id="markdown-toc-dynamodb-auto-scaling-vs-on-demand">DynamoDB Auto Scaling vs On Demand</a> <ul>
<li><a href="#when-to-use-auto-scaling-vs-on-demand" id="markdown-toc-when-to-use-auto-scaling-vs-on-demand">When to use Auto Scaling vs On Demand?</a></li>
</ul>
</li>
<li><a href="#dynamodb-and-dax" id="markdown-toc-dynamodb-and-dax">DynamoDB and DAX</a></li>
</ul>
<h2 id="what-is-dynamodb-auto-scaling">What is DynamoDB Auto Scaling?</h2>
<p>DynamoDB Auto Scaling feature lets you automatically manage throughput in response to your traffic patterns without throttling your users. You can assign minimum and maximum provisioned capacities to a table (or Global Secondary Index). When the traffic goes up, the table will increase its provisioned read and write capacity. When the traffic goes down, it will decrease the throughput so that you don’t pay for unused provisioned capacity.</p>
<h3 id="historical-perspective">Historical Perspective</h3>
<p>Before Auto Scaling feature was introduced in 2017 (fun fact: I was there at the re:Invent when they announced it), it was challenging for Engineering and DevOps teams to keep track of and keep up with manually adjusting provisioning in response to external events. I worked in a team where this was a huge challenge. If you overprovision, you can end up paying thousands of dollars for unused capacity. People came up with custom solutions like Lambda scripts, sometimes even adding a cache to reduce calls to DynamoDB during peak load (Caching has other uses as well.) Then there were third-party solutions as well. <a href="https://github.com/sebdah/dynamic-dynamodb">Dynamic DynamoDB</a> was arguably the most widely used. It’s open-source and has additional features such as time-based auto scaling and <a href="https://dynamic-dynamodb.readthedocs.io/en/latest/granular_scaling.html">granular</a> control over how many read and write capacity units to add or remove during auto scale up or down. However, now that this feature is built-in to DynamoDB and enabled by default, I doubt that third-party solutions will be used very much, if at all. <em>So long and thanks for all the fish</em>.</p>
<p>Auto Scaling is an optional feature and you must enable it explicitly using the ‘Auto Scaling’ settings in the AWS Console, as shown in the image below. You may also enable this using <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.HowTo.SDK.html" rel="nofollow">AWS SDK</a>.</p>
<p><img src="https://codeahoy.com/img/blogs/dynamodb-autoscale.png" alt="Setting Dynamodb auto scale settings in AWS console" height="253px" width="700px" /></p>
<h3 id="auto-scale-target-utilization">Auto Scale Target Utilization</h3>
<p>In the image above, I’m specifying that the table utilization (<strong>target utilization</strong>) is kept at 80%. If the utilization exceeds this threshold, auto-scaling will increase the provisioned read capacity. Likewise, if the utilization falls below 80%, it will decrease the capacity to reduce unnecessary costs. In other words, auto-scaling uses an algorithm to adjust the provisioned throughput of the table so that the actual utilization remains at or near your target utilization.</p>
<p>Target utilization is useful to deal with sudden increases in traffic. Why? Because auto-scaling doesn’t kick in and take effect instantly. It takes time (~5 to 10 minutes to take effect.) If you set target utilization to a very number e.g. 100%, the benefit you’d get is that you’d always pay for used capacity. The drawback is that if the demand on the table increases suddenly, you’ll likely see throttling errors while DynamoDB auto scaling adjusts capacities and increase to match demand after you run out of your <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html#bp-partition-key-throughput-bursting" rel="nofollow">burst capacity</a> quota. As a general rule of thumb, 70%-75% is a good number.</p>
<h2 id="dynamodb-auto-scaling-pricing">DynamoDB Auto Scaling Pricing</h2>
<p>There is no additional cost to use DynamoDB Auto Scaling. You only pay for the provisioned capacity, which auto scaling will adjust for you in response to traffic patterns. If you do it right, auto scaling will reduce costs and ensure you don’t pay for unused capacity. One important thing to keep in mind is not to set the maximum provisioned capacity too high. Because if you do, you risk scaling all the way up to that capacity and be responsible for costs. Estimate what’s the maximum cost you could bear, and then set maximum capacity accordingly.</p>
<h2 id="dynamodb-auto-scaling-vs-on-demand">DynamoDB Auto Scaling vs On Demand</h2>
<p>DynamoDB On-Demand is a related feature for managing throughput of a table. With On-Demand, you do not do any capacity planning or provisioning. You don’t specify read or write capacities anywhere. None, Nada, Zilch. Instead, you only pay for what you use. On-Demand was introduced in 2018, a year after Auto Scaling was launched.</p>
<p>On-Demand sounds too good to be true. If you think about it, why would anyone ever use Auto Scaling if On-Demand is available? Is there a catch?</p>
<p>Yes, there’s a catch. It’s <strong>cost</strong>. On-Demand is more expensive than Auto Scaling (provisioned capacity.)</p>
<ul>
<li>On-Demand: Cost for 1 million write requests is <strong>$1.25</strong></li>
<li>Provisioned Capacity: Cost for 1 million write requests is roughly <strong>$0.20</strong></li>
</ul>
<p>On Demand is about 6 times more expensive than provisioned capacity.</p>
<h3 id="when-to-use-auto-scaling-vs-on-demand">When to use Auto Scaling vs On Demand?</h3>
<p>Auto Scaling works great when you have predictable traffic patterns or if you can approximate based on historical data. The other reason for using provisioned capacity is cost. At my last company, DynamoDB was our main database and cost control was a constant challenge. As much as I loved On-Demand, I couldn’t justify the 6x cost. Besides, Auto Scale works really well in most scenarios. The only time we had issues was when there was a sudden traffic spike (in game event) but that was a problem across the infrastructure not just DynamoDB. Another downside of On-Demand is the unexpected bill due to huge traffic increase. I’m not aware of a way to control this at the moment.</p>
<p>On the other hand, if the traffic volume is low and the traffic patterns are unknown, On-Demand might make perfect sense. For example, if you estimate doing a <strong>max</strong> of 1 million requests a month, $1.25 vs. $0.18 a month doesn’t really make that much of a difference.</p>
<h2 id="dynamodb-and-dax">DynamoDB and DAX</h2>
<p><a href="https://twitter.com/jeffbarr">Jeff Barr</a> wrote a <a href="https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/">blog post</a> introducing the new feature and has screenshots of CloudWatch graphs. Check it out if you want to learn more. One key takeaway from his blog is that auto scaling isn’t ideal for <strong>bursts</strong>:</p>
<blockquote>
<p>DynamoDB Auto Scaling is designed to accommodate request rates that vary in a somewhat predictable, generally periodic fashion. If you need to accommodate unpredictable bursts of reading activity, you should use Auto Scaling in combination with <a href="https://aws.amazon.com/dynamodb/dax/">DAX</a> (read <a href="https://aws.amazon.com/blogs/aws/amazon-dynamodb-accelerator-dax-in-memory-caching-for-read-intensive-workloads/">Amazon DynamoDB Accelerator (DAX) – In-Memory Caching for Read-Intensive Workloads</a> to learn more).</p>
</blockquote>
<p>If you enjoyed this post, please share using the social sharing icons below. It’ll help CodeAhoy grow and every share counts. Thank you!</p>
AI Is Not Magic. How Neural Networks Learn2017-07-28T00:00:00+00:00https://codeahoy.com/2017/07/28/ai-is-not-magic-how-neural-networks-learn<p>In my previous <a href="https://codeahoy.com/2017/07/27/ai-winter-is-coming/">blog post</a>, I claimed that “<em>AI is not magic.</em>” In this post, my goal is to discuss how neural networks learn, and show that AI isn’t a crystal ball or magic, just science and some very slick mathematics. I’ll keep this very <em>high level</em>.</p>
<p>Let’s start with a <em>hypothetical scenario</em>. Suppose we are building an app to <a href="https://www.theverge.com/2017/6/26/15876006/hot-dog-app-android-silicon-valley">identify hot dogs</a>. Take a picture and the app will tell you if it’s a hotdog or not. Total App Store domination.</p>
<p><img src="https://codeahoy.com/img/blogs/nothodogapp.jpg" alt="hotdog" /></p>
<!--more-->
<p>To recognize images, we choose to implement a popular machine learning algorithm called the <em>neural network</em>. (In this hypothetical scenario) This decision was made after reading an online article which talked about how neural networks can learn to recognize objects by training on lots of labelled examples. Once trained, it can start identifying images it has never seen before. We go ahead and obtain a <strong>training set</strong> of <em>6000</em> images gathered from online sources. <em>1000</em> images of different types of hotdogs: New York vs Chicago, ketchup, no ketchup, hotdogs on a grill, etc. The other <em>5000</em> images are of various non-hotdog objects: shoes, hamburgers, burrito, human legs. Now all that remains is to build our neural network.</p>
<p>To understand neural networks, we must first understand its elementary building block: the artificial neuron.</p>
<blockquote>
<p>An artificial neuron takes <strong>one ore more inputs</strong> and produces a <strong>single output</strong>.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/perceptron.png" alt="perceptron" /></p>
<p>Looks familiar? It looks a lot like <strong><a href="https://en.wikipedia.org/wiki/Logic_gate">logic gates</a></strong>, which are elementary building blocks of digital circuits. The similarity ends there. Unlike logic gates, neurons can have several inputs and can <strong>change output</strong> for the same input values. This is possible because neurons have <strong>weights</strong> associated with each input, which is multiplied with the input value. These weights allow neurons to rate how important each input is. E.g. if the second input to a neuron isn’t very important, neuron can assign it a weight close to 0 essentially cancelling it out. Neurons also have <strong>biases</strong> which controls how easy it is to get neuron to output or <em>fire</em>. If the bias is huge, the neuron will fire very easily. I may have simplified, but what I have just described is the most basic type of neuron called the <strong>perceptron</strong>. In real world, we use more complex types of neurons.</p>
<p>An individual neuron is nifty, but it is <strong>not enough for sophisticated decision making</strong>. We want to group a bunch of these neurons together to form a <strong>neural network</strong> where different neurons get <em>tuned</em> on different aspects of the image. For example, a subset of neurons may only fire when they detect sliced bun, while others may fire when they detect sausage. All these sub decisions are weighed in the output layer before the final judgement is passed. For example, if the sausage neurons aren’t firing but the sliced bun one’s are, the output layer will classify the image as <em>not</em> hotdog because it could be something else with a sliced bun, e.g. Philly cheese steak sandwich. Output layer will assign more weight to the input coming from sausage neurons because there’s a very high chance that the image is of hotdog when the sausage neurons are firing.</p>
<blockquote>
<p>When we connect bunch of neurons together, we get a <strong>neural network</strong>.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/neural_network.svg.png" alt="neural_network" /></p>
<p>This particular type of neural network is called feedforward because information just flows in one direction and there are no cycles. Let’s also quickly talk about how we’ll input images to our neural network. Suppose that all images are the same resolution, say 128 by 128 pixels. We represent each image as a 2d array where each element of the array contains color information for the corresponding pixel. This 2d array of pixel color values is fed to the input layer of our neural network which contains <code class="language-plaintext highlighter-rouge">128*128=16384</code> neurons.</p>
<p>Back to weights and biases. The ability to change output for the same input values is arguably the most important feature of an artificial neuron (and by extension, the neural network) and it is the <strong>key to its learning</strong>. The goal of learning is to <em>find</em> the best <strong>combination of weights and biases</strong> for the neural network. Let’s express this objective formally so we can measure it and give it a name: <strong>cost function</strong>:</p>
<blockquote>
<p>Cost Function (weights,biases) <strong>=</strong> (# of images incorrectly identified) <strong>÷</strong> (Total # of images)</p>
</blockquote>
<p>Great. Now we can measure the performance with a clear objective: <strong>find weights and biases which minimize the cost function</strong>. How to find <em>best</em> weights and biases? One way is to just <strong>randomly</strong> pick them, run neural network over entire training data (6,000 images) and calculate the cost function which is the ratio of images incorrectly identified and the total number of images. Keep repeating until we’ve found a low enough value of the cost function that we like. Let’s say we want to stop when the neural network has a success rate of 99%. When we reach this goal, the combination of weights and biases <em>reflect some similarity between hotdogs</em> and our neural network can identify never seen before hotdog images. Let’s try it out. To keep it simple, suppose we are only doing this for 2 weights and biases.</p>
<p><strong>Iteration 1</strong>: Weight1 = 1, Weight2 = 3, Bias1 = 1, Bias2 = 4. Let’s say it correctly classifies 300 out of 500 images of hot dog correctly. We’ll say the error rate is 200/500 = 40%</p>
<p><strong>Iteration 2</strong>: Weight1 = 14, Weight2 = 6, Bias1 = 3, Bias2 = 0.2. This time is classifies 250 out of 500 images of hot dog correctly. The error rate is 200/500 = 50%, worst than last time.</p>
<p><strong>Iteration 3</strong>: Choose weights and biases randomly again, roll the dice and error rate is 35%. Hmmm…</p>
<p>…</p>
<p>I hope you can see why this approach won’t work. There is no science, even heuristics to it. If the luck isn’t on our side, we could keep iterating until the end of time. And this is with just 2 weights and biases. In reality, it’s not uncommon for neural networks to have hundreds or even thousands of weights and biases. The shot in the dark approach just isn’t practical, it’s impossible.</p>
<p><img src="https://codeahoy.com/img/blogs/deep_neural_network.png" alt="deep_neural_network" /></p>
<p>But thanks to mathematics and <strong>calculus</strong>, we have a way of finding optimum weights and biases quicker, or in other words, minimizing the cost function in a scientific manner. This is done using an algorithm called the ‘Gradient Descent.’ The first time it runs, the algorithm picks up random weights and biases, but in <strong>subsequent iterations, it doesn’t chooses randomly but in a calculated manner to minimize the cost function even further</strong>. It keeps iterating until it finds the minimum value of the cost function. At this point, our neural network is said to have been ‘trained’ and it could start classifying pictures that it hasn’t seen before.</p>
<p>Of course, there is much, much more happening under the hood. If you are interested, Andrew Ng has an <a href="https://www.youtube.com/watch?v=yFPLyDwVifc">excellent lecture</a> on gradient descent and I encourage you to watch the video so understand it in more detail.</p>
<p>Let’s revisit our cost function:</p>
<blockquote>
<p>Cost Function (weights,biases) <strong>=</strong> (# of images incorrectly identified) <strong>÷</strong> (Total # of images)</p>
</blockquote>
<p>This cost function is too simple and isn’t practical for gradient descent. Gradient descent cannot make incremental updates to weights and biases to minimize it. We need a <strong>smooth</strong> cost function. In practice, people usually use a quadratic cost function also known as the <em>mean squared error</em> (MSE).</p>
<p><img src="https://codeahoy.com/img/blogs/mse.svg" alt="mse" /></p>
<p>Here <strong>Ŷ</strong> is the prediction of the learning algorithm, <strong>Y</strong> is the actual value and <strong>n</strong> is the size of the training data. It’s a measure of how close predictions are to actual values. Data scientists often multiply the cost function by 1/2 because that way the squared term is easier to cancel when taking derivative of the function. Speaking of derivative, gradient descent minimizes the cost function by taking its <a href="https://en.wikipedia.org/wiki/Derivative">derivative</a> (slope at that point) and moving in the direction where it is decreasing.</p>
<p><img src="https://codeahoy.com/img/blogs/gradient_descent.png" alt="gradient_descent" /></p>
<p>Unlike the first simplified cost function we created, MSE is a <strong>smooth</strong> cost function. In each iteration, gradient descent updates weights and biases to move downhill to the lowest point where the cost function is minimum. As a side note, there are <a href="https://stats.stackexchange.com/questions/154879/a-list-of-cost-functions-used-in-neural-networks-alongside-applications">many types of cost functions</a> but MSE works well in many cases.</p>
<p>There is one more concept that you should know: the <strong>learning rate</strong>. The learning rate is the amount by which gradient descent updates weights and biases in each step. If the learning rate is too low, the algorithm may take a very long time to find the minimum. If the learning rate is too high, the algorithm may overshoot and miss the minimum (i.e. jump to the other side of the hill.) In practice, several parameters like learning rate and others that we’ll see in later posts needs to be adjusted to get the best results and performance.</p>
<p>Gradient descent, or rather its variations and several optimizations (as we’ll see in later posts), remains in wide use in machine learning algorithms like linear regression and neural networks. In linear regression, it gives us the line that best fits the data we are modeling; in neural networks, it gives us the best weights and biases.</p>
<p><em>Is AI magic?</em> Magic of matrix multiplication and gradient descent, sure. But not smart enough to <a href="https://www.extremetech.com/extreme/252781-elon-musk-warns-us-ai-destroy-world">take over and destroy the world</a>. Many CEO’s and executives <strong>overestimate</strong> the power of AI because of the unrealistic picture painted by the media. AI is extremely powerful and is proving itself with positive ROI in many domains, but it also has its <em>limitations</em> and it is not an off-the-shelf crystal ball. You should understand what the AI can and cannot do and then incorporate into your overall strategy. A good <a href="https://hbr.org/2016/11/what-artificial-intelligence-can-and-cant-do-right-now">rule of thumb</a>:</p>
<blockquote>
<p>If a typical <strong>person can do a mental task with less than one second of thought</strong>, we can probably <strong>automate it using AI</strong> either now or in the near future. (-Andrew Ng)</p>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/machinelearningcansandcannots.png" alt="machinelearningcansandcannots" /></p>
<p>See you next time.</p>
AI Winter is Coming?2017-07-27T00:00:00+00:00https://codeahoy.com/2017/07/27/ai-winter-is-coming<h2 id="what-is-ai-winter">What is AI Winter?</h2>
<p><a href="https://en.wikipedia.org/wiki/AI_winter">AI winter</a> is a period of ‘reduced funding and interest in the field of artificial intelligence.’ AI winters are preceded by hype cycles and ambitious claims of what AI can do. Money into research and AI companies pours in and expectations are inflated. But it doesn’t last and after a while, pessimism takes over the community and spreads to press, investors and government. Budgets are slashed, funding is stopped and AI research virtually dries up. There have been two AI winters: first one in the 1970’s and the last one was in the 1980’s.</p>
<!--more-->
<h2 id="is-another-ai-winter-coming">Is Another AI Winter Coming?</h2>
<p>We are there again as far the hype is concerned. There has been no shortage of buzz around AI in the past few years. Everyone, everywhere is talking about AI and how it can predict revenues, increase sales, create chatbots who can do natural language conversations like just like real customer service people.</p>
<p>Is the boom going to end soon? <a href="https://twitter.com/AndrewYNg">Andrew Ng</a>, chief scientist at Baidu research and a prominent figure in the AI community <a href="https://www.technologyreview.com/s/603062/ai-winter-isnt-coming/">doesn’t think so</a>. The advancements in computing power and availability of huge amounts of training data are providing “the fuel required to make emerging AI techniques feasible.”</p>
<blockquote>
<p>“There are multiple experiments I’d love to run if only we had a 10-x increase in performance,” Ng adds. For instance, he says, instead of having various different image-processing algorithms, greater computer power might make it possible to build a single algorithm capable of doing all sorts of image-related tasks.</p>
</blockquote>
<p>Addressing concerns about hype, Andrew Ng says:</p>
<blockquote>
<p>“There’s definitely hype,” adds Ng, “but I think there’s such a strong underlying driver of real value that it won’t crash like it did in previous years.”</p>
</blockquote>
<p>Andrew Ng have good reasons to be optimistic. So let’s look at the other side of the coin and some recent AI misses. Consider the case of <a href="https://www.ibm.com/watson/">IBM Watson</a> and claims that it’s going to <a href="https://www.forbes.com/sites/matthewherper/2017/02/19/md-anderson-benches-ibm-watson-in-setback-for-artificial-intelligence-in-medicine/#72856f203774">eradicate cancer</a>:</p>
<blockquote>
<p>It was one of those amazing “we’re living in the future” moments. In an October 2013 press release, IBM declared that MD Anderson, the cancer center that is part of the University of Texas, “is using the <strong>IBM Watson cognitive computing system for its mission to eradicate cancer</strong>.”</p>
<p>Well, now that future is past. The partnership between IBM and one of the world’s top cancer research institutions is falling apart. <strong>The project is on hold</strong>, MD Anderson confirms, and has been since late last year. MD Anderson is actively requesting bids from other contractors who might replace IBM in future efforts. And a scathing report from auditors at the University of Texas says <strong>the project cost MD Anderson more than $62 million and yet did not meet its goals</strong>.</p>
</blockquote>
<p>In one of the many Watson ad campaigns IBM ran, Watson tells Bob Dylan that he has read all his lyrics and that the meaning of Dylan’s music is all about ‘<em>time passing and love fading</em>.’</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/hVZeR-RmhcM" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Did Watson get it right? Roger Schank <a href="http://www.rogerschank.com/fraudulent-claims-made-by-IBM-about-Watson-and-AI">disagrees</a>:</p>
<blockquote>
<p>Really? I am a child of the 60s’ and I remember Dylan’s songs well enough. Ask anyone from that era about Bob Dylan and no one will tell you his main theme was “love fades”. He was a protest singer, and a singer about the hard knocks of life. He was part of the anti-war movement. Love fades? That would be a dumb computer counting words. How would Watson see that many of Dylan’s songs were part of the anti-war movement? Does he say anti-war a lot? He probably never said it in a song.</p>
</blockquote>
<p>IBM Watson is a good product and I won’t be so harsh on it. Their ads and marketing efforts got lots of people and companies interested in AI. A colleague of mine told me that his CEO rolled his eyes (metaphorically speaking) when he told him to use machine learning to make predictions about customer behavior and revenue a few years ago. While he will never know for sure if it was because of IBM PR, but the CEO changed his mind recently and now thinks of AI as crystal ball that will transform his business and bring millions and millions of dollars in revenue.</p>
<p>AI is not magic. It’s just science and mathematics.</p>
<p>I used Watson to build a gaming chatbot with AI that could have conversations with users on a small number of topics. Watson uses <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a> and Machine Learning algorithms to understand user messages and requires lots of training. We weren’t able to get it to converse at a level of a 2 or 3-year old and it was easily confused. In the end, we decided to use a cloud based API called <a href="https://wit.ai/">Wit.ai</a> which gave us pretty much the same results as Watson.</p>
<p>Chatbots with AI replacing humans to provide customer service is a long, long shot. That doesn’t mean that chatbots or AI isn’t useful. Even if chatbots can increase 1%-2% of customer service efficiency for a major enterprise, it will result in huge savings. Chatbots can troubleshoot simple issues or prescreen users before passing them off to a live agent. Richard Socher, chief scientist at Salesforce <a href="https://www.technologyreview.com/s/603062/ai-winter-isnt-coming/">said</a>:</p>
<blockquote>
<p>“If we were to make the 150,000 companies that use Salesforce 1 percent more efficient through machine learning, you would literally see that in the GDP of the United States,” he says.</p>
</blockquote>
<p>Hype wasn’t the only reason for the last two AI winters, although it did play a big hand. It initially fueled the interest but failed to live up to the expectations and didn’t provide real value to corporations and government. While the hype has outpaced the reality once again, it is different this time. Aside from obvious beneficiaries like Google, Amazon, Microsoft and Tesla who are sitting on mountains of data and resources, <strong>medium to small sized companies are applying AI into direct actions to grow revenue and see positive ROI</strong>. In the mobile gaming domain, publishers and studios are experimenting with <a href="http://www.scientificrevenue.com/">dynamic pricing</a> for in-app purchases. For a large gaming company, even a slight improvement over traditional models such as segmented or A/B pricing, boosts revenue. Likewise, mobile operators are using <a href="https://codeahoy.com/2017/02/19/cluster-analysis-using-k-means-explained/">machine learning to predict when prepaid subscribers</a> will next recharge and use this prediction to grant or deny loans.</p>
<p>Is another AI winter on the horizon? I don’t think so. In fact I believe the interest, investment and research will continue to grow. It is already providing immense utility to not just the big corporations or companies building self-driving cars, but to companies of all sizes and in different industries.</p>
<p>However, we must curb our expectations a little. AI won’t out-think humans anytime soon, understand deep meaning of music, eradicate cancer, or replace customer service people entirely, but it will instead provide real value in many domains and incremental improvements.</p>
<p><strong><a href="/2017/07/28/ai-is-not-magic-how-neural-networks-learn/">AI is not magic</a></strong> and the hype will die down, but the next AI winter will be more like <strong>California</strong> winter, not Canadian.</p>
<p>See you next time.</p>
Fix Employee Weaknesses or Focus on Their Strengths?2017-07-26T00:00:00+00:00https://codeahoy.com/2017/07/26/fix-employee-weaknesses-or-focus-on-their-strengths<p>In “<a href="https://www.amazon.com/First-Break-All-Rules-Differently/dp/1531865208">First, Break All the Rules: What the World’s Greatest Managers Do Differently</a>” the authors, Marcus Buckingham and Curt Coffman, have put together their observations from more than 80,000 Gallup interviews they conducted with various leaders and managers over a period of 25 years. The book is full of excellent insight into what <em>great managers do and don’t do</em> and debunks several traditional management myths. One such myth is that people are capable of almost anything if they work hard enough or <strong>everyone has unlimited potential</strong>. According to the authors, this is a complete fallacy and while it is an uplifting thought, it is far from reality.</p>
<!--more-->
<blockquote>
<p>“There once lived a scorpion and a frog.</p>
<p>The scorpion wanted to cross the pond, but, being a scorpion, he couldn’t swim. So he scuttled up to the frog and asked: “Please, Mr. Frog, can you carry me across the pond on your back?”</p>
<p>“I would,” replied the frog, “but, under the circumstances, I must refuse. You might sting me as I swim across.”</p>
<p>“But why would I do that?” asked the scorpion. ”</p>
<p>“It is not in my interests to sting you, because you will die and then I will drown.”</p>
<p>Although the frog knew how lethal scorpions were, the logic proved quite persuasive. Perhaps, felt the frog, in this one instance the scorpion would keep his tail in check. So the frog agreed. The scorpion climbed onto his back, and together they set off across the pond. Just as they reached the middle of the pond, the scorpion twitched his tail and stung the frog. Mortally wounded, the frog cried out: “Why did you sting me? It is not in your interests to sting me, because now I will die and you will drown.”</p>
<p>“I know,” replied the scorpion as he sank into the pond. “<strong>But I am a scorpion. I have to sting you. It’s in my nature</strong>.”</p>
</blockquote>
<p>In this old parable, the frog made a fatal mistake in believing that scorpion’s nature will change.</p>
<blockquote>
<p>Great managers reject this out of hand. They remember what the frog forgot: that each individual, like the scorpion, is true to his unique nature … They know that there is a <em>limit</em> to how much remolding they can do to someone. But they don’t bemoan these differences and try to grind them down. Instead they <em>capitalize</em> on them. They try to help each person become more and more of who he already is.</p>
</blockquote>
<p>Under the same situation, different people react differently according to their nature. People are motivated differently. For example, I worked with a software developer who was very competitive by nature and his productivity would go through the roof when he heard that someone else on the team did it better or faster. That was his trigger. If a task carries too much risk, it is best assigned to a person who is meticulous than to someone who is a risk taker.</p>
<p>Everyone has some talents which the authors define as ‘<em>recurring patterns that could be applied productively.</em>’ Willpower is a talent. So is empathy and competitiveness. They key is to select and hire for talent and <strong>cast in the right role</strong>. This includes identifying an individual’s talents and assigning responsibilities which maximizes the strengths and neutralizes weaknesses.</p>
<blockquote>
<p>Casting for talent is one of the unwritten secrets to the success of
great managers. On occasion it can be as simple as knowing that your
aggressive, ego-driven salesperson should take on the territory that re
quires a fire to bel it beneath it. And, by contrast, your patient, relation
ship-building salesperson should be offered the territory that requires
careful nurturing.</p>
</blockquote>
<p>This may sound like common knowledge but all too often <strong>hiring managers put excessive emphasis on skills and experience over talent</strong>. Skill or how-to’s of a role can be taught. Talent cannot be taught. A Java software developer can learn Python, but may not become a good marketer. An aggressive, ego-driven person generally makes a poor team player but put that person in a situation that requires a fire to be lit under it, and that person might just become a rockstar.</p>
<blockquote>
<p>People don’t change that much.
Don’t waste time trying to put in what was left out.
Try to draw out what was left in.
That is hard enough.</p>
</blockquote>
<p>This is the essence of the ‘<a href="http://www.strengthsfinder.com/home.aspx">focus on strengths</a>’ school of thought. There is <em>some</em> <a href="http://www.gallup.com/businessjournal/442/four-disciplines-sustainable-growth.aspx">scientific evidence</a> to support this theory:</p>
<blockquote>
<p>Beyond a person’s mid-teens, that unique network of synaptic connections, in which some are strong and robust and others non-existent, does not change significantly. This means that a person’s recurring patterns of thought, of feeling and of behavior do not change significantly. If he is empathic when he is hired, he will stay empathic. If he is impatient for action when he is hired, he will stay impatient.</p>
</blockquote>
<p>There is also criticism of the ‘focus on strengths’ based approach. Dr. Tomas Chamorro-Premuzic suggests that focusing too much on our strengths <a href="https://hbr.org/ideacast/2016/01/stop-focusing-on-your-strengths.html">can be counterproductive</a>:</p>
<blockquote>
<p>it’s important to understand that even the smartest, brightest, and most brilliant individuals have a dark side. They have certain elements of their personality, of their typical behaviors, that are quite counterproductive. And if those tendencies are left unchecked, no matter how smart, competent, and talented they are, their careers at risk of derailing.</p>
<p><strong>Think of an employee or an individual who is very driven and ambitious. If we developed their ambition and drive even further, they might just become greedy</strong>. Or somebody who is very socially skilled, if they develop their social skills even further, they might become almost Machiavellian and manipulative. People who are very creative can become odd and eccentric, and people who are already a little bit confident, if we make them even more confident, they might become arrogant or overconfident.</p>
</blockquote>
<p>I generally agree with the idea of focusing on strengths as too many managers focus on <em>irrelevant</em> weaknesses or non-talents of their reports. There isn’t enough time to change an employee’s nature even a little or to give birth to a new talent. Does this mean we should completely ignore weaknesses? If the weakness is relevant and it is affecting performance, the manager must determine if the weakness is trainable (i.e. missing skill), whether the person is casted in the wrong role or if the person can be paired up with someone who has complementary strengths. Either way, poor performance should be tackled head on as soon as possible.</p>
<p>Until next time.</p>
Tweaking TCP for Real-time Applications: Nagle's Algorithm and Delayed Acknowledgment
2017-03-19T00:00:00+00:00https://codeahoy.com/2017/03/19/tweaking-tcp-for-real-time-applications-nagle-algorithm-and-delayed-acknowledgment<p><a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> is a <em>complex</em> protocol.</p>
<p>Don’t get me wrong. It is a marvelous piece of engineering that gives us the reliable data transmission guarantee that other protocols don’t provide. <em>Reliable</em> data transmission between two devices on the internet is no walk in the park and TCP uses a <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Flow_control">lot</a> <a href="https://en.wikipedia.org/wiki/TCP_congestion_control">of</a> <a href="https://en.wikipedia.org/wiki/TCP_window_scale_option">magic</a> under the hood to make things happen. Generally, it does a fine job of <a href="https://codeahoy.com/2016/05/06/good-abstractions-have-fewer-leaks/">abstracting away low level details</a> and its default settings work fine for most general purpose use cases. However, once in a while, things don’t go according to plan and we need to pop open the hood and do some tweaking. It is in these situations, that some knowledge of TCP comes in very handy.</p>
<p><!--more--></p>
<p>By default, <strong>TCP uses two buffering techniques</strong> to optimize and minimize overhead for general purpose applications. However, if you are building applications that require real-time message delivery for small messages (e.g. chat or control messages), you must have some knowledge of these techniques.</p>
<ol>
<li>Nagle’s algorithm</li>
<li>TCP delayed acknowledgment</li>
</ol>
<p>Let’s look at them in more detail.</p>
<h2 id="nagles-algorithm">Nagle’s Algorithm</h2>
<p>If there’s no congestion, TCP tacks on a header and sends data out as soon as it gets it from the application. If the application is generating a lot of small messages, the headers can add a lot of <strong>overhead</strong>: TCP/IP headers are 40-byte, so 1-byte of data is sent as 41-byte packet on the network. A computer programmer named John Nagle came up with an algorithm to reduce the overhead by combining many small messages into a single message. Nagle’s algorithm, named after its inventor, <strong>is a technique to make TCP more efficient by reducing the number of packets that are sent over the network</strong>. Here’s the <a href="(https://en.wikipedia.org/wiki/Nagle's_algorithm)">pseudo code</a> for the algorithm:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if there is new data to send
if the window size >= MaximumSegmentSize and available data is >= MaximumSegmentSize
send complete MaximumSegmentSize segment now
else
if there is unconfirmed data still in the pipe
enqueue data in the buffer until an acknowledge is received
else
send data immediately
end if
end if
end if
</code></pre></div></div>
<p>So what the algorithm is saying is that if the data to be sent is <em>smaller</em> than the <a href="https://en.wikipedia.org/wiki/Maximum_segment_size">maximum segment size (MSS)</a> (~ 1.4 KB), it is sent immediately <em>ONLY</em> if TCP has received acknowledgment for all the data that was previously sent. Another way to see it: if the newly generated data on the sender is small and its TCP is waiting for the receiver to acknowledge receipt of data that is in flight, Nagle’s algorithm will tell TCP to buffer the data and it won’t be sent immediately.</p>
<p>Nagle’s algorithm works great for most TCP applications like video streaming that produce data at a very high rate which exceeds the MSS very quickly and has to be sent out. But for multiplayer gaming servers, it creates performance issues. Tens of thousands of clients connect to a front-end server which sits between clients and backend gaming services. Even though the overall throughput is very high, only a small amount of data (a chat message roughly 100 bytes) is available for each client/socket and hence the data is buffered using Nagle’s algorithm.</p>
<p>So if you are building <strong>TCP applications that expect responses to arrive in real-time, Nagle’s algorithm will result in poor performance and higher latency</strong>. Nagle’s algorithm is turned on by default at the system level, but you can disable it for you application using <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/tcp_nodelay_and_small_buffer_writes">TCP_NODELAY</a> socket option.</p>
<p>However, Nagle’s algorithm is not the only culprit when it comes to higher message latencies for real-time application. As we’re about to see, another type of buffering on the receiver side, prevents acknowledgments for received data to be sent out immediately.</p>
<h2 id="tcp-delayed-acknowledgment">TCP Delayed Acknowledgment</h2>
<p>TCP delayed acknowledgment is an optimization technique to combine multiple acknowledgments (ACKs) into a single response to reduce the overhead. Upon receiving data, receivers (using delayed acknowledgment) don’t send acknowledgment right away but instead wait for a few hundred milliseconds (200ms to 500ms) so it can be sent bundled together with any other acknowledgments or data that it might generate in during that window.</p>
<p><strong>Nagle’s algorithm (sender side) and TCP delayed acknowledgement (receiver side) is a double whammy for real-time applications</strong>: receivers wait a few hundred milliseconds before acknowledging senders. Without receiving acknowledgement, senders keep on buffering small packets until they receive acknowledgement or the buffered data exceeds maximum segment size, which can take a long time (> 200ms) if the data is small. This <em>double buffering</em> wreaks havoc on real-time applications and increases message latency. Nagle’s algorithm would probably perform better without the TCP delayed acknowledgment, which can be disabled using the <a href="https://linux.die.net/man/7/tcp">TCP_QUICKACK</a> socket option. However, it’s not always possible to control behavior of client devices. Besides, disabling Nagle’s algorithm on the server for real-time applications does the job.</p>
<p>Until next time!</p>
Cluster Analysis Using K-means Explained2017-02-19T00:00:00+00:00https://codeahoy.com/2017/02/19/cluster-analysis-using-k-means-explained<p>Clustering or <a href="https://en.wikipedia.org/wiki/Cluster_analysis">cluster analysis</a> is the process of <strong>dividing data into groups (clusters) in such a way that objects in the same cluster are more similar to each other than those in other clusters</strong>. It is used in data mining, machine learning, pattern recognition, data compression and in many other fields. In machine learning, it is often a starting point. In a machine learning application I built couple of years ago, we used clustering to divide six million prepaid subscribers into five clusters and then built a model for each cluster using <a href="https://en.wikipedia.org/wiki/Linear_regression">linear regression</a>. The goal of the application was to predict future recharges by subscribers so operators can make intelligent decisions like whether to grant or deny emergency credit. Another (trivial) application of clustering is for dividing customers into groups based on spending habits or brand loyalty for further analysis or to determine the best promotional strategy.</p>
<!--more-->
<p>There are various models and techniques for cluster analysis. When I first started, I was <em>mistakenly searching for ‘the best clustering model or technique.’</em> I wasn’t aware that there is no universal best algorithm and the choice depends on your requirements and the dataset. There are <a href="https://en.wikipedia.org/wiki/Cluster_analysis#Density-based_clustering">density-based</a>, <a href="https://en.wikipedia.org/wiki/HCS_clustering_algorithm">graph based</a> or <a href="https://en.wikipedia.org/wiki/Cluster_analysis#Centroid-based_clustering">centroid based</a> clustering models. We finally settled on a clustering technique called k-means. This blog post is a brain-dump of everything I’ve learned about clustering and k-means so far.</p>
<h1 id="k-means">K-means</h1>
<p>K-means is a very simple and widely used clustering technique. It divides a dataset into ‘<em>k</em>’ clusters. The ‘<em>k</em>’ must be supplied by the users, hence the name k-means. It is general purpose and the <a href="http://johnloeber.com/docs/kmeans.html">algorithm is straight-forward</a>:</p>
<blockquote>
<p>We call the process k-means clustering because we assume that there are <strong>k</strong> clusters, and each cluster is defined by its center point — its mean. To find these clusters, we use <strong>Lloyd’s Algorithm</strong>: we start out with <strong>k</strong> random centroids. A centroid is simply a datapoint around which we form a cluster. For each centroid, we find the datapoints that are closer to that centroid than to any other centroid. We call that set of datapoints its cluster. Then we take the mean of the cluster, and let that be the new centroid. We repeat this process (using the new centroids to form clusters, etc.) until the algorithm stops moving the centroids.[0] We do this in order to minimize the total sum of distances from every centroid to the points in its cluster — <em>that is our metric for how well the clusters split up the data</em>.</p>
</blockquote>
<p>Here’s an animation (from - broken link: http://simplystatistics.org/2014/02/18/k-means-clustering-in-a-gif/) showing how it works.</p>
<p><img src="https://codeahoy.com/img/kmeans/kmeans-animated.gif" alt="kmeans-animated" /></p>
<p>As far as its performance is concerned, k-means and its <a href="https://upcommons.upc.edu/bitstream/handle/2117/23414/R13-8.pdf">variants can usually process large datasets very quickly</a>, as long as the number of clusters isn’t very high.</p>
<h2 id="finding-k-number-of-clusters-using-the-elbow-method">Finding ‘k’: number of clusters using the elbow method</h2>
<p>If you know the number of clusters before hand, you have everything you need to apply k-means. However, in practice, it’s rare that the number of clusters in the dataset is known. When I’m modeling, I’m not sure if there are 3 clusters or 13 in the dataset. Luckily, there’s a technique called the ‘<a href="https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set">elbow method</a>’ that you can use to determine the number of clusters:</p>
<blockquote>
<p>One should choose a number of clusters so that adding another cluster doesn’t give much better modeling of the data. More precisely, if one plots the percentage of variance explained by the clusters against the number of clusters, the first clusters will add much information (explain a lot of variance), but at some point the marginal gain will drop, giving an angle in the graph. The number of clusters is chosen at this point, hence the “elbow criterion”.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/kmeans/kmeans-elbow.JPG" alt="kmeans-elbow" /></p>
<p>Silhouette analysis is another popular technique for <a href="http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html">finding the optimum number of clusters visually</a>, and the one we actually used in our application.</p>
<h2 id="k-means-limitations-and-weaknesses">K-means limitations and weaknesses</h2>
<p>Unfortunately, k-means has limitations and doesn’t work well with different types of clusters. It doesn’t do well when:</p>
<ol>
<li>the clusters are of unequal size or density.</li>
<li>the clusters are non-spherical.</li>
<li>there are outliers in the data.</li>
</ol>
<p>I have read that the dataset must have well-separated clusters in order for k-means to properly work. However, in practice, I have experienced k-means doing a pretty good job of diving data up into clusters, even when the clusters are not well separated or obvious. I will keep it off the list.</p>
<h3 id="1-the-clusters-are-of-unequal-size-or-density">1. The clusters are of unequal size or density</h3>
<p>K-means won’t identify clusters properly if clusters have an uneven size or density. To illustrate the point, I generated a 2d dataset.</p>
<p><img src="https://codeahoy.com/img/kmeans/uneven-unlabelled.png" alt="uneven-unlabelled" /></p>
<p><strong>I see 3 clusters</strong> of uneven densities. Do you?</p>
<p><img src="https://codeahoy.com/img/kmeans/uneven-unlabelled-manual.png" alt="uneven-unlabelled-manual" /></p>
<p>Let’s see how k-means does. I wrote a simple python script with scikit-learn and ran k-means with <em>k=3</em>.</p>
<p><img src="https://codeahoy.com/img/kmeans/uneven-applied.png" alt="uneven-applied" /></p>
<p>As it was expected, k-means messed up and couldn’t find naturally occurring clusters that we recognized in the dataset. It identified the larger cluster correctly but mixed the smaller ones. If you have datasets where naturally occurring clusters have unequal densities, try <a href="https://en.wikipedia.org/wiki/Cluster_analysis#Density-based_clustering">density based models</a>.</p>
<h3 id="2-the-clusters-are-non-spherical">2. The clusters are non-spherical</h3>
<p>Let’s generate a 2d dataset with non-spherical clusters.</p>
<p><img src="https://codeahoy.com/img/kmeans/kmeans-spherical-unlabelled.png" alt="kmeans-spherical-unlabelled" /></p>
<p>It’s how you look at it, but <strong>I see 2 clusters</strong> in the dataset. Let’s run k-means and see how it performs.</p>
<p><img src="https://codeahoy.com/img/kmeans/kmeans-spherical-applied.png" alt="kmeans-spherical-applied" /></p>
<p>Looking at the result, it’s obvious that k-means couldn’t correctly identify the clusters. If you have a similar dataset, try a hierarchical or density based algorithm like <a href="https://en.wikipedia.org/wiki/Spectral_clustering">spectral clustering</a> or <a href="https://en.wikipedia.org/wiki/DBSCAN">DBSCAN</a> that are better suited.</p>
<h3 id="3-there-are-outliers-in-the-data">3. There are outliers in the data.</h3>
<p>If outliers are present in the dataset, they can influence clusterings results and change the outcome. The dataset should be pre-processed before applying k-means to detect and remove any outliers. There are many techniques for <a href="https://web.archive.org/web/20191126130103/http://www.pmg.it.usyd.edu.au/outliers.pdf">outlier detection</a> <a href="https://pdfs.semanticscholar.org/49f3/d110f87ae245127d2e30049628785e95d23e.pdf">and</a> <a href="http://stackoverflow.com/questions/13989419/removing-outliers-from-a-k-mean-cluster">removal</a>. Unfortunately, discussing these techniques is beyond the scope of this post and my expertise.</p>
<h2 id="evaluating-k-means-results">Evaluating k-means results</h2>
<p>Let’s assume you have just applied k-means to a dataset. <strong>How do you tell if k-means did its job right and identified all clusters correctly?</strong> <em>If</em> your dataset is 2-dimensional and isn’t very large, you could probably plot the results and visually inspect to assess k-means, just like how I’ve been showing results to you so far. But what if your dataset is high-dimensional, as is often the case? We are trapped and can’t visualize data beyond 3-dimensions. So we need some way to evaluate result of k-means and there are a couple that I know of:</p>
<ol>
<li>Supervised evaluation</li>
<li>Unsupervised evaluation</li>
</ol>
<h3 id="1-supervised-evaluation">1. Supervised evaluation</h3>
<p>Supervised evaluation can be used to check the results of k-means when you have a pre-classified dataset available that can act as a benchmark. Typically, an expert would carefully process a small dataset assigning datapoints to clusters manually. This is the <strong>gold standard</strong> for evaluation and tells us how close the results are to the benchmark. Keep in mind that in order to use supervised evaluation, you’d need to be able to train k-means and then supply the pre-classified benchmarks to check expectations against actual results (not possible if you are using a standalone app like Weka or Tableau). This <a href="http://scikit-learn.org/stable/modules/clustering.html#homogeneity-completeness-and-v-measure">page describes how to use supervised evaluation</a> in scikit-learn, a popular python machine learning library.</p>
<h3 id="2-unsupervised-evaluation">2. Unsupervised evaluation</h3>
<p>Unsupervised evaluation doesn’t rely on external information. In our app, we used inter-cluster (<em>within</em> cluster sum of squares) and intra-cluster (<em>between</em> cluster sum of squares) variances to decide if the results were good enough. These are just fancy terms to describe <strong>cohesion</strong> of points that are within a cluster and <strong>separation</strong> between clusters. <strong>We want good cohesive clusters that are well separated from others</strong>.</p>
<p><img src="https://codeahoy.com/img/kmeans/cohesion-separartion.jpg" alt="cohesion-separartion" /></p>
<p>For more information on, this <a href="https://en.wikipedia.org/wiki/Cluster_analysis#Evaluation_and_assessment">wikipedia article</a> has more information and describes several algorithms.</p>
<p>That’s everything I have learned about k-means and clustering in general. If you have any comments or want to suggest improvements to this post, please leave them in the comments section below.</p>
Certificate Authorities - Do You Know Who You Trust?2017-02-18T00:00:00+00:00https://codeahoy.com/2017/02/18/certificate-authorities-do-you-know-who-you-trust<p>HTTPS (aka <em>HTTP over the secure <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> protocol</em>) provide a secure communication channel between web browsers and servers to guard against <a href="https://owasp.org/www-community/attacks/Manipulator-in-the-middle_attack">man-in-the-middle attacks</a>. Although researchers have identified and reported a <a href="https://drownattack.com/">few</a> <a href="http://heartbleed.com/">vulnerabilities</a>, TLS is still the best option out there and all <a href="https://codeahoy.com/2017/01/18/if-your-site-isnt-using-https-you-are-doing-it-wrong/">websites should be using it</a>.</p>
<!--more-->
<p>Arguably, the most famous TLS fiasco was not a vulnerability but an enormously miscalculated and incompetent attempt to increase ad revenues by a top selling laptop manufacturer. “<a href="http://www.slate.com/articles/technology/bitwise/2015/02/lenovo_superfish_scandal_the_result_of_evil_or_incompetence.html">Lenovo incident / scandal</a>” got its start when Lenovo thought it would be a brilliant idea to pre-install an adware (<a href="https://en.wikipedia.org/wiki/Superfish">Superfish</a>) on their laptops to inject ads on webpages, both encrypted and non-encrypted. To allow Superfish to view and alter encrypted traffic, they pre-loaded its <a href="https://en.wikipedia.org/wiki/Self-signed_certificate">self-signed</a>, <a href="https://en.wikipedia.org/wiki/Root_certificate">root certificate</a> on their laptops. By doing this, Lenovo let Superfish become man-in-the-middle and allowed it to view and alter traffic without the user ever knowing. If that was the end of the story, it might not have been all that terrible. However, by installing the self-signed root certificate, <strong>which used the same private key on all laptops, Lenovo exposed sensitive and confidential communication of their users to attackers</strong> or eavesdroppers connected to the same WiFi. Communication including emails, bank transactions, messages, and passwords were all exposed. After <a href="https://forums.lenovo.com/t5/Security-Malware/Potentially-Unwanted-Program-Superfish-VisualDiscovery/td-p/1794457">users complained</a> and there was public outcry, Lenovo finally acknowledged that it “<a href="http://www.pcworld.com/article/2886690/lenovo-cto-admits-company-messed-up-and-will-publish-superfish-removal-tool-on-friday.html">messed up</a>” and apologized to users for betraying their trust. It quickly dumped Superfish. Microsoft stepped in and provided an update of Windows Defender to remove Superfish.</p>
<p><img src="https://codeahoy.com/img/lenovo-super-fish.png" alt="lenovo-super-fish" /></p>
<p>As someone who purchased a <a href="http://www.techradar.com/reviews/pc-mac/laptops-portable-pcs/laptops-and-netbooks/lenovo-y50-1207894/review/2">Y50</a> around the same time (2014), I’m happy to report that Superfish is dead. (<em>So long, and no-thanks for the fish.</em>)</p>
<p>It is not always the incompetence of laptop vendors - user trust has been violated intentionally for <a href="https://thenextweb.com/insider/2015/04/02/google-to-drop-chinas-cnnic-root-certificate-authority-after-trust-breach/">malicious purposes</a> as well:</p>
<blockquote>
<p>a Chinese certificate authority issued valid security certificates for a number of domains, including Google’s, without their permission, which resulted in a major trust breach in the crypto chain.</p>
<p>CNNIC had delegated its authority to Egyptian intermediary MCS Holdings to issue the certificates in question and the company installed it in a man-in-the-middle proxy internally.</p>
</blockquote>
<p>If China feels far away, <a href="http://www.itpro.co.uk/security/25315/symantec-employees-fired-over-fake-security-certificates">Symantec fired its employees</a> after it discovered they issued fake google security certificates.</p>
<p>As a user, you can protect your communication and your privacy by <strong>never ignoring security warnings from your web browser</strong>. Web browsers, especially Chrome and Firefox, do a pretty good job of recognizing potential threats and warning users, so don’t skip through unless you are absolutely sure what you are doing. They also have a <strong>“<a href="https://www.chromium.org/Home/chromium-security/root-ca-policy#TOC-Removal-of-Trust">Removal of Trust</a>“</strong> policy where they would distrust a root certificate authority if it is compromised, even if it is trusted by the operating system.</p>
<p>See you next time!</p>
Testers Make Software Teams Highly Productive2017-02-17T00:00:00+00:00https://codeahoy.com/2017/02/17/testers-make-software-teams-highly-productive<p>To put it mildly, developers are not great at testing their own products. Bias, pride, wrong assumptions, lack of time, switching contexts, all play a role in making developers ineffective at testing their code. Most companies, especially startups, don’t fully understand the role of a tester. Very early on in my career, we made the <strong>mistake of hiring people to be testers who applied for a software developer position but weren’t good enough programmers</strong>. We paid for it in terms of software quality and over-worked team. It wasn’t until I worked with some great testers that I realized how effective and productive software teams become when they have great testers on board.</p>
<!--more-->
<p>The best testers or quality assurance engineers I’ve ever worked with weren’t developers. One was a DevOps guy and the other was a Network Engineer. They became testers coincidentally because they were very smart and tremendously effective at finding bugs, even when they tested the system as a black-box. They treated the whole exercise like puzzle solving. Here’s an example bug report from them:</p>
<blockquote>
<p>“Found an issue with API to retrieve all widgets right after registering as a new user. Requests kept timing out. Upon digging further, I discovered that the system is throwing NullPointerExceptions. This is happens because the <em>xyz</em> counter in the database wasn’t properly initialized in the previous step. Initialize the counter in the previous step and also catch all un-handled exceptions and return an error to the user.”</p>
</blockquote>
<p>Or this one.</p>
<blockquote>
<p>“Found an issue. The system throws exception when the user selects option 5. I checked the logs and found nothing. I ran wireshark to look at the request and response and found the issue to be caused by developers sending a unicode character in the request. Don’t send unicode characters and log error reasons which is field_10 in the <a href="/learn/xml/toc/">XML</a> payload.”</p>
</blockquote>
<p>These bug reports not only clearly identified the issue, but also the root cause and suggested a better course of action, thus refining the product and improving its quality.</p>
<p>Some organizations <strong>naively assume that automated tests written by developers can replace testers</strong>. Wrong, very wrong. I love automated tests. But a good tester finds deficiencies and suggests improvements that a developer or an automated suite may overlook:</p>
<blockquote>
<p><em>Tester</em>: When I pass a string instead of a number for the billing amount, the app doesn’t capture it. You need to check the type and ensure its a number.</p>
<p><em>Developer</em>: What do you mean? I wrote the test for it just last week and my end-to-end client did in fact receive the status code 4000 which means it’s an error.</p>
<p><em>Tester</em>: Yes, the error is returned. But the error comes from the database in the form of an exception when it tries to store a string where a number should go. This is inefficient because the database call is expensive.</p>
<p><em>Developer</em>: Yeah. That makes sense when I think a about it. I will get it fixed.</p>
</blockquote>
<p>Another great reason for having dedicated testers on the team is to provide <strong><a href="https://www.joelonsoftware.com/2010/01/26/why-testers/">positive reinforcement</a> and closure to developers</strong> who may otherwise be doubtful whether they are on the right track or not. Developers usually breathe a sigh of relief when their releases are certified by testers and are told that everything works as expected.</p>
<blockquote>
<p>A great tester gives programmers immediate feedback on what they did right and what they did wrong. Believe it or not, one of the most valuable features of a tester is providing positive reinforcement. There is no better way to <strong>improve a programmer’s morale, happiness, and subjective sense of well-being than <del>a La Marzocco Linea espresso machine</del> to have dedicated testers who get frequent releases from the developers, try them out, and give negative and positive feedback</strong>.</p>
</blockquote>
<p>Until next time.</p>
What Is Yak Shaving? Advice for Software Developers on Staying Focused2017-02-13T00:00:00+00:00https://codeahoy.com/2017/02/13/yak-shaving-the-less-you-do-the-better<p>Yak shaving is defined as:</p>
<blockquote>
<p>what you are doing when you’re doing some stupid, fiddly little task that bears no obvious relationship to what you’re supposed to be working on, but yet a chain of twelve causal relations links what you’re doing to the original meta-task.</p>
</blockquote>
<p>Fun-fact about the origin of the term ‘yak shaving’: The term was <a href="http://projects.csail.mit.edu/gsb/old-archive/gsb-archive/gsb2000-02-11.html">coined</a> at the MIT AI Lab in the 90s. Its scientists got inspiration from an episode of the <em>Ren and Stimpy</em> show called “Yak Shaving Day”.</p>
<p>A picture is worth a thousand words. A video, perhaps billions. Here’s a video of Malcolm originally wanting to change a lightbulb but ending up chasing down yaks.</p>
<div style="width:100%;height:0;padding-bottom:56%;position:relative;">
<iframe src="https://giphy.com/embed/llbEoVMhkLngWlzVVa" width="100%" height="100%" style="position:absolute" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe>
</div>
<p>To understand it more clearly, suppose you are required to perform a task. We’ll call it task A. As you start working on task A, it leads you to another task, e.g. task B. Task B leads you to Task C, and so on. Before you know it, you are working on Task Z, completely distracted from your original goal of completing task A.</p>
<p>Here’s an example dialog in the world of software development.</p>
<blockquote>
<p>Manager: “Did you fix the issue where we had to update the column name in our code because someone changed it in the DB? “</p>
<p>Software Developer: “Ah, not yet. I’m still working on it.”</p>
<p>Manager: “What happened? It was a one-line change.”</p>
<p>Software Developer: “As I looked into the code, I realized we were using a really old version of Hibernate. I tried to upgrade it but there were some breaking changes in the new version. They recommended switching to the Repository pattern so I refactored a few classes but now the DB is throwing errors. I’m debugging.”</p>
</blockquote>
<!--more-->
<p>In this fictional scenario, the developer ‘went down a rabbit hole,’ which had nothing to do with the original task of changing the column name in code. The key is staying focused and not letting distractions pull you away from the main goal. Assuming the developer was right that the code needed to be upgraded, he mixed the two unrelated tasks up. Stay focused, finish the original task. Re-evaluate and prioritize the second task of upgrading Hibernate version and then go back to it.</p>
<p>Moral of the story: Stay focused and stop chasing those yaks!</p>
<p>One thing I have always loved about startups is their ability to stay focused. Solo or even small teams of software developers at a startup can get things done much faster compared to their counterparts in large corporations because they have fewer yaks to shave. At a startup, you won’t hear: “I didn’t get any time to write code for the new feature because as I was creating the JIRA story, I realized that the epic needs to be split. I did that but then I had to move tasks and link them to the right macro on Confluence.”</p>
<!-- ![yak shaving gif](https://codeahoy.com/img/replacing-a-lightbulb-imgur.gif) -->
<p>I’m not suggesting that yak shaving in evil: sometimes, you have no choice but to go on side quests before you can reach your final destination. In fact, in larger companies, yak shaving is inevitable. Developers spend the vast majority of their time shaving yaks.</p>
<p>Ben Ramsey said it better: yak shaving “<a href="https://benramsey.com/blog/2015/11/yak-shaving/">isn’t just part of our jobs, it’s the entire job description.</a>” So the minute you start going down the yak shaving path, stop and ask yourself if shaving the yak is really necessary. <strong>The fewer yaks you have to shave, the faster you’ll get to your destination</strong>.</p>
<p>You might want to also familiarize yourself with <strong><a href="/2017/08/19/yagni-cargo-cult-and-overengineering-the-planes-wont-land-just-because-you-built-a-runway-in-your-backyard/">YAGNI which stands for “You Aren’t Gonna Need It.</a></strong>. See you next time.”</p>
Committing Teamicide by Micromanagement2017-02-08T00:00:00+00:00https://codeahoy.com/2017/02/08/committing-teamicide-by-micromanagement<h2 id="what-is-micromanagement">What is micromanagement?</h2>
<p>Micromanagement <a href="https://en.wikipedia.org/wiki/Micromanagement">is a</a> “<em>management style whereby a manager closely observes or controls the work of subordinates or employees</em>”.</p>
<p><img src="https://codeahoy.com/img/micromanagement-rsz.jpg" alt="micromanager picture" /></p>
<p>Micromanagement is bad. It hurts morale and works against making individuals or teams productive. An effective manager or a leader makes people use their brains instead of acting like mindless zombies who require constant babysitting and instructions. A micromanager is like a helicopter parent closely watching and monitoring the employees, which is very demoralizing especially to smart people.</p>
<!--more-->
<p>While some managers micromanage out of fear or job insecurity, most do it because <strong>they don’t trust</strong> their direct reports to carry out the tasks as well as themselves or they fear that without their supervision, mistakes will be made. This lack of trust forces the manager to become defensive and assume that the only way employees will do good work is if they are being constantly monitored and reviewed and they cannot be trusted to make decisions on their own. Employees reporting to a micromanager become cynical and sometimes even despise the manager. They start acting in their own self-interest which is counter-productive and inhibits team formation.</p>
<p>An acquaintance once asked for my help on a project that was stuck. The technical manager who was in charge of a small team (less than 10 people) was drowning in work. I quickly realized that I was dealing with a micromanager. Employees were disenchanted and had very low morale or motivation to do a good job. They didn’t care. The micromanager, through his actions, made it clear to the team that he had no trust in them. He had put elaborate processes in place so that nothing could get marked as “done” until he had reviewed it down to the tiniest details. He told me that the team sucks and he was brought in to basically kick ass and get the project done. It was amusing because his lack of trust had become a self-fulfilling prophecy: he was frustrated that people were producing such low quality work and that that he had to redo everything and was swamped with menial coding tasks so much that he didn’t have time for anything else.</p>
<p><strong>Chapter 20 of <a href="https://www.amazon.com/Peopleware-Productive-Projects-Teams-Second/dp/0932633439">Peopleware</a> is called Teamicide</strong> and describes a list of things to not do if you are trying to grow productive teams. Micromanagement is at the top of that list because <strong>if there’s one thing you cannot protect yourself against, it’s your own people’s incompetence</strong>:</p>
<blockquote>
<p>It makes good sense for you the manager to take a defensive posture in most areas of risk. If you must work with a piece of failure prone gear, you get a backup; […]</p>
<p>There’s one area, though, where defensiveness will always backfire: <strong>You can’t protect yourself against your own people’s incompetence. If your staff isn’t up to the job at hand, you will fail. Of course, if the people are badly suited to the job, you should get new people</strong>. But once you’ve decided to go with a given group, your best tactic is to trust them. Any. defensive measure taken to guarantee success in spite of them will only make things worse. It may give you some relief from worry in the short term, but it won’t help in the long run, and it will poison any chance for the team to jell.</p>
</blockquote>
<p>This is why companies should choose managers or leaders very wisely. Good leaders hire very, very carefully for both brains and cultural fit. They coach employees about their values and set goals and expectations clearly instead of holding hands all along the way. They show their employees that they are trusted to get the job done right and <a href="https://codeahoy.com/2016/04/12/let-them-own-it/">let them own it</a>. Good leaders don’t treat <a href="https://codeahoy.com/2016/04/14/mistakes-at-work-are-not-sins/">mistakes at work like sins</a>. They allow their employees to learn from their mistakes and develop skills and build knowledge. Trust is the key to growing productive teams.</p>
<p>There is however another problem on the other end of the spectrum: some managers fall into the opposite trap where they think that they can hire smart people who get things done and just disappear, leaving people on their own with vague goals and expectations. This style is called <strong>m<em>a</em>cromanagement, and it is equally as bad as m<em>i</em>cromanagement</strong>. You are not micromanaging if you provide direction, occasionally override a decision or help the team reach consensus. It requires the right balance.</p>
<p>In the end, it’s all about trust. Trust is a feeling and it takes time to build it. Trust build and jell teams - it makes people feel safe and when smart people feel safe and trusted, they do everything in their power to achieve the goal.</p>
htop Explained Visually2017-01-20T00:00:00+00:00https://codeahoy.com/2017/01/20/hhtop-explained-visually<p><a href="http://hisham.hm/htop/">htop</a> is an interactive process viewer and system monitor. It’s one of my favorite linux tools that I use regularly to monitor system resources. If you take <a href="http://man7.org/linux/man-pages/man1/top.1.html">top</a> and put it on steroids, you get htop.</p>
<p><img src="https://codeahoy.com/img/htop-small.png" alt="htop" /></p>
<!--more-->
<p>htop has an awesome visual interface that you can also interact with using your keyboard. The screen packs a lot of information which can be daunting to look at. I tried to find a nice infographic to explain what each number, value or color coded bars mean, but couldn’t find any. Hence I decided to make one myself over the Christmas break.</p>
<p>When you first launch htop, you’ll be greeted with a colorful interface showing a list of all processes running on the system. These are normally ordered by the amount of CPU usage, ordered from highest to lowest. It also shows the status of CPU usage, physical and swap memory.</p>
<p><img src="https://codeahoy.com/img/htop.jpg" alt="htop" /></p>
<p>There’s a lot information in the screenshot. To explain, I’ve separated the interface into <em>upper</em> and <em>lower sections</em> so I have enough room to annotate. (If you want to know, I used Photoshop to annotate the screenshot.)</p>
<p>Let’s start with the <strong>upper section</strong>. To see <strong>higher resolution</strong>, please click on the image.</p>
<p><a href="https://codeahoy.com/img/htop-top.png"><img src="https://codeahoy.com/img/htop-top.jpg" alt="htop" /></a></p>
<p>Here’s the <strong>lower section</strong> of htop.</p>
<p><a href="https://codeahoy.com/img/htop-bottom.png"><img src="https://codeahoy.com/img/htop-bottom.jpg" alt="htop" /></a></p>
<p>I hope you found this post useful. Here’s a <strong><a href="/compare/top-vs-htop.html">comparison between top and htop</a></strong>, comparing different features and properties.</p>
<p>If you are using macOS, please note that htop doesn’t come installed by default. You can install it easily using <code class="language-plaintext highlighter-rouge">brew</code>. Open the Terminal and type:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ brew install htop
</code></pre></div></div>
<p>After the installation is complete, you can launch it by typing <code class="language-plaintext highlighter-rouge">htop</code> on the Terminal:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo htop
</code></pre></div></div>
<p>Note: <code class="language-plaintext highlighter-rouge">sudo</code> is needed to give <code class="language-plaintext highlighter-rouge">htop</code> required access on macOS. On Linux, <code class="language-plaintext highlighter-rouge">sudo</code> isn’t required.</p>
<p>Until next time.</p>
Review of Andrew Ng's Machine Learning Course and Next Steps2017-01-19T00:00:00+00:00https://codeahoy.com/2017/01/19/online-course-review-the-best-machine-learning-course-for-beginners<p>Back when I was in college, I enrolled in a couple of introductory <a href="https://en.wikipedia.org/wiki/Artificial_intelligence">AI</a> courses. I quickly got bored: <a href="https://en.wikipedia.org/wiki/Artificial_neural_network">artificial neural networks</a> didn’t sound very practical and the dry mathematics was off-putting. After I finished the courses, I graduated and moved on. A while ago, I started noticing articles and blogs on self-driving cars that use “machine learning”. It sounded like a fancy new way to position the decades-old field of AI. I still wasn’t sure what the hype was all about. That was about to change.</p>
<!--more-->
<p>One evening by chance, I came across a link to what sounded like a <a href="https://news.ycombinator.com/item?id=9713802">Mario video</a>. Being a fan of retro video games, I opened it. The video shows a skilled player playing Super Mario World. Here’s the twist: <strong>the skilled player isn’t a human</strong>. It is a neural network program that taught itself how to play Super Mario World with zero help. When it started, it knew absolutely nothing about Super Mario World or Super Nintendo. It didn’t even know that pressing the A key on the controller makes Mario jump over obstacles. <strong>It learned to play and complete the first level all by itself… made possible by machine learning</strong>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/qv6UVOQ0F44" frameborder="0" allowfullscreen=""></iframe>
<p>Needless to say, my mind was blown.</p>
<p>I wish my professors had shown something similar at the beginning of the course. I would have been all over it. It was a fantastic demonstration of the power of machine learning. I spent the weekend reading blogs and news articles about machine learning applications. I tried to run some simple applications but didn’t get far. I wanted to understand what problems it can solve. I wanted to apply it to a real-world problem.</p>
<p>At some point, I decided that I need to take a course so I could read and understand machine learning blogs and research papers. I researched online found <a href="https://www.coursera.org/learn/machine-learning">a course on Coursera</a> offered by <a href="https://twitter.com/AndrewYNg">Andrew Ng</a>. For those of you who don’t know who Andrew is, he is a highly respected and very influential scientist in the fields of machine learning and AI. He’s leading machine learning efforts at Google and Baidu. Before that, he taught at Stanford as an associate professor.</p>
<h2 id="my-review-of-the-machine-learning-course-by-andrew-ng">My Review of the Machine Learning Course by Andrew Ng</h2>
<p>I enrolled in <strong><a href="https://www.coursera.org/learn/machine-learning">Andrew’s course on Machine Learning</a></strong> and I’m super glad I did. If I have to rate Andrew’s course out of 5 stars, <strong>I would give it 6 stars</strong>.</p>
<p>It was literally one of <em>the</em> best learning experiences of my life. I had fun throughout and learned many useful concepts, many of which I was able to apply to solve real-world problems.</p>
<ul>
<li>The course is introductory level and is designed for <strong>complete beginners</strong> to machine learning. You don’t need any prior experience with machine learning tools and libraries.</li>
<li>The course is <strong>100% free</strong>. You’ll need to pay about $50 <em>if</em> you want the <em>course certificate</em> after completion.</li>
<li>The course itself is 11 weeks. I spent 3-4 hours a week. If you have more time, you can definitely finish it sooner.</li>
<li>Andrew Ng has an amazing teaching style. It’s super fun and very engaging. He clearly articulates complex algorithms and mathematical equations which make it very easy to grasp the subject matter.</li>
<li>The course will <strong>introduce you to various flavors of machine learning algorithms</strong>: <a href="https://en.wikipedia.org/wiki/Linear_regression">linear regression</a>, <a href="https://en.wikipedia.org/wiki/Logistic_regression">logistic regression</a>, <a href="https://en.wikipedia.org/wiki/K-means_clustering">k-means</a>, (artificial) <a href="https://en.wikipedia.org/wiki/Artificial_neural_network">neural networks</a>, <a href="https://en.wikipedia.org/wiki/Support_vector_machine">support vector machines</a>, <a href="https://en.wikipedia.org/wiki/Unsupervised_learning">unsupervised learning</a>. By covering many different algorithms, it lays the groundwork and sets up the foundation so you can continue learning in the areas that interest you.</li>
<li><strong>Programming assignments focus on solving real-world problems</strong>: handwritten digit recognition using neural networks and spam classification with support vector machines (SVM) were my favorites.</li>
<li>A couple of friends who took the course <strong>complained</strong> about one aspect: <em>the assignments must be done in MATLAB or Octave</em>. They were hoping they’d be able to use their favorite language and learn a machine learning library like <a href="https://www.tensorflow.org/">TensorFlow</a>. However, I feel MATLAB/Octave is a great choice for this course. It forces you to think about applying machine learning algorithms using matrices and matrix operations, without getting caught in the nuances of a high-level language or a library. I would however, strongly recommend that you <strong>do not skip the tutorial sections covering MATLAB/Octave</strong> and pay very close attention to them. Review the tutorial sections twice if you need to, or you’ll spend a lot of time <em>stuck</em> on assignments.</li>
</ul>
<p>If you are interested in machine learning (you should) and you are a beginner or know very little about it, <a href="https://www.coursera.org/learn/machine-learning">Andrew’s course</a> is <strong>the best investment of your time that you can make</strong>. The only regret you’d have is that you didn’t enroll sooner :)</p>
<h2 id="what-to-do-next">What to do next?</h2>
<p>Deep Learning is one of the most sought after skills in the field of machine learning, and it is transforming many industries. Coursera and Andrew Ng offer some great courses on Deep Learning:</p>
<ol>
<li><a href="https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning" rel="nofollow">Neural Networks and Deep Learning</a></li>
<li><a href="https://www.coursera.org/learn/deep-neural-network?specialization=deep-learning" rel="nofollow">Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization</a></li>
<li><a href="https://www.coursera.org/learn/machine-learning-projects?specialization=deep-learning" rel="nofollow">Structuring Machine Learning Projects</a></li>
<li><a href="https://www.coursera.org/learn/convolutional-neural-networks?specialization=deep-learning" rel="nofollow">Convolutional Neural Networks</a></li>
</ol>
<p>In addition, you should also join <a href="https://www.kaggle.com/" rel="nofollow">Kaggle</a>. It is an online community of data scientists and machine learning enthusiasts. It runs competitions which are slightly advanced for beginners but a good way to explore the field. For many beginners, Kaggle’s best feature is the no-setup, customizable, Jupyter Notebooks environments, and access free GPUs plus a huge repository code published by other developers.</p>
<p>Happy machine learning.</p>
If Your Site Isn't Using HTTPS, You Are Doing It Wrong2017-01-18T00:00:00+00:00https://codeahoy.com/2017/01/18/if-your-site-isnt-using-https-you-are-doing-it-wrong<p>We live in a day and age where we simply cannot <em>take our right to privacy for granted</em>. When we communicate over <strong>unprotected channels, we expose our messages to everyone who happens to be along the way</strong>: The WiFi hotspots, corporate IT providers, ISPs, cloud providers, can listen in to our communication. We leave a trail of digital footprints behind. When aggregated, it can reveal information about ourselves. Eavesdroppers and intruders can make inferences about our behaviors and intentions: ISPs can determine what types of news stories we are interested in, employers can monitor our activities even on personal devices at work, look at our searches, see our messages, all when we communicate over unprotected channels.</p>
<p><img src="https://codeahoy.com/img/privacy.jpg" alt="Privacy" /></p>
<!--more-->
<p>Google has been <a href="https://developers.google.com/web/fundamentals/security/encrypt-in-transit/why-https">urging site owners to switch to HTTPS</a> for many years now. They started using HTTPS as a <a href="https://webmasters.googleblog.com/2014/08/https-as-ranking-signal.html">ranking indicator</a> for their search results. To tighten the screws, Chrome, starting with version 56 that is coming soon, will start <strong>showing “not secure” alerts on sites that collect login or credit card information over HTTP</strong>. Firefox will also <a href="https://blog.mozilla.org/security/2017/01/20/communicating-the-dangers-of-non-secure-http/">start displaying a red icon in the address bar as well as an in-context warning</a> for pages that ask users to login over HTTP.</p>
<p><img src="https://codeahoy.com/img/firefoxwarning.png" alt="Firefox" /></p>
<p>While I don’t know the real reason that compel Google to drive the HTTPS campaign, it’s a great direction for the future of the web, a direction that we should all support.</p>
<p>So how do you make your website secure? It’s way easier to secure sites with HTTPS these days than it used to be. In the past, obtaining a digital certificate that is required for HTTPS required paperwork and hundreds of dollars. This is no longer the case. <strong><a href="https://letsencrypt.org/">Let’s Encrypt</a> is a certificate authority that provides FREE certificates to anyone</strong>. It’s backed by organizations such as Mozilla, Facebook and Google to name a few. Let’s Encrypt makes it possible for anyone to have an HTTPS website for free. As an alternative, if you host your servers on the AWS, <a href="https://aws.amazon.com/blogs/aws/new-aws-certificate-manager-deploy-ssltls-based-apps-on-aws/">Certificate Manager</a> provides free certificates and handle certificate renewals as an added bonus.</p>
<p>Getting an HTTPS-enabled website is easier (and cheaper) now than ever. If you are concerned that HTTPS slows things down, <a href="https://istlsfastyet.com/">think again</a>. <strong>HTTPS can even be <a href="https://www.troyhunt.com/i-wanna-go-fast-https-massive-speed-advantage/">faster</a> than HTTP</strong>.</p>
<p><img src="https://codeahoy.com/img/httpvshttps.jpg" alt="HTTP vs HTTPS" /></p>
<p>I didn’t intentionally touch upon security threats to unencrypted traffic since they are well known to most people. The privacy aspects, unfortunately, aren’t as well known.</p>
<p>I would also like to make a quick announcement: <strong>starting today, all traffic to my blog is 100% fully encrypted and secured using HTTPS</strong>. I enabled HTTPS without spending a single penny and the whole process took less than an hour. If you have a website that isn’t using HTTPS yet, what are you waiting for?</p>
There's No Backdoor in WhatsApp. Just a Weakness That Could Be Exploited2017-01-17T00:00:00+00:00https://codeahoy.com/2017/01/17/theres-no-backdoor-in-whatsapp-just-a-weakness-that-could-be-exploited<p>Last week, Guardian <a href="https://www.theguardian.com/technology/2017/jan/13/whatsapp-backdoor-allows-snooping-on-encrypted-messages">ran a story</a> claiming that a backdoor built into WhatsApp can allows its parent company, Facebook, to read user messages despite advertising end-to-end encryption and complete privacy:</p>
<blockquote>
<p>Facebook claims that no one can intercept WhatsApp messages, not even the company and its staff, ensuring privacy for its billion-plus users. But new research shows that the <strong>company could in fact read messages due to the way WhatsApp has implemented its end-to-end encryption protocol</strong>.</p>
</blockquote>
<!--more-->
<p>Open Whisper Systems, the nonprofit behind <a href="https://en.wikipedia.org/wiki/Signal_Protocol">Signal protocol</a> that powers WhatsApp’s end-to-end message encryption, came to WhatsApp’s defense and <a href="https://whispersystems.org/blog/there-is-no-whatsapp-backdoor/">fired back</a> at the Guardian story:</p>
<blockquote>
<p>Today, the Guardian published a story falsely claiming that WhatsApp’s end to end encryption contains a “backdoor.” … The way this story has been reported has been disappointing. There are many quotes in the article, but it seems that the Guardian put very little effort into verifying the original technical claims they’ve made.</p>
</blockquote>
<p>So what really happened? <a href="https://twitter.com/tobiasboelter">Tobias Boelter</a>, who “<a href="https://www.theguardian.com/technology/2017/jan/16/whatsapp-vulnerability-facebook">discovered the vulnerability</a>”, followed up to further explain and support the backdoor/vulnerability theory and why it matters:</p>
<blockquote>
<p>… [WhatsApp] encrypted messaging works using secret and public keys. <strong>Every user has both a secret key known only to them, and a public key</strong>.</p>
<p>A user’s public key can be used to encrypt messages which can then only be made readable again with the associated secret key.</p>
</blockquote>
<p>Okay, so public key encrypts and private key decrypts. Public key is publicly available to anyone, while private key stays private. Let’s continue.</p>
<blockquote>
<p>A difficult problem in secure communication is getting your friend’s public keys. Apps such as WhatsApp and Signal make the process of getting those [public] keys easy for you by storing them on their central servers and allowing your app to download the public keys of your contacts automatically.</p>
<p>The problem here is that the <strong>WhatsApp server could potentially lie about the public keys</strong>. Instead of giving you your friend’s key, it could give you a public key belonging to a third party, such as the government.</p>
</blockquote>
<p>So a third party, impersonating your friend, can give you its own public key and WhatsApp will overwrite your friends actual key with it thinking that it has changed. You will encrypt messages using the wrong key which allows the third party to read your messages. The third party can act as the <strong><a href="https://owasp.org/www-community/attacks/Manipulator-in-the-middle_attack">man in the middle</a></strong> between you and your friend if it can compromise both, eavesdropping on the conversation while staying under your radar. But, in order to do any of this, <strong>it will require Facebook’s support</strong> and access to WhatsApp server infrastructure.</p>
<p>In reality, a <strong>user’s keys can change for any number of reasons</strong>. Wipe your device and reinstall the app or get a new device and you’ll get a new key. Open Whisper blog posts suggest that the way WhatsApp handles key change is appropriate:</p>
<blockquote>
<p>The only question it might be reasonable to ask is whether these safety number [keys] change notifications should be “blocking” or “non-blocking.” In other words, when a contact’s key changes, <strong>should WhatsApp require the user to manually verify the new key before continuing, or should WhatsApp display an advisory notification and continue without blocking the user</strong>.</p>
<p>[…] we feel that their choice to display a non-blocking notification is appropriate. It provides transparent and cryptographically guaranteed confidence in the privacy of a user’s communication, along with a simple user experience.</p>
</blockquote>
<p>What the Open Whisper blog post doesn’t mention is that the “<em>advisory notifications</em>” are <strong>optional</strong> and not <a href="https://www.whatsapp.com/faq/en/general/28030014">shown by default</a>. Users need to turn the “Show security notifications” option on explicitly in order to receive notifications that their friend’s key has changed.</p>
<p><img src="https://codeahoy.com/img/ws-showsecurity.png" alt="Show security notifications" /></p>
<p>WhatsApp has done the right thing considering its market and users. Most of WhatsApp’s users can’t be bothered to verify security codes, much less understand what they mean. End to end encryption was introduced <em>after</em> WhatsApp was already very successful and had very large number of users. I doubt “blocking” or even “non-blocking” key change notifications that could potentially confuse users was even an option. They had to find the right balance between security and usability. WhatsApp is secure enough. But <strong>it does have a weakness and it’s plausible that if the Big Brother wants to tap into your WhatsApp messages or phone calls, it totally can… probably without you ever knowing</strong>.</p>
Leadership vs Management - Leaders Have a Dream, A Vision...2017-01-16T00:00:00+00:00https://codeahoy.com/2017/01/16/leadership-vs-management-leaders-have-a-dream<p>Today is the Martin Luther King Jr. Day - a holiday to celebrate the life and legacy of a great <a href="https://en.wikipedia.org/wiki/African-American_Civil_Rights_Movement">Civil Rights Movement</a> leader. <a href="https://en.wikipedia.org/wiki/Martin_Luther_King_Jr.">Dr. King</a> was an incredibly effective leader who <em>challenged the status quo</em> and transformed the American society forever. He had a <a href="http://www.americanrhetoric.com/speeches/mlkihaveadream.htm">dream</a> and got people to rally behind it to make it a reality. And that’s what great leaders do. <em>Dr. King, Gandhi, Henry Ford, Jeff Bezos,</em> all had <strong>vision</strong> of a world that was very different from the one they lived in and they were able to <strong>inspire people</strong> to work towards making their vision a reality.</p>
<p><img src="https://codeahoy.com/img/leadership/showing-the-way.jpg" alt="boss vs leader picture" /></p>
<!--more-->
<p>The question is this: in an organization, is there a difference between a manager and a leader or are these synonymous? Some companies regularly refer to their managers as leaders but are these terms <em>always</em> interchangeable.</p>
<p>Let us look at differences between a leader and a manager.</p>
<p>Let’s start with managers. Managers are in charge of people and their responsibility is to <strong>get people to achieve some goal or target effectively</strong>. They don’t define these goals or targets but rather derive them directly or indirectly based on the strategy or direction the leadership has defined for the company. A bad manager can be very dangerous and can permanently damage team culture and morale. Managers have certain powers and influence that come attached to their position and their subordinates have to dance to their tunes (to varying degrees, but some degree). They are the ones who get to decide who gets the bonus or the promotion.</p>
<blockquote>
<p>In fact, <strong>management is a set of well-known processes, like planning, budgeting, structuring jobs, staffing jobs, measuring performance and problem-solving, which help an organization to predictably do what it knows how to do well</strong>. Management helps you to produce products and services as you have promised, of consistent quality, on budget, day after day, week after week. In organizations of any size and complexity, this is an enormously difficult task. <strong>We constantly underestimate how complex this task really is, especially if we are not in senior management jobs</strong>. So, management is crucial — but it’s not leadership.</p>
<p><strong>Leadership is entirely different. It is associated with taking an organization into the future, finding opportunities that are coming at it faster and faster and successfully exploiting those opportunities</strong>. Leadership is about vision, about people buying in, about empowerment and, most of all, about producing useful change.</p>
</blockquote>
<p>Managing people is tough. Imagine if Dr. King had to plan the logistics of his rallies, figure out the carpools or deal with time-off requests. To be effective, organizations need both leaders who are setting direction and looking into the future, and managers who are motivating the crew to keep the ship moving in that direction.</p>
<p>So while there is a difference between leadership and management, both can be done effectively or poorly. <a href="https://www.amazon.com/Levels-Leadership-Proven-Maximize-Potential/dp/1619692155">John C. Maxwell’s</a> hierarchy or the “levels of leadership”, I believe speak to both leaders and managers since both are in charge of people or teams:</p>
<p><img src="https://codeahoy.com/img/leadership/Maxwell1.jpg" alt="Leadership Levels" /></p>
<p>Good leaders and managers have a few things in common. They deeply believe that people, and not some process, are their most important asset. They share a clear and compelling vision and have <strong>followers who enroll voluntarily</strong>, because they want to be a part of it. <strong>They are extremely good at identifying the right people</strong>, who don’t need to be babysit or micromanaged to get something done. You can’t go wrong if you hire the right people who have the talents you need, define clear outcomes, <em>trust</em> them with getting the task done, <em>empower</em> them and <a href="http://avc.com/2012/02/the-management-team-guest-post-from-joel-spolsky/">“get the hell of out of their way”</a>:</p>
<blockquote>
<p>… there are a thousand leaders who learned to hire smart people and let them build great things in a nurturing environment of empowerment and it was AWESOME. That doesn’t mean lowering your standards. <strong>It doesn’t mean letting people do bad work. It means hiring smart people who get things done—and then getting the hell out of the way</strong>.</p>
</blockquote>
<p>That’s it. Hope you enjoyed this post. If you have any comments, please leave them in the comments section below.</p>
<p>Happy Martin Luther King Day.</p>
Tutorial - Configuring Photoshop for 2D Pixel Art2016-12-11T00:00:00+00:00https://codeahoy.com/2016/12/11/photoshop-pixel-art<p>I’m a huge fan of retro video games and <a href="http://imgur.com/a/d30KO">pixel art</a>. Over the Christmas break, I tried (after a long hiatus) to create some pixel art for a retro-style 2D mobile game I was building in Unity for fun. I had to struggle a little in setting up Photoshop to create 2D sprites and the background, so here’s a quick step-by-step tutorial on how to configure Photoshop to create pixel art.</p>
<!--more-->
<h2 id="step-1-create-a-tiny-image">Step 1: Create a Tiny Image</h2>
<p>Pixel art is done in very low resolutions. What this means is that you’ll start by creating a <strong>very small image, one that you can barely see without zooming in</strong>. I can’t give you a rule of thumb, but I generally use 20x20 pixels for sprites (sometimes 40x40 pixels if I want to put in more details) and about 150x80 pixels for backgrounds.</p>
<p>So go ahead and create a new image in Photoshop.</p>
<p><img src="https://codeahoy.com/img/ps-pixel/1.png" alt="photoshop-pixel-art" /></p>
<p>After you have created the image, you’ll be able to see it barely. So <strong>zoom in</strong> so you are able to see it.</p>
<p><img src="https://codeahoy.com/img/ps-pixel/1-zoom.png" alt="photoshop-pixel-art" /></p>
<h2 id="step-2-setup-image-interpolation-to-nearest-neighbors">Step 2: Setup Image Interpolation to Nearest Neighbors</h2>
<p>When your pixel art is resized or scaled, you’ll want the <strong>edges or corners to look hard and jagged instead of smooth and blurred</strong>. Be default, Photoshop uses <em>Bicubic interpolation</em> (or Bilinear) that produces a blurred effect when images are enlarged. While Bicubic interpolation works great for normal images, pixel art scaled using Bicubic look terrible and blurry as hell. As as example (<a href="http://blog.demofox.org/2015/08/15/resizing-images-with-bicubic-interpolation/">source</a>):</p>
<blockquote>
<p>Here’s the old man from The Legend of Zelda who gives you the sword. (<em>You may want to squint to see it</em>)</p>
</blockquote>
<p><img src="https://codeahoy.com/img/ps-pixel/LozMan.bmp" alt="old-man-from-zelda-original" /></p>
<p>Here he is scaled up <strong>4x with Bicubic interpolation</strong>:</p>
<p><img src="https://codeahoy.com/img/ps-pixel/lozman_4_2.bmp" alt="old-man-from-zelda-nn" /></p>
<p>Scale here 4x using <strong>Nearest Neighbor</strong>:</p>
<p><img src="https://codeahoy.com/img/ps-pixel/lozman_4_0.bmp" alt="old-man-from-zelda-nn" /></p>
<p>See the difference? Here’s how to configure Photoshop to use the ‘Nearest Neighbor’ image interpolation algorithm.</p>
<p><img src="https://codeahoy.com/img/ps-pixel/6.png" alt="photoshop-pixel-art" /></p>
<p>Note: If you are exporting the image (‘Save for Web’ option) and resizing it, make sure that ‘Nearest neighbor’ is selected under ‘Quality’ or ‘Resample’.</p>
<h2 id="step-3-set-up-the-tools">Step 3: Set up the Tools</h2>
<p>You’ll need to setup your drawing tools and get the desired pixelated effects. For pencil and eraser tools, here are the settings I used:</p>
<ul>
<li>size to 1 pixel (px).</li>
<li>hardness to 100%.</li>
<li>opacity to 100%.</li>
<li>for the eraser tool, mode was set to ‘Pencil’.</li>
</ul>
<p><img src="https://codeahoy.com/img/ps-pixel/2.png" alt="photoshop-pixel-art" /></p>
<p>The only other tool I used was the paint bucket which didn’t require any customization.</p>
<h2 id="step-4-show-the-grid-optional">Step 4: Show the Grid (optional)</h2>
<p>Grid is helpful in positioning and aligning things precisely. I find grid very useful when creating sprites. Grid can be enabled from the ‘View’ menu.</p>
<p><img src="https://codeahoy.com/img/ps-pixel/3.png" alt="photoshop-pixel-art" /></p>
<p>Next, we’ll need to <strong>adjust the grid so it can display each pixel individually</strong>. Open “Guides, Grids and Slices” settings from the Preferences menu and update the grid settings.</p>
<p><img src="https://codeahoy.com/img/ps-pixel/4.png" alt="photoshop-pixel-art" /></p>
<p><img src="https://codeahoy.com/img/ps-pixel/5.png" alt="photoshop-pixel-art" /></p>
<p>That’s all there is. I hope you found this tutorial helpful and that you go on to create magnificent pixel art :) To <a href="http://www.bobrossquotes.com/quotes.shtml">quote</a> <a href="https://en.wikipedia.org/wiki/Bob_Ross">Bob Ross</a>:</p>
<blockquote>
<p>“<strong>People might look at you a bit funny, but it’s okay. Artists are allowed to be a bit different.</strong>”</p>
</blockquote>
Should You Unit Test Private Methods?2016-11-19T00:00:00+00:00https://codeahoy.com/2016/11/19/should-you-unit-test-private-methods<p>To unit test private methods or not to test, that’s the question. There are two kinds of software developers in this world: those who <em>never</em> subject private methods to unit testing directly and those who do. Let’s look at both sides of the arguments to understand this better.</p>
<!--more-->
<p>This <a href="http://stackoverflow.com/questions/105007/should-i-test-private-methods-or-only-public-ones?noredirect=1&lq=1">Stackoverflow question</a> highlights the divide. The accepted answer says:</p>
<blockquote>
<p><strong>I do not unit test private methods. A private method is an implementation detail that should be hidden to the users of the class</strong>. Testing private methods breaks encapsulation.</p>
<p>If I find that the private method is huge or complex or important enough to require its own tests, I just put it in another class and make it public there</p>
</blockquote>
<p>User Dave Sherohman <a href="http://stackoverflow.com/a/105209">disagrees</a>:</p>
<blockquote>
<p>[…] Personally, my primary use for code tests is to ensure that future code changes don’t cause problems and to aid my debugging efforts if they do. I find that testing the private methods just as thoroughly as the public interface (if not more so!) furthers that purpose.</p>
<p>Consider: You have public method A which calls private method B. A and B both make use of method C. C is changed (perhaps by you, perhaps by a vendor), causing A to start failing its tests. Wouldn’t it be useful to have tests for B also, even though it’s private, so that you know whether the problem is in A’s use of C, B’s use of C, or both?</p>
</blockquote>
<p>Another user <a href="http://stackoverflow.com/questions/34571/how-to-test-a-class-that-has-private-methods-fields-or-inner-classes?noredirect=1&lq=1#comment76873_34586">writes</a>:</p>
<blockquote>
<p>[…] It’s totally valid to have an algorithm in a private method which needs more unit testing than is practical through a class’s public interfaces.</p>
</blockquote>
<p>I like my encapsulation and stay away from unit testing private methods directly. Most of the time, the functionality provided by private methods is covered by unit tests for public methods. If not, then it’s a sign that the private method should be given a class of its own. But sometimes, <strong>breaking away a new class isn’t easy or feasible, and you might end up introducing more complexity to the design</strong>. If that’s the case, it’s alright to go ahead and write unit tests for private methods.</p>
Performance Testing Serverside Applications2016-11-16T00:00:00+00:00https://codeahoy.com/2016/11/16/performance-testing-serverside-applications<p>Performance testing server-side applications is a crucial process to help understand <em>how the application behaves under load</em>. It helps software teams fine-tune their applications to get the best performance while keeping the infrastructure costs low. Performance testing answers several important questions such as:</p>
<ul>
<li>Is the application ready to handle the traffic that’s going to hit it?</li>
<li>What do average <em>response times</em> and <em>latencies</em> look like under normal and peak loads?</li>
<li>Can the application be <em>scaled out</em>?</li>
<li>What are the bottlenecks? (could be CPU, memory, an external service or a database server)</li>
<li>How many instances are needed for supporting the estimated traffic (i.e. max RPS)?</li>
<li>What type of instances are needed? Does the application requires an instance with higher CPU to Memory ratio? Or does it need an instance type that supports high network utilization?</li>
<li>Does the application slowly degrade in performance under load? Is it slowly leaking a resource that eventually crashes it after a few hours or days?</li>
</ul>
<!--more-->
<p>A lesson I learned recently is that <strong>performance testing should not be an after-thought</strong>. Software teams should start performance testing early in the release cycle and not wait until the end to do it. I once worked on a team that built a backend service that passed all unit, integration and end-to-end tests with flying colors. QA engineers didn’t find any bugs in the application’s logic. However, the <em>performance was just terrible when we ran load tests on it</em>. On a single <a href="https://aws.amazon.com/ec2/instance-types/">m4.large</a> instance, the application supported 80% fewer requests than the team had estimated! The <em>main bottleneck</em> was found to be the 2-core CPU that was utilized to its maximum capacity as the application issued several queries to the database and applied complex algorithms to build a graph. Investigations by developers revealed that to reduce the amount of work the CPU was doing, it would require significant design changes. But it was already too late - the deadline was just weeks away. We decided to proceed with the release - albeit by over-provisioning the hardware and over-running our cost estimates by a factor of 3.</p>
<p>Performance testing is a broad topic. Teams I work with run <a href="https://en.wikipedia.org/wiki/Load_testing">load</a> and <a href="https://en.wikipedia.org/wiki/Soak_testing">soak</a> tests to measure performance metrics such as throughput, latency, resource utilization, etc. using a wide variety of tools. At <a href="http://www.glu.com">Glu</a>, we build REST services in Java and use the following tools for our performance tests:</p>
<ul>
<li><a href="https://yourkit.com/">YourKit</a> Profiler to profile CPU and memory usage at a fine-grained level.</li>
<li><a href="http://jmeter.apache.org/">Apache Jmeter</a> to generate load (in reality, Blazemeter or distributed Jmeter)</li>
<li><a href="https://aws.amazon.com/cloudwatch/">Amazon Cloudwatch</a> to monitor resource utilization because we deploy our services on the Amazon cloud.</li>
<li><a href="https://www.hostedgraphite.com/">Hosted Graphite</a> to observe custom metrics that the application generates and we are interested in.</li>
<li><a href="https://www.elastic.co/products/kibana">Kibana</a> dashboards to look at the logs, errors, etc.</li>
</ul>
<p>Before I wrap this post up, there are few other important lessons I’d like to share:</p>
<ul>
<li><strong>Run performance tests on a production-like environment</strong>. I have seen teams run performance tests on their MacBooks Pros with 8-core CPU’s and get <a href="https://forums.aws.amazon.com/thread.jspa?threadID=16912">drastically different results</a> than from the actual cloud instances with puny, virtualized hardware.</li>
<li>Create a good load test plan which requires careful thought. <strong>The goal is to emulate the load that real users would generate</strong> otherwise you might spend a lot of time chasing ghosts and fixing issues that are unlikely to happen in production. For example, I was investigating an issue with the performance of a Chat service under load. After looking at the load test script, I found that it grouped tens of thousands of users together. In reality, a group has on average about 10-50 people. The fix was to update the load test script to use a random group-id or a group-id from a pool instead of reusing the same id for each request.</li>
</ul>
<p>I hope this post was helpful. Would love to hear your thoughts, the tools and the approach you take for performance testing. Till next time.</p>
Taking Responsibility for Your Actions2016-11-14T00:00:00+00:00https://codeahoy.com/2016/11/14/taking-responsibility-for-your-actions<p>I read <a href="https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">Pragmatic Programmer</a> whenever I get a chance. It’s such a great book. I was skipping through the pages today and landed on the first chapter. After (re)reading it, I can say that it is arguably <em>the</em> best chapter in the book. All software developers must read it once and live by its philosophy:</p>
<blockquote>
<p>One of the cornerstones of the pragmatic philosophy is the idea of <strong>taking responsibility for yourself and your actions in terms of your career advancement, your project, and your day-to-day work</strong>. A Pragmatic Programmer takes charge of his or her own career, and <strong>isn’t afraid to admit ignorance or error</strong>. It’s not the most pleasant aspect of programming, to be sure, but it will happen—even on the best of projects. Despite thorough testing, good documentation, and solid automation, things go wrong. Deliveries are late. <strong>Unforeseen technical problems come up</strong>.</p>
</blockquote>
<p>In your career, you <strong>will make mistakes</strong>. <a href="http://codeahoy.com/2016/04/14/mistakes-at-work-are-not-sins/">They are inevitable</a>. I can’t count the number of times I have made mistakes. So when you do make a mistake for something you <em>accepted</em> responsibility for:</p>
<ul>
<li>Don’t deflect responsibility.</li>
<li>Don’t blame another team member.</li>
<li>Don’t blame a vendor.</li>
<li>Don’t blame a library or a tool that you use.</li>
<li>Don’t blame management.</li>
<li>Don’t become defensive.</li>
</ul>
<!--more-->
<p>Sure any of the above factors could have played a role in the failure. But deflecting responsibility by making excuses or blaming someone or something is the worst possible way to handle such situations. What you should do instead is own up and offer solutions. That’s what your coworkers and management are interested in.</p>
<p>I’m reminded of this <a href="https://lkml.org/lkml/2012/12/23/75">email</a> from Linus in which he gets furious at a kernel programmer for making up “<em>lame excuses</em>”. (For the record, I strongly disagree with the language in the email.)</p>
<p>Responsibility is a two-way street. It has to be both given and accepted. I have seen managers foolishly thinking that they can walk up to an employee say out loud “<em>You are responsible for this</em>” or “<em>You own this</em>” and walk away. If the employee doesn’t accept or accepts half-heartedly, the responsibility gets diluted. The employee must be willing to accept the responsibility. Quoting from Pragmatic Programmer:</p>
<blockquote>
<p><strong>Responsibility is something you actively agree to</strong>. You make a commitment to ensure that something is done right, but you don’t necessarily have direct control over every aspect of it. In addition to doing your own personal best, you must analyze the situation for risks that are beyond your control. <strong>You have the right not to take on a responsibility for an impossible situation</strong>, or one in which the risks are too great. You’ll have to make the call based on your own ethics and judgment.</p>
</blockquote>
<p>This doesn’t mean that you should never accept responsibility for anything. That’s not how you grow in an organization and it will not help your career. Be fair, professional and ethical - analyze the risks, determine additional needs such as staff, time or budget and negotiate with your boss or manager. You have every right to think about the situation, but be fair in your assessment.</p>
Git Tips - Undoing Accidental Commits2016-11-13T00:00:00+00:00https://codeahoy.com/2016/11/13/git-tips-undoing-accidental-commits<p>Here are couple of git <em>undo’s</em> to get yourself out of trouble. Before you use anything from this post, please make a copy of the your working directory and store it somewhere safe. <em>Git can be very unforgiving and you may lose your changes without any warning</em>.</p>
<!--more-->
<h3 id="accidentally-committed-to-master-instead-of-a-new-branch">Accidentally committed to <code class="language-plaintext highlighter-rouge">master</code> instead of a new branch</h3>
<p>Use the commands below if you have accidentally committed your changes to the master branch instead of a new branch but haven’t pushed them to the remote repository yet.</p>
<pre class="prettyprint lang-sh">
# Create a new branch copying current state of master
git branch new-branch
# Restore master to second to last commit.
git reset --hard HEAD^
# Switch to the new branch to see your changes.
git checkout new-branch
</pre>
<h3 id="accidentally-committed-to-the-wrong-branch">Accidentally committed to the wrong branch</h3>
<p><code class="language-plaintext highlighter-rouge">git cherry-pick <commit-hash></code> is a handy command to choose a commit from one branch and apply it to another.</p>
<pre class="prettyprint lang-sh">
# Note the commit-hash you want to move
git log
# Restore branch to second to last commit
git reset --hard HEAD^
# Switch to the right branch
git checkout right-branch
# Apply the commit to the right branch
git cherry-pick commit-hash
</pre>
<p>Till next time.</p>
Automated Tests Help Developers Sleep Better2016-11-12T00:00:00+00:00https://codeahoy.com/2016/11/12/automated-tests-help-developers-sleep-better<p>In <a href="https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">Pragmatic Programmer</a>, Andy and Dave wrote:</p>
<blockquote>
<p>Most developers hate testing. They tend to test gently, subconsciously knowing where the
code will break and avoiding the weak spots. Pragmatic Programmers are different. <strong>We are
driven to find our bugs now, so we don’t have to endure the shame of others finding our
bugs later</strong>.</p>
</blockquote>
<p>There hasn’t been any shortage of literature on the benefits of automated testing in the last 10-15 years. Yet it comes as a surprise to me that most developers still don’t like to write unit tests. Some managers <a href="http://codeahoy.com/2016/04/16/do-not-misuse-code-coverage/">force developers to write tests by making an arbitrary code coverage number mandatory</a> for releases. This is dangerous because the result is often low quality and ineffective tests that do not catch the bugs that they were supposed to.</p>
<!--more-->
<p>But why do some developers dislike testing? One possible explanation is that they don’t buy the idea. Software developers are very smart people and they cannot be coerced into accepting an idea just because management thinks its great. In their defense, the management often fails to coach or convey the idea that <strong>automated tests benefit developers the most</strong>: they are written <em>by</em> the developers, <em>for</em> the developers themselves. They help developers sleep better at night.</p>
<p>Automated tests are very good at catching bugs before the code is released - <em>not all the bugs, but most of them</em>. <strong>Software developers, whether on-call or not, are on the hook for any bugs in their code. When there’s a bug that affects millions of users, it’s the developers who have to get out of their beds at 2am or stay late evenings to provide a fix</strong>. Not to mention the embarrassment of introducing a bug that irritated thousands/millions of users or resulted in lost revenue. It takes a long time to find the root cause just because at the time, a developer failed to take a break from implementing the functionality to think about all the ways it could break and write good tests to ensure that the bug was caught before production release.</p>
<p>Developers must be driven to test the <em>sh</em><em>*t</em> out of their code. We must put a comparable (sometimes more) effort into writing tests as we do on the actual feature itself. <a href="http://codeahoy.com/2016/07/05/unit-integration-and-end-to-end-tests-finding-the-right-balance/">Different testing techniques</a> should be used to catch different types of bugs.</p>
<blockquote>
<p>Finding bugs is somewhat like fishing with a net. We use fine, small nets (unit tests) to
catch the minnows, and big, coarse nets (integration tests) to catch the killer sharks.</p>
</blockquote>
<p>Pragmatic programmers don’t just stop with unit, integration or end-to-end tests. They also load test with 10x the peak traffic to catch the killer whales (even this <a href="http://www.shacknews.com/article/96998/pokemon-go-had-10x-more-users-at-launch-than-the-expected-worst-case-scenario">wasn’t enough for Niantic</a>). They test their system by replaying and replicating actual production traffic patterns. They perform <a href="https://en.wikipedia.org/wiki/Monkey_testing">monkey testing</a> seeing if their system crashes. <strong>They work very hard to break their own code, motivated by the desire to minimize the number of times they’ll have to be rocketed out of sleep or spend their weekend to find and fix bugs in their code</strong>.</p>
Building Microservices in Python and Flask (GitHub Project Included)2016-07-10T00:00:00+00:00https://codeahoy.com/2016/07/10/writing-microservices-in-python-using-flask<p>After years of building applications and platforms using the <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">Service Oriented Architecture</a>, I became very interested in <a href="http://martinfowler.com/articles/microservices.html">microservices</a> last year. So much so that I chose the job offer based on the sole fact that it was providing me an opportunity to design and develop microservices on the AWS platform. I’ll share the pros and cons of microservices in a later post.</p>
<p>In this post, we’ll see how to build microservices in Python using a <em>light-weight</em> framework called <a href="http://flask.pocoo.org/">Flask</a>. Unlike other web frameworks (e.g. Rails,) Flask is very flexible and doesn’t force you to adopt a specific layout style for your projects. It’s light-weight because it doesn’t require users to use particular tools or libraries. For example, Flask doesn’t come with any database access libraries. You will use extensions to add the functionality that you want.</p>
<!--more-->
<h2 id="cinema-3">Cinema 3</h2>
<p>I have created a fictional project called <a href="https://github.com/umermansoor/microservices">Cinema 3</a> that demonstrates the use of microservices using Python and Flask. In this hypothetical project, we have a few microservices working together to allow users to find movies and books tickets online. The microservices which make up the project are:</p>
<ul>
<li><a href="https://github.com/umermansoor/microservices/blob/master/services/movies.py">Movies</a>: Manages information related to movies e.g. title, rating, etc.</li>
<li><a href="https://github.com/umermansoor/microservices/blob/master/services/showtimes.py">Show Times</a>: Provides show times for movies.</li>
<li><a href="https://github.com/umermansoor/microservices/blob/master/services/bookings.py">Bookings</a>: Handles online booking.</li>
<li><a href="https://github.com/umermansoor/microservices/blob/master/services/user.py">Users</a>: Manages user accounts for our project.</li>
</ul>
<p>The microservices talk to each other using REST API. How do they know the address (host:port) of other services? In this example, each microservice runs on a separate port so we could identify them. In real-world projects, people use more advanced techniques such as service discovery (either server or client side; <a href="https://www.consul.io/">consul</a> is a popular tool that people use for this purpose.)</p>
<h3 id="flask-vs-other-python-web-application-frameworks">Flask VS Other Python Web Application Frameworks</h3>
<p>Before you decide to use Flask, see how it holds up to against other Python web application frameworks on popularity, features and more. Click on the image below to start comparing.</p>
<p><a href="/compare/python-frameworks.html">
<img src="/assets/images/compare/python-frameworks/compare-home-header.jpg" alt="Compare Top Python Frameworks" />
</a></p>
<h2 id="sample-code-on-github">Sample Code on GitHub</h2>
<p><strong>Here’s a <a href="https://github.com/umermansoor/microservices">link to the project on GitHub</a></strong>. The source code itself is pretty simple as this is just an example to give you a basic understanding of building microservices using Flask.</p>
<p>If you have any comments or question about the project, please let me know in the comments section below.</p>
<p>If you’re looking to learn Flask, here’s an excellent YouTube video series you should watch.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/MwZwr5Tvyxo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
Should the US Allow Foreign Developers?2016-07-09T00:00:00+00:00https://codeahoy.com/2016/07/09/should-the-us-allow-foreign-developers<p>Hiring good developers is really difficult. It’s even more difficult when the market is red hot. The Bay Area is ripe with opportunities for job-seekers and good developers are almost impossible to come by. Today it’s easier than ever for developers to choose the job they really like, at least in the US. While it’s great for the developers, <strong>companies and especially start-ups are hurting because there just aren’t enough talented developers to go around</strong>. Start-ups and technology companies want the US government to relax its immigration policies so they can bring more developers from other countries. The Anti-immigration groups are against letting “foreigners” in. They argue that the focus should be on training more Americans to learn programming. While they are certainly not wrong about training more people to learn programming, <em>Paul Graham</em> <a href="http://www.paulgraham.com/95.html">argues that the technology companies are right</a> and the training cannot match the resources elsewhere in the world:</p>
<!--more-->
<blockquote>
<p>The technology companies are right. What the anti-immigration people don’t understand is that <strong>there is a huge variation in ability between competent programmers and exceptional ones, and while you can train people to be competent, you can’t train them to be exceptional</strong>. Exceptional programmers have an aptitude for and interest in programming that is not merely the product of training.</p>
<p>The US has less than 5% of the world’s population. Which means if the qualities that make someone a great programmer are evenly distributed, 95% of great programmers are born outside the US.</p>
</blockquote>
<p>He’s right. If people don’t have an interest in programming, they give up or even worse, stick around just for the money. I have seen too many “developers” (both American-born and foreign) doing it just for the money. <strong>They religiously punch the clock 9 to 5 just for one thing: the paycheck</strong>. They have absolutely zero passion towards their profession and no respect for software engineering. To be fair to them, staring at lines of code and thinking in terms of algorithms, isn’t for everyone. Good developers who actually love software but are working in environments where their passion isn’t shared by their colleagues, either quit and mentally check-out.</p>
<p>And what about the claim that technology companies like foreigners because they can pay them lower wages? Or by having more options available, they can drive the prices down? I personally haven’t seen the wage difference. Foreigners are paid about the same as locals and bringing them over to US isn’t cheap either: there are legal fees, relocation bonuses and a huge risk that the person may not like the job and quit within a year.</p>
<blockquote>
<p>So they [Anti-immigration groups] claim it’s because they want to drive down salaries. But if you talk to startups, you find practically every one over a certain size has gone through legal contortions to get programmers into the US, where they then paid them the same as they’d have paid an American. <strong>Why would they go to extra trouble to get programmers for the same price? The only explanation is that they’re telling the truth: there are just not enough great programmers to go around</strong>.</p>
</blockquote>
<p>I know at least three start-ups that shipped their entire software development to India (that’s 30 software development jobs between them) just because they couldn’t find developers locally.</p>
<p>So should the US allow a few thousand great programmers to come in every year? I think the answer is that it <em>absolutely</em> should. Growing up, my dad used to tell me that one of the reasons why the US is the greatest country in the world is because it attracts bright people from all over the planet and provides them plenty of opportunities to succeed. China may have the potential to <a href="http://www.infoworld.com/article/2862396/techology-business/chinas-competition-with-silicon-valley-is-about-to-get-serious.html">challenge</a> the Silicon Valley. Paul Graham asks what would happen if great programmers gathered someplace else:</p>
<blockquote>
<p>What if most of the great programmers collected in one hub, and it wasn’t here [in the US]? That scenario may seem unlikely now, but it won’t be if things change as much in the next 50 years as they did in the last 50.</p>
<p><strong>We have the potential to ensure that the US remains a technology superpower just by letting in a few thousand great programmers a year</strong>. What a colossal mistake it would be to let that opportunity slip. <strong>It could easily be the defining mistake this generation of American politicians later become famous for</strong>. And unlike other potential mistakes on that scale, it costs nothing to fix.</p>
</blockquote>
<p>Do you agree or disagree? Please leave your comments in the section below.</p>
Interactive Emails with Email Markup2016-07-08T00:00:00+00:00https://codeahoy.com/2016/07/08/interactive-emails-with-email-markup<p>It was a pleasant surprise when itinerary for my upcoming flight automatically showed up on my <a href="https://en.wikipedia.org/wiki/Samsung_Galaxy_S6">Galaxy S6</a>. I didn’t have open my Gmail and search for the confirmation email. It was right there, just when I needed it to check the departure time.</p>
<p><img src="https://codeahoy.com/img/blogs/google-flight-trip.png" alt="google-flight-trip" />
<em>(disclaimer: not my itinerary)</em></p>
<!--more-->
<p>I immediately started wondering how it did that. At first, I thought Google did this by parsing and analyzing the text in the confirmation email and making a best-effort extraction of the flight information. But that wouldn’t be very reliable and luckily that’s not how it works. In reality, the confirmation email that was sent to me <strong>contained additional information, as metadata, which Google used to display the itinerary</strong>. Google calls it the <a href="https://developers.google.com/gmail/markup/overview">Email Markup</a>. In essence, senders create and put <a href="http://schema.org/">markup</a> in their emails to instruct Google to display an interactive <a href="https://developers.google.com/gmail/markup/actions/actions-overview">action</a> to the recipients, such as RSVP an event, confirm subscription, fetch boarding passes, etc.</p>
<blockquote>
<p>Email is an important part of how we get things done – from planning an event with friends to organizing a trip to Paris. So much information is contained inside emails – like the details of a dinner party or travel itinerary – <strong>and so many emails require action – like RSVP, or flight checkin</strong>.</p>
<p>By <strong>adding <a href="http://schema.org/">schema.org</a> markup to the emails you send your users, you can make that information available across their Google experience, and make it easy for users to take quick action</strong>. Inbox, Gmail, Google Calendar, Google Search, and Google Now all already use this structured data.</p>
</blockquote>
<p>Some actions show up as buttons next to the subject line in the inbox. While this feature has been around for a couple of years at least, I believe Google has only recently started using this information across other services like Google Now and Google Calendar.</p>
<p><img src="https://codeahoy.com/img/blogs/gmail-action-buttons.png" alt="gmail-action-buttons" /></p>
<p>I personally find this very useful and it has some interesting <a href="https://moz.com/blog/markup-for-emails">use cases</a>:</p>
<blockquote>
<ul>
<li>
<p><a href="https://developers.google.com/gmail/markup/reference/flight-reservation">Flight reservations</a> - Includes options for displaying basic flight confirmation information, boarding pass, check-in, update a flight, cancel a flight, and additional options. This Highlight is also supported in Google Now.</p>
</li>
<li>
<p><a href="https://developers.google.com/gmail/markup/reference/order">Orders</a> - Includes options for displaying basic order information, view order action, and order with billing details.</p>
</li>
<li>
<p><a href="https://developers.google.com/gmail/markup/reference/parcel-delivery">Parcel deliveries</a> - Includes options for displaying basic parcel delivery information and detailed shipping information.</p>
</li>
<li>
<p><a href="https://developers.google.com/gmail/markup/reference/hotel-reservation">Hotel reservations</a> - Includes options for displaying basic hotel reservation information, updating a reservation, and canceling a reservation. This Highlight is also supported in Google Now.</p>
</li>
<li>
<p><a href="https://developers.google.com/gmail/markup/reference/restaurant-reservation">Restaurant reservations</a> - Includes options for displaying basic restaurant reservation information, updating a reservation, and canceling a reservation. This Highlight is also supported in Google Now.</p>
</li>
<li>
<p><a href="https://developers.google.com/gmail/markup/reference/event-reservation">Event reservation</a> - Includes options for basic event reminders without a ticket, event with ticket & no reserved seating, sports or music event with ticket, event with ticket & reserved seating, multiple tickets, updating an event, and canceling an event. This Highlight is also supported in Google Now.</p>
</li>
</ul>
</blockquote>
<p>For more information:</p>
<ul>
<li>Kristi Hines has written a good <a href="https://moz.com/blog/markup-for-emails">article</a> if you are interested in using Email Markup for your product your services.</li>
<li><a href="https://developers.google.com/gmail/markup/getting-started">Guide</a> from Google.</li>
</ul>
Unit, Integration and End-To-End Tests - Finding the Right Balance2016-07-05T00:00:00+00:00https://codeahoy.com/2016/07/05/unit-integration-and-end-to-end-tests-finding-the-right-balance<p>This is something I have regrettably noticed in many backend projects that I have worked on. Developers write “unit tests” that in reality are ‘end-to-end’ tests. They test the entire flow of the application from start to the end. There is no isolation of units and <strong>the notion of the unit is the whole system</strong>, along with all of its external dependencies like databases, queues, caches, and other services. For a web server project, these tests start the server, initialize a HTTP client, make a HTTP request and check the response to make sure it has all the expected information. If so, the test is declared a success. By treating the whole system as a unit and not testing independent units in isolation and their interplay, we loose many benefits that unit and integration tests offer.</p>
<!--more-->
<p>Technically speaking, these developers aren’t violating the definition or principles of unit testing. Unit testing is <a href="http://martinfowler.com/bliki/UnitTest.html">ill-defined</a>. I don’t claim to be an expert, but in my humble opinion:</p>
<ul>
<li>Unit testing should focus on testing small units (typically a Class or a complex algorithm).</li>
<li>Units should be tested in isolation and independent of other units. This is typically achieved by <a href="https://en.wikipedia.org/wiki/Mock_object">mocking</a> the dependencies.</li>
<li>Unit tests should be fast. Usually shouldn’t take more than a few seconds to provide feedback.</li>
</ul>
<p>Most projects benefit from having a <strong>balanced mix of various automated tests to capture different types of errors</strong>. The exact composition of the mix varies depending on the nature of the project, as we’ll see later.</p>
<p>End-to-end tests are good at capturing certain kinds of bugs, but their <strong>biggest drawback is that they cannot pin-point the root cause of failure</strong>. Anything in the entire flow could have contributed to the error. In large and complex systems, it’s like finding a needle in the haystack: you’ll find the root cause, but it will take time. Because unit tests focus on small modules that are tested independently, they can <strong>identify the lines of code that caused the failure</strong> with laser-sharp accuracy, which can save a lot of time.</p>
<p>Another nice thing about unit tests is that <strong>they always work, and they work fast</strong>. Unlike end-to-end tests that rely on external components, <strong>unit tests are not flaky</strong>. If I can build a project on my machine, I should be able to run its unit tests. In contrast, end-to-end tests would fail if some external component, like a database or a messaging queue, is not available or cannot be reached. And they can take a lvery ong time to run.</p>
<p>Unit tests allow developers to <strong>refactor and add new features with confidence</strong>. When I’m refactoring a complex project that has well-written unit tests, I run them often, usually after every small change. In a matter of few seconds, I know whether I broke something or not. Even better, a failing test usually prints a nice message telling me what broke: whether some <a href="http://xunitpatterns.com/Guard%20Assertion.html">GuardAssertion</a> failed or the expected response was off by one, helps me isolate the failure.</p>
<p>Between unit and end-to-end tests lie integration tests. They have one major advantage over unit tests: <strong>they ensure that modules which work well in isolation, also play well together</strong>. Integration tests typically focus on a small number of modules and test their interactions.</p>
<p>The key is to find the <strong>right balance between unit, integration and end-to-end tests</strong>. According to <a href="http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html">Google’s Testing Blog</a>:</p>
<blockquote>
<p>To find the right balance between all three test types, the best visual aid to use is the testing pyramid. Here is a simplified version of the testing pyramid […]:</p>
<p><img src="https://codeahoy.com/img/blogs/test_pyramid.png" alt="test_pyramid" /></p>
<p>The bulk of your tests are unit tests at the bottom of the pyramid. As you move up the pyramid, your tests gets larger, but at the same time the number of tests (the width of your pyramid) gets smaller.</p>
<p>As a good first guess, <strong>Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests</strong>. The exact mix will be different for each team, but in general, <em>it should retain that pyramid shape</em>. <strong>Try to avoid these anti-patterns</strong>:</p>
<ul>
<li>
<p><strong>Inverted pyramid/ice cream cone</strong>. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests.</p>
</li>
<li>
<p><strong>Hourglass</strong>. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle.</p>
</li>
</ul>
</blockquote>
<p><em>70/20/10</em> split between unit, integration and end-to-end tests is a good, general rule of thumb. If a project has large number of integrations or complex interfaces, it should have more integration and end-to-end tests. A project that is primarily focused on computation or data, should have more unit tests and fewer integration tests. The right mix depends on the nature of the project but the <strong>key is to retain the pyramid shape</strong> of the testing pyramid, that is, <code class="language-plaintext highlighter-rouge">Unit > Integration > End-to-End Tests</code>.</p>
RESTful - What Are Idempotent and Safe Methods and How to Use Them?2016-07-04T00:00:00+00:00https://codeahoy.com/2016/07/04/rest-design---choosing-the-right-http-method<p>One of the challenges when designing a REST API is choosing the right HTTP method (<em>GET</em>, <em>PUT</em>, <em>POST</em> etc.) that corresponds with the operation being performed. Some people incorrectly assume that they can freely choose any method as long as the client and the server agree on it. This is wrong because a request passes through many intermediaries and middleware applications which perform optimizations based on the HTTP method type. These optimizations depend on two key characteristics of HTTP methods: <a href="http://codeahoy.com/2016/06/30/idempotent-and-safe-http-methods-why-do-they-matter/">idempotency and safety</a>, which are defined in the <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html">HTTP specification</a>.</p>
<!--more-->
<p><strong>Safe HTTP Methods</strong>: Safe methods aren’t expected to cause any side effects. These operations are read-only. E.g. querying a database.</p>
<p><strong>Idempotent HTTP Methods</strong>: Idempotent methods guarantee that repeating a request has the same effect as making the request once.</p>
<p>Idempotency and safety are properties of HTTP methods that server applications must correctly implement. This means if you are implementing an operation and choose an idempotent HTTP method to invoke the operation, you must ensure that the implementation returns the same result if invoked once or multiple times for the same input.</p>
<h3 id="get-idempotent--safe">GET: Idempotent & Safe</h3>
<p>GET requests are used for retrieving information. These requests must be idempotent and <strong>safe</strong>: any operation invoked using GET must not alter the state of any resource.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">GET</span> <span class="o">/</span><span class="nx">books</span>
</code></pre></div></div>
<p>To get a specific book,</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">GET</span> <span class="o">/</span><span class="nx">books</span><span class="o">/<</span><span class="nx">title</span><span class="o">></span>
</code></pre></div></div>
<h3 id="post-non-idempotent">POST: Non-Idempotent</h3>
<p>POST requests are not idempotent. They are used for creating new resources or updating existing ones. For example, suppose we have a resource called <code class="language-plaintext highlighter-rouge">Student</code> with the following attributes: <code class="language-plaintext highlighter-rouge">name</code>, <code class="language-plaintext highlighter-rouge">college</code>, <code class="language-plaintext highlighter-rouge">major</code> and <code class="language-plaintext highlighter-rouge">gpa</code>. To enroll a new student, we can use POST to create a new resource:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">POST</span> <span class="o">/</span><span class="nx">students</span><span class="o">/</span> <span class="c1">// Create a new student</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">name</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Michael Scarn</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">college</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Stanford</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">major</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">computer science</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>
<p><strong>POST requests are allowed to perform partial updates</strong>. For example, to update the GPA of a student, we’ll make the POST request on the specific record (given by the <code class="language-plaintext highlighter-rouge">student_id</code>) and only supply the attribute we want to update:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">POST</span> <span class="o">/</span><span class="nx">students</span><span class="o">/<</span><span class="nx">student_id</span><span class="o">></span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">gpa</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">3.9</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In the above example, if the <code class="language-plaintext highlighter-rouge">student_id</code> doesn’t exist, the application should return <code class="language-plaintext highlighter-rouge">HTTP 404: Not Found</code> error.</p>
<h3 id="put-idempotent">PUT: Idempotent</h3>
<p>PUT requests are idempotent. This means that identical requests can be repeated multiple times, and the state of the resource on the server doesn’t change any further after the first request. PUT can be used to create a new resource or update an existing one. For updates, <strong>PUT requests must contain all the attributes of the resource, unlike POST requests which can have partial attributes</strong>. Here’s a PUT request to update a student’s GPA:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">PUT</span> <span class="o">/</span><span class="nx">students</span><span class="o">/<</span><span class="nx">student_id</span><span class="o">></span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">name</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Michael Scarn</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">college</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Stanford</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">major</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">computer science</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">gpa</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">3.9</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In the above example, if the <code class="language-plaintext highlighter-rouge">student_id</code> in the PUT request doesn’t exist, the application may create a new record and assign it the <code class="language-plaintext highlighter-rouge">student_id</code>.</p>
<p>If you are wondering why PUT requests must have all the attributes and not just the one we want to update, consider the following hypothetical situation in which we allow partial updates in the PUT request. Suppose that a student is switching majors. The client issues two PUT requests: the first one to update student’s GPA for the existing major and a second request to update the major and reset the GPA. The PUT requests are made in the right order:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">PUT</span> <span class="o">/</span><span class="nx">students</span><span class="o">/<</span><span class="nx">student_id</span><span class="o">></span> <span class="c1">//Partial update: Violates idempotency contract</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">gpa</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">3.85</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">PUT</span> <span class="o">/</span><span class="nx">students</span><span class="o">/<</span><span class="nx">student_id</span><span class="o">></span> <span class="c1">//Partial update: Violates idempotency contract</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">major</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">engineering</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">gpa</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">0.0</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Assume that the request for the second requests arrives before the first one which is lost due to a network error. Because the client thinks that PUT requests are idempotent, it may retry the first request a second time, that will incorrectly update the student’s GPA to 3.85 for the new major, leaving the resource in an <em>inconsistent state</em>. It could be very hard to trace and breaks the idempotency contract: multiple invocation result in different states.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">PUT</span> <span class="o">/</span><span class="nx">students</span><span class="o">/<</span><span class="nx">student_id</span><span class="o">></span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">name</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Michael Scarn</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">college</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Stanford</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">major</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">engineering</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">gpa</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">3.85</span><span class="dl">"</span> <span class="c1">// GPA updated for the wrong major!</span>
<span class="p">}</span>
</code></pre></div></div>
<p><strong>Partial updates are important especially if the resource contains a lot of attributes. It is wasteful to first fetch the resource using GET and then send all the attributes when the goal is to update just a few</strong>. In reality, a lot of developers allow partial updates with PUT, which is a violation of the idempotency contract. For partial updates, use POST, or as we’ll later see, PATCH.</p>
<h4 id="post-vs-put---partial-vs-complete-updates-to-resource">POST vs PUT - Partial vs Complete Updates to Resource</h4>
<p>There is a common <a href="http://www.tbray.org/ongoing/When/200x/2009/03/20/Rest-Casuistry">misconception</a> that it isn’t RESTful to use the HTTP POST method to update existing resources. People who hold this view suggest that one should always use the PUT method for updates and POST for creating new resources. This isn’t quite right. The REST standard doesn’t stop us from using POST requests for updates. In fact, it doesn’t even talk about it because <strong>idempotency and safety guarantees are properties of the HTTP protocol, not of the REST standard</strong>. <a href="http://roy.gbiv.com/">Roy Fielding</a>, <a href="http://roy.gbiv.com/untangled/2009/it-is-okay-to-use-post">writes</a>:</p>
<blockquote>
<p>Some people think that REST suggests not to use POST for updates. Search my dissertation and you won’t find any mention of CRUD or POST. The only mention of PUT is in regard to HTTP’s lack of write-back caching. The main reason for my lack of specificity is because the methods defined by HTTP are part of the Web’s architecture definition, not the REST architectural style. […] For example, it isn’t RESTful to use GET to perform unsafe operations because that would violate the definition of the GET method in HTTP, which would in turn mislead intermediaries and spiders.</p>
</blockquote>
<p>Therefore, <strong>the choice between PUT and POST boils down to one thing: idempotency guarantee of these methods</strong>. Because PUT is idempotent, clients or intermediaries can repeat a PUT request if the the response for the first request doesn’t arrive on time, even though the request may have been processed by the server. In order to stay idempotent, PUT requests must replace the entire resource and hence must send all the attributes. For partial updates, POST or PATCH (non-idempotent methods) must be used.</p>
<h3 id="delete-idempotent">DELETE: Idempotent</h3>
<p>DELETE requests are idempotent and are used for deleting a resource. One common confusion people have with DELETE calls is what type of HTTP status code to return on repeat calls. Some people assume that because DELETE is idempotent, it must always return the same HTTP status e.g. <code class="language-plaintext highlighter-rouge">HTTP 200</code>. This assumption is wrong. Although there is no harm in returning the same status code, if your use case requires it, <strong>returning a different status code like the <code class="language-plaintext highlighter-rouge">HTTP 404: Not Found</code> doesn’t violate the idempotency contract</strong>. Idempotency doesn’t concern itself with what is returned to the client. It refers to the state of some resource on the server. So it is perfectly valid to return <code class="language-plaintext highlighter-rouge">HTTP 200: OK</code> on the first delete call, and <code class="language-plaintext highlighter-rouge">HTTP 404: Not Found</code> on subsequent ones, since in both cases, the resource is deleted and its state isn’t changed on the server side. You might also use <code class="language-plaintext highlighter-rouge">HTTP 204: No Content</code> if the response body is empty.</p>
<h3 id="patch-non-idempotent">PATCH: Non-Idempotent</h3>
<p>I like to think of PATCH as the non-idempotent cousin of the PUT request. Because it is non-idempotent, it could be used partial updates:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">PATCH</span> <span class="o">/</span><span class="nx">students</span><span class="o">/<</span><span class="nx">student_id</span><span class="o">></span> <span class="c1">//Partial update: OK</span>
<span class="p">{</span>
<span class="dl">"</span><span class="s2">gpa</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">3.85</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It is up to you whether you want to adopt PATCH for partial updates or use POST.</p>
<p>Here’s a table summarizing the results:</p>
<table>
<thead>
<tr>
<th>HTTP Method</th>
<th>Idempotent?</th>
<th>Operation?</th>
</tr>
</thead>
<tbody>
<tr>
<td>GET</td>
<td>yes</td>
<td>Retrieval or query.</td>
</tr>
<tr>
<td>POST</td>
<td>NO</td>
<td>Create or update resources. Partial updates are allowed.</td>
</tr>
<tr>
<td>PUT</td>
<td>yes</td>
<td>Create or update resources. Partial updates are not allowed.</td>
</tr>
<tr>
<td>DELETE</td>
<td>yes</td>
<td>Delete resources.</td>
</tr>
<tr>
<td>PATCH</td>
<td>NO</td>
<td>For partial updates.</td>
</tr>
</tbody>
</table>
Idempotent and safe HTTP methods - REST API2016-06-30T00:00:00+00:00https://codeahoy.com/2016/06/30/idempotent-and-safe-http-methods-why-do-they-matter<p>If you are designing or building <strong>REST API</strong>s, you should be aware of two very important properties of HTTP methods: <strong>idempotency and safety</strong>. These properties are defined in the <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html">HTTP specification</a>. I’m calling them properties, but ‘guarantees’ would be a better term: you don’t automatically get them; you actually need to design for these guarantees because your clients expect you to follow the contract. Let’s get the definitions out of the way first and then we’ll look at the contract and why it’s important to stick to it.</p>
<!--more-->
<h2 id="idempotent-http-methods">Idempotent HTTP Methods</h2>
<p>An operation is <strong>idempotent</strong> if it will produce the same results when executed once or multiple times. For example, it doesn’t matter how many times I submit a request to set my current location to ‘San Francisco’. The final outcome will be the same: the city field in the database is set to ‘San Francisco’. On the other hand, a request to POST a new message to the forum is <em>not idempotent</em>: the same message will be stored or sent multiple times if the client. Some people, wrongly, assume that for a request to be idempotent, the same response must be sent back to the client each time: idempotency has nothing to do with the response that’s sent back to the client. It’s a server side guarantee ensuring that the <strong>state of the resource on the server</strong> does not change any further after the first request, no matter how many times the request is duplicated.</p>
<p>Some idempotent operations have an additional, <em>special property</em>: they do not modify the state on the server side at all. Simply put, these methods are read-only and have absolutely zero side-effects. For example, a query to retrieve my current city doesn’t change the database. These types of operations are given a <em>special</em> name: <strong>safe</strong> or <strong><a href="http://www.less-broken.com/blog/2011/07/why-you-should-care-about-idempotence.html">nullipotent methods</a></strong>:</p>
<blockquote>
<p>Related is the idea of nullipotence: a function is nullipotent if not calling it at all has the same side effects as calling it once or more. In practice, this simply means that the function doesn’t have any side effects at all. A database query saying “get row 42” is a good example. Nullipotence is clearly a stronger condition than idempotence.</p>
</blockquote>
<p>Here’s a <strong>list</strong> of the most commonly used HTTP methods and whether they are idempotent and/or safe as defined by the contract:</p>
<table>
<thead>
<tr>
<th>HTTP Method</th>
<th>Idempotent?</th>
<th>Safe?</th>
</tr>
</thead>
<tbody>
<tr>
<td>GET</td>
<td>yes</td>
<td>yes</td>
</tr>
<tr>
<td>HEAD</td>
<td>yes</td>
<td>yes</td>
</tr>
<tr>
<td>PUT</td>
<td>yes</td>
<td>NO</td>
</tr>
<tr>
<td>DELETE</td>
<td>yes</td>
<td>NO</td>
</tr>
<tr>
<td>POST</td>
<td>NO</td>
<td>NO</td>
</tr>
</tbody>
</table>
<h2 id="is-patch-idempotent">Is PATCH idempotent?</h2>
<p>Patch is not idempotent and not safe. PATCH is used for partial updates to resources. It is a common source of confusion because it is <em>possible</em> to use PATCH in such a way as to be idempotent such as using etags of If-Modified-Since header.</p>
<h2 id="why-idempotency-matters">Why idempotency matters?</h2>
<p>Idempotency and safety (nullipotentcy) are <em>guarantees</em> that server applications make to their clients and the world. It is a <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html">contract defined by the HTTP</a> standard that developers must adhere to when implementing <strong>REST API</strong>s over HTTP. An operation doesn’t automatically become idempotent or safe just because it is invoked using the <code class="language-plaintext highlighter-rouge">GET</code> method, if it isn’t implemented in an idempotent manner. A poorly written server application might use <code class="language-plaintext highlighter-rouge">GET</code> methods to update a record in the database or to send a message to a friend (I have seen applications that do this.) This is a really, really bad design.</p>
<p>Adhering to the idempotency and safety <strong>contract</strong> helps make an API fault-tolerant and robust. Clients, middleware applications and various servers that requests pass through before reaching your application, use this contract for various optimizations. Clients may automatically cancel a <code class="language-plaintext highlighter-rouge">GET</code> request if it is taking too long to process and repeat it because they assume it has the same effect (since <code class="language-plaintext highlighter-rouge">GET</code> is idempotent). However, they won’t do the same thing for <code class="language-plaintext highlighter-rouge">POST</code> requests because the first one may have already altered some state on the server side. This is the reason why web browsers display a warning message that you are about to re-submit a form when you hit the back button to go to a form (For this reason, always redirect after a successful <code class="language-plaintext highlighter-rouge">POST</code> operation.)
In the same veins, cache servers <a href="http://stackoverflow.com/questions/626057/is-it-possible-to-cache-post-methods-in-http">don’t cache</a> POST requests and safe methods can pre-fetched to stored in cache to enhance performance.</p>
<p>In summary, when building RESTful applications using HTTP, it is important to implement HTTP methods in a manner that satisfies their idempotency and safety contract, because clients and intermediaries are free to use this contract to optimize and enhance the user experience. Don’t use <code class="language-plaintext highlighter-rouge">GET</code> method for operations that alter the database and don’t use <code class="language-plaintext highlighter-rouge">POST</code> to retrieve information (<a href="https://blogs.dropbox.com/developers/2015/03/limitations-of-the-get-method-in-http/">with one exception</a>).</p>
<p>See you next time.</p>
Behind Monty Hall's Closed Doors - Our Limited Minds2016-06-26T00:00:00+00:00https://codeahoy.com/2016/06/26/behind-monty-halls-closed-doors-our-limited-minds<p>There’s a classic brain-teaser in the field of probability that goes like this:</p>
<blockquote>
<p>Imagine that you’re on a television game show and the host presents you with three closed doors. Behind one of them, sits a sparkling, brand-new Lincoln Continental; behind the other two, are smelly old goats. The host implores you to pick a door, and you select door <em>1</em>. Then, the host, who is well-aware of what’s going on behind the scenes, opens door <em>3</em>, revealing one of the goats.</p>
<p>“Now,” he says, turning toward you, “do you want to keep door <em>1</em>, or do you want to switch to door <em>2</em>?”</p>
</blockquote>
<p><img src="https://codeahoy.com/img/blogs/montyhall.jpg" alt="Would you switch" /></p>
<!--more-->
<p><strong>Would you switch when given the choice or stick to your guns</strong>? If you haven’t heard this puzzle before, think about your answer carefully. It might just surprise you.</p>
<p>Go on, think about it. Continue reading only after you have thought of an answer.</p>
<p>Most people who attempt this problem reach the conclusion that the odds don’t change and it doesn’t matter whether you switch or stick to your original choice. <strong>The correct answer, however, is that you have a much better chance of winning if you switch</strong>. Sal Khan explains the math behind getting better odds of wining by switching in this <a href="https://www.khanacademy.org/math/precalculus/prob-comb/dependent-events-precalc/v/monty-hall-problem">video</a>. You can even try it for yourself <a href="http://math.ucsd.edu/~crypto/cgi-bin/MontyKnows/monty2?2+611">here</a>.</p>
<p>This problem is know as the <a href="https://en.wikipedia.org/wiki/Monty_Hall_problem">Monty Hall Problem</a>. It is named after the host of a television gameshow “Let’s Make a Deal”. What’s interesting about this puzzle is the <strong>controversy it caused in the academic world</strong>. It started when <a href="https://en.wikipedia.org/wiki/Marilyn_vos_Savant">Marilyn vos Savant</a>, listed in the The Guinness Book of Records for highest recorded IQ, <a href="http://marilynvossavant.com/game-show-problem/">answered the puzzle</a> in her column in response to a question from one of her readers. She wrote:</p>
<blockquote>
<p><strong>Yes; you should switch</strong>. The first door has a <em>1/3</em> chance of winning, but the second door has a <em>2/3</em> chance. Here’s a good way to visualize what happened. Suppose there are a million doors, and you pick door <em>1</em>. Then the host, who knows what’s behind the doors and will always avoid the one with the prize, opens them all except door <em>777,777</em>. You’d switch to that door pretty fast, wouldn’t you?</p>
</blockquote>
<p>Her response provoked outrage from thousands of academics who insisted that she is wrong - the odds do not change:</p>
<blockquote>
<p>I’m receiving <strong>thousands of letters, nearly all insisting that I’m wrong</strong>, including the Deputy Director of the Center for Defense Information and a Research Mathematical Statistician from the National Institutes of Health! Of the letters from the general public, <strong>92% are against my answer, and and of the letters from universities, 65% are against my answer. Overall, nine out of ten readers completely disagree with my reply</strong>.</p>
</blockquote>
<p>It’s just mind boggling that the puzzle evaded so many talented mathematicians and scientists who vehemently criticized Marilyn. Here’s what one PhD wrote:</p>
<blockquote>
<p>You blew it, and you blew it big!</p>
<p>Since you seem to have difficulty grasping the basic principle at work here, I’ll explain. After the host reveals a goat, you now have a one-in-two chance of being correct. Whether you change your selection or not, the odds are the same. There is enough mathematical illiteracy in this country, and we don’t need the world’s highest IQ propagating more. Shame!</p>
<p><em>Scott Smith, Ph.D., University of Florida</em></p>
</blockquote>
<p>From a true statistical standpoint, Marilyn is 100% correct. So why did so many experts have trouble wrapping their heads around this problem? <a href="http://www.nytimes.com/1991/07/21/us/behind-monty-hall-s-doors-puzzle-debate-and-answer.html">According</a> to Stanford professor Persi Diaconis:</p>
<blockquote>
<p>“Our brains are just not wired to do probability problems very well, so I’m not surprised there were mistakes,”</p>
</blockquote>
<p>So there’s a psychological side to the puzzle and, in reality, <strong>most contestants did not accept the switch</strong> and <a href="http://www.nytimes.com/1991/07/21/us/behind-monty-hall-s-doors-puzzle-debate-and-answer.html">sticked to their initial choice</a>:</p>
<blockquote>
<p>Mr. Hall said he was not surprised at the experts’ insistence that the probability was 1 out of 2. “That’s the same assumption contestants would make on the show after I showed them there was nothing behind one door,” he said. “<strong>They’d think the odds on their door had now gone up to 1 in 2, so they hated to give up the door no matter how much money I offered</strong>. By opening that door we were applying pressure. We called it the Henry James treatment. It was ‘The Turn of the Screw.’”</p>
</blockquote>
<p><a href="http://priceonomics.com/the-time-everyone-corrected-the-worlds-smartest/j">Apparently</a> for contestants and experts alike, the Monty Hall Problem causes <a href="https://en.wikipedia.org/wiki/Cognitive_dissonance">cognitive dissonance</a> which is the mental stress when an individual holds two or more contradictory beliefs or is presented with new information that conflicts with their existing beliefs:</p>
<blockquote>
<p>When people are confronted with evidence that is “inconsistent with their beliefs” (ie. the odds of winning by switching doors being ⅔, instead of ½), <strong>they first respond by refuting the information, then band together with like-minded dissenters and champion their own hard-set opinion</strong>. This is precisely the mentality of vos Savant’s thousands of naysayers.</p>
</blockquote>
<p>The way I see it: after one door is eliminated, most people <em>believe</em> that they have a 50/50 percent change of winning a brand new car or a smelly goat since there are only two doors left - the distribution is uniform. They believe in it so strongly that even after hearing correct explanations, that the distribution isn’t uniform, they refuse to believe in them because they conflict with their initial, more strongly held belief. They conveniently ignore the new information. <strong>The Monty Hall Problem is so counter-intuitive to our beliefs that <a href="https://en.wikipedia.org/wiki/Paul_Erd%C5%91s">Paul Erdos</a>, one of the most brilliant mathematicians of the last century, rejected Marilyn’s correct solution and remained unconvinced until he was shown a computer simulation</strong>. According to <a href="https://blog.codinghorror.com/monty-hall-monty-fall-monty-crawl/">Jeff Atwood</a>, Paul Erdos finally realized his own limits:</p>
<blockquote>
<p>Paul Erdos was brilliant, but even he realized his own limits when presented with the highly unintuitive Monty Hall problem. For his epitaph, he suggested, in his native Hungarian, “Végre nem butulok tovább”. This translates into English as “<strong>I’ve finally stopped getting dumber</strong>.”</p>
<p>If only the rest of us could be so lucky.</p>
</blockquote>
What is Serverless Architecture? AWS Lambda Features (2020)2016-06-25T00:00:00+00:00https://codeahoy.com/2016/06/25/serverless-architectures--lets-ditch-the-servers<p>Serverless is the new buzzword that is <a href="http://martinfowler.com/articles/serverless.html">quickly</a> <a href="http://highscalability.com/blog/2015/12/7/the-serverless-start-up-down-with-servers.html">gaining</a> <a href="https://www.manning.com/books/serverless-architectures-on-aws" rel="nofollow">momentum</a> and attention.</p>
<p>The concept is to be able to run server-side code without worrying about the messy details of provisioning and setting up servers, disk drives and other resources. You write code, upload it and — voilà! — it starts running. All the complications of managing the infrastructure, provisioning servers, auto-scaling, installing languages and frameworks are eliminated and hidden away by the cloud provider (AWS, Azure, Google Cloud). The cloud provider takes care of allocating and managing the resources, invoking the code in response to a request, providing it the context and input information it needs to do its job and return the result to the client. By focusing less time on managing scaling and availability, software developers are increasingly using serverless architectures for more advanced workloads.</p>
<!--more-->
<h2 id="functions-as-a-service">Functions as a Service</h2>
<p>There is no clear view or consensus on what serverless is; for many people, it means writing your code as <strong>function</strong> and giving it to cloud providers for execution. This is referred to as <em>Functions as a Service** or *FaaS</em>. This view of serverless is the main focus of this article.</p>
<p>All major cloud vendors provide FaaS:</p>
<ul>
<li><a href="https://aws.amazon.com/lambda/">AWS Lambda</a> on AWS. The most popular implementation of FaaS.</li>
<li><a href="https://azure.microsoft.com/en-us/services/functions/" rel="nofollow">Azure Functions</a> on Microsoft Azure</li>
<li><a href="https://cloud.google.com/functions/" rel="nofollow">Cloud Functions</a> on Google Cloud</li>
</ul>
<p>Going serverless requires a different approach to application design. The backend service is broken down into stand-alone functions that perform a single task in response to a user action or event. In <strong>serverless</strong> architectures, the backend is composed of thin, single-purpose functions that are event driven. The business logic shifts from the backend to the client e.g. mobile app. It becomes the main orchestrator, calling various functions to perform some action for the user when needed. For example, running in serverless architecture, a photo-sharing app like Instagram might call one function to upload image to the server followed by another call to a function that reads all the followers’ information from a database and notifies them.</p>
<p>Serverless architectures require smart clients that know about and talk to a wide range of remote functions. While mobile app developers have had rich frameworks and platforms that allowed them to build complex logic on the client easily, things weren’t so simple for web applications. But thanks to rich client-side application frameworks like React and Angular, and <a href="http://codeahoy.com/2016/04/23/what-is-http2/">fast HTTP/2 protocol</a>, it is now possible to build complex applications seamlessly into the browser. This will help drive the serverless trend even further.</p>
<h2 id="amazon-lambda---features-pros-and-cons">Amazon Lambda - Features, Pros and Cons</h2>
<p>Amazon Web Services (<em>AWS</em>), the undisputed leader of cloud computing, launched a product called <a href="https://aws.amazon.com/lambda/">Lambda</a> for serverless applications back in 2014.</p>
<blockquote>
<p>AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is <strong>no charge</strong> when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all <strong>with zero administration</strong>. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.</p>
</blockquote>
<p>I gave it a try and converted one of our small microservices into a Lambda function.</p>
<p>Setting up a Lambda function in AWS was straightforward. The only challenge I faced was connecting the function to another AWS product, the <a href="https://aws.amazon.com/api-gateway/" rel="nofollow">API Gateway</a>, to expose it as a <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a> end-point. Here are the <strong>pros and cons</strong> I discovered during the process.</p>
<h4 id="1-lambda-programming-languages">1. Lambda Programming Languages</h4>
<p>AWS Lambda supports not all, but major languages. Java 8, Python, Go, Powershell, Ruby, C# and Node.js. In addition, there is a Runtime API which allows you to use any programming languages for your functions.</p>
<p>Another thing which I liked about Lambda is that there are many <strong>templates</strong> to choose from when creating a new Lambda function that could come very handy. They contain examples on how to access various databases and integrate with other AWS products and services.</p>
<h4 id="2-lambda-latency--cold-starts">2. Lambda Latency & Cold Starts</h4>
<p>Up until recently, Lambda startup times were an issue for latency-sensitive applications. ‘Cold functions’ are those that haven’t run in some time. When a new request or event triggers a cold Lambda function, the cloud provider needs to find an environment (server) to load the function and its related resources into and run. This usually took 50 to 500 milliseconds. Once Lambda is loaded onto a server, it will stay there for some time ~ 30 minutes or so. If a new request comes in, it is executed immediately because the function is already loaded. This might not sound like an issue, but for latency-sensitive applications that dealt with thousands of simultaneous requests with irregular traffic patterns, warming up functions to keep them ready was a problem.</p>
<p>AWS recently introduced <strong><a href="https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/">Provisioned Capacity</a></strong> that fixes this issue by allowing developers to specify the number of warm instances they want to keep ready always to handle incoming requests. This is an excellent feature that gives developers greater control on ensuring low latencies and faster response times. Provisioned capacity isn’t free so please do check the pricing so you understand the costs before you use it.</p>
<h4 id="3-lambda-execution-duration-limit">3. Lambda Execution Duration Limit</h4>
<p>The execution duration of lambda functions has an upper time limit. It is currently at 15 minutes. This is more than sufficient for many use cases but could be an issue for batch type applications or tasks that are performing a long running-task like converting videos, etc. I don’t see this as a huge issue.</p>
<h4 id="4-pricing">4. Pricing</h4>
<p>The <a href="https://aws.amazon.com/lambda/pricing/">pricing</a> is based on the number of requests <em>and</em> the duration of script’s execution, billed in 100 millisecond increments. So if a Lambda function runs for 15 milliseconds, it will be billed for 100. This could be an issue for very high-volume applications with lots of short-running functions. A crude hack to get the best bang for the buck would be to combine short-running Lambda operations into a larger one. Also, if you want to expose your Lambda methods as REST end-points using AWS API Gateway, you’d incur extra costs as the API Gateway has <a href="https://aws.amazon.com/api-gateway/pricing/" rel="nofollow">separate pricing</a>.</p>
<h4 id="5-lambda-is-stateless">5. Lambda is Stateless</h4>
<p>Lambda functions are stateless and asynchronous. Each function invocation has no idea about the state of previous invocations and its output or state isn’t automatically available to subsequent functions. You can still access external data by calling other services such as S3 or ElastiCache.</p>
<p>It would be wonderful to share a few things like connection pools, that are expensive to setup. Connection pooling isn’t <a href="https://forums.aws.amazon.com/thread.jspa?threadID=216000" rel="nofollow">properly supported</a>. Setting up and tearing down database connections for each request increases latency and affect performance. Although there are work arounds, like using <a href="https://aws.amazon.com/rds/proxy/" rel="nofollow">Amazon RDS Proxy</a> to maintain connection pools.</p>
<h4 id="6-lambda-debugging-and-logging">6. Lambda Debugging and Logging</h4>
<p>Debugging and logging isn’t easy and has a learning curve. When testing my Lambda function, I spent a lot of time scrolling through <a href="https://aws.amazon.com/cloudwatch/" rel="nofollow">CloudWatch</a> entries to find issues.</p>
<h4 id="7-lambda-cicd">7. Lambda CI/CD</h4>
<p>Lambda functions can be setup for automatic deployments through CI/CD pipelines. You could host your function code on GitHub, setup a new pipeline using AWS CodePipeline and then use AWS CodeBuild to build and deploy the function.</p>
<h2 id="future-of-serverless-vs-server-should-we-ditch-the-servers">Future of serverless vs server. Should we ditch the servers?</h2>
<p>While the AWS Lambda and serverless haven’t yet broken the threshold, <strong>the future looks promising</strong>. Going serverless requires a shift in thinking, re-inventing tooling and setting up the right processes for version control, deployments, cost monitoring and control, monitoring, testing, security, etc.</p>
<p>No, developers are not going to ditch the servers and move everything to servless. Instead, developers are adopting serverless for certain use cases where it fits great, while still continuing to use servers, containers and microservices.</p>
<p>Serverless is a great concept. Infrastructure management is challenging and can be <em>very</em> painful and requires a dedicated team who can manage resources. It shifts the focus away from the real problem to the undifferentiated heavy lifting of managing servers, auto-scaling groups, instance tagging, and even worse, building infrastructure specific logic in applications like the health checks. Development time increases because developers now have the burden of managing their infrastructure. That’s the biggest beef I have with DevOps: it forces skilled developers to spend their time and energy worrying about infrastructure intricacies instead of building useful applications that solve real problems. Serverless architectures take some of these barriers away and reduce the friction that allows developers to get started quickly.</p>
<p><em>Updated: March 1, 2020</em></p>
Blameless Postmortems - Examining Failure Without Blame2016-06-20T00:00:00+00:00https://codeahoy.com/2016/06/20/blameless-postmortems---examining-failure-without-blame<p>Let’s face it: failure is inevitable in complex systems. It cares not for the number of tests you ran, code reviews or your monitoring tools. It just happens. And how is failure usually dealt with? Instead of learning from it to improve the resilience of the system, <strong>the traditional view is to assign blame and point fingers at individuals responsible for the failure</strong>. It’s easier to identify a culprit than to find the real cause. In <a href="https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648265">The Field Guide to Understanding Human Error</a>, author <a href="http://sidneydekker.com/">Sidney Dekker</a> refers to this as the “old view” that leads us nowhere:</p>
<!--more-->
<blockquote>
<p>When faced with a human error problem, you may be tempted to ask ‘<strong>Why didn’t they watch out better? How could they not have noticed?</strong>’. You think you can solve your human error problem by telling people to be more careful, by reprimanding the miscreants, by issuing a new rule or procedure. These are all expressions of <strong>‘The Bad Apple Theory’, where you believe your system is basically safe if it were not for those few unreliable people in it</strong>. This old view of human error is increasingly outdated and will lead you nowhere. <strong>The new view, in contrast, understands that a human error problem is actually an organizational problem</strong>.</p>
</blockquote>
<p>When employees are blamed and shamed by their superiors, who have the the power of hindsight on their side, few things happen:</p>
<ul>
<li>Employees become defensive and lose motivation. <strong>The overall team sociology and culture suffers</strong>.</li>
<li>Employees start hiding mistakes. The team and the company doesn’t learn any lessons and nothing is done to prevent failures from happening again.</li>
<li>No one actually takes the responsibility and everybody blames each other.</li>
</ul>
<p>So how should companies handle mistakes?</p>
<p>When failure occurs, the role of the management should be to figure out what happened so they can improve something to prevent it from happening again. <strong>But the management doesn’t have a crystal ball that can give out all the details</strong>. They have to rely on their employees for this information. In order for employees to come forward and admit responsibility for their mistakes, the right type of culture and environment must be present. One that doesn’t punish people for their mistakes. It requires a <strong>“Just Culture”</strong>. The CTO of Etsy, John Allspaw, <a href="https://codeascraft.com/2012/05/22/blameless-postmortems/">describes it</a>:</p>
<blockquote>
<p>Having a <strong>Just Culture</strong> means that you’re making effort to balance safety and accountability. It means that <strong>by investigating mistakes in a way that focuses on the situational aspects of a failure’s mechanism and the decision-making process of individuals</strong> proximate to the failure, an organization can come out safer than it would normally be if it had simply punished the actors involved as a remediation.</p>
<p><strong>Having a “blameless” Post-Mortem process</strong> means that engineers whose actions have contributed to an accident can give a detailed account of:</p>
<ul>
<li>what actions they took at what time,</li>
<li>what effects they observed,</li>
<li>expectations they had,</li>
<li>assumptions they had made,</li>
<li>and their understanding of timeline of events as they occurred.</li>
</ul>
<p>…and that they can give this detailed <strong>account without fear of punishment or retribution</strong>.</p>
</blockquote>
<p>For management, it starts with hiring good people and assuming that everyone involved in the failure had good intentions; they didn’t think that the mistake was possible. If you have incompetent turkeys on your team, no amount of processes or metrics will save the project. After you have hired good people, trust them to make the right decisions. <strong>Managers who constantly question the competence of their teams can’t build productive teams</strong>.</p>
<p>When failures occur, the goal should be to understand ‘<em>what</em>’ caused the failure without focusing on the ‘<em>who</em>’. There are various techniques to get to the <a href="https://en.wikipedia.org/wiki/Root_cause">root cause</a>. I have been using a technique known as the ‘5 Whys’ which works really well in most situations. It was developed by <a href="https://en.wikipedia.org/wiki/Sakichi_Toyoda">Sakichi Toyoda</a> and its goal is to:</p>
<blockquote>
<p>determine the root cause of a problem by <strong>repeating the question “Why?” Each question forms the basis of the next question</strong>.</p>
</blockquote>
<p>Here’s an example of 5 Whys in practice from <a href="http://www.joelonsoftware.com/items/2008/01/22.html">Joel’s blog</a>:</p>
<blockquote>
<p>[Problem:] <strong>Our link to Peer1 NY went down</strong></p>
<ul>
<li><em>Why?</em> – Our switch appears to have put the port in a failed state</li>
<li><em>Why?</em> – After some discussion with the Peer1 NOC, we speculate that it was quite possibly caused by an Ethernet speed / duplex mismatch</li>
<li><em>Why?</em> – The switch interface was set to auto-negotiate instead of being manually configured</li>
<li><em>Why?</em> – We were fully aware of problems like this, and have been for many years. But - we do not have a written standard and verification process for production switch configurations.</li>
<li><em>Why?</em> – Documentation is often thought of as an aid for when the sysadmin isn’t around or for other members of the operations team, whereas, it should really be thought of as a checklist.</li>
</ul>
</blockquote>
<p>For more comprehensive and formal analysis that goes beyond ‘first stories’, <a href="https://codeascraft.com/2012/05/22/blameless-postmortems/">John Allspaw</a> collecting “second stories” from multiple sources and perspectives as part of the formal postmortem process:</p>
<blockquote>
<p>From <a href="https://www.amazon.com/Behind-Human-Error-David-Woods/dp/0754678342">Behind Human Error</a> here’s the difference between “first” and “second” stories of human error:</p>
</blockquote>
<table>
<thead>
<tr>
<th>First Stories</th>
<th>Second Stories</th>
</tr>
</thead>
<tbody>
<tr>
<td>Human error is seen as cause of failure</td>
<td>Human error is seen as the effect of systemic vulnerabilities deeper inside the organization</td>
</tr>
<tr>
<td>Saying what people should have done is a satisfying way to describe failure</td>
<td>Saying what people should have done doesn’t explain why it made sense for them to do what they did</td>
</tr>
<tr>
<td>Telling people to be more careful will make the problem go away</td>
<td>Only by constantly seeking out its vulnerabilities can organizations enhance safety</td>
</tr>
</tbody>
</table>
<p>A <em>Just Culture and blameless postmortems</em> aren’t about avoiding accountability. In fact they achieve the opposite effect by <strong>creating a culture</strong> where people can freely admit their mistakes and learn from them. <strong>A mistake can be a great opportunity to learn a valuable lesson from</strong>. When failure happens, a thorough analysis is performed, with an emphasis on <strong>process over people, to make it better</strong>. These analyses or ‘postmortems’ should be performed <em>after</em> the problem has been solved; when emotions aren’t running high. The results or findings must be shared with the entire team or the company.</p>
<p>It’ll be helpful to keep in mind that <strong>most people have a natural tendency to assign blame to others</strong>. Some do it more explicitly than the others. As managers when you are trying to create a <em>Just Culture</em>, you need to keep your eyes and ears open and when <strong>you detect finger pointing, you must tactfully shift the focus away from people and back to the process</strong>.</p>
Continuous Delivery - Automating the Release Process2016-06-18T00:00:00+00:00https://codeahoy.com/2016/06/18/continuous-delivery---automating-the-release-process<p>For many software developers, <strong>release days are stressful events</strong>. There’s always some risk that things might go wrong in the process or that a bug would surface in production. At my <a href="http://starscriber.com/">previous company</a>, we had a manual release process that was very human intensive; hence, error prone. On release days, DevOps would load binaries on a staging environment and perform user acceptance tests (UAT) manually. If the tests were successful, the software was copied to production servers and verified with smoke tests and occasionally, a trimmed down version of the UAT was run again. Here are the common problems we faced:</p>
<!--more-->
<ul>
<li>2 out 3 times when we had to roll back a release because of an issue, it was due to a <strong>configuration mismatch</strong> between staging and production environments.</li>
<li>The release process was <strong>slow</strong> and it took a long time to put new features in the hands of our users. It wasn’t uncommon for the release process to take days or sometimes even weeks.</li>
<li>The slow release process and the manual UAT had another side-effect: <strong>the developers didn’t get timely feedback</strong>. By the time feedback arrived, they were often in the middle of another feature. This incurred additional overhead because their memories were no longer fresh and on rare occasions, the error got re-introduced due to mix-up between branches.</li>
</ul>
<p>In short, manual and ad-hoc release processes are suboptimal and the release day is fraught with stress. In our case, it was tolerable until releases became more frequent and the team grew. To improve and automate the release process, there’s a software engineering approach know as Continuous Delivery (CD).</p>
<p><strong>Continuous delivery makes it possible to release new features quickly and reliably. It provides fast feedback to developers</strong>. The software is built in a way where it <em>can</em> be automatically and safely released to production at any time. This is ensured by <strong>delivering every change to a production-like environment and running extensive automated tests on it</strong>. According to <a href="http://martinfowler.com/bliki/ContinuousDelivery.html">Martin Fowler</a>, you are doing continuous delivery if:</p>
<blockquote>
<p>[…] a business sponsor could request that <strong>the current development version of the software can be deployed into production at a moment’s notice</strong> - and nobody would bat an eyelid, let alone panic.</p>
</blockquote>
<p><strong>Continuous delivery has a pre-requisite known as Continuous Integration (CI)</strong>. CI is an <a href="http://www.extremeprogramming.org/">Extreme Programming</a> practice which requires that the new changes are integrated into the main branch as soon as they are finished so that <strong>the project is always kept in a working state</strong>. Usually, it works like this: developers commit their changes to GitHub (or another VCS) which triggers the build process. The entire application along with the required dependencies is built and <strong>a comprehensive set of unit and integration tests are run against it</strong>. If the tests fail, the team stops working and fixes the issue until the software is in working state again. Without CI, integration can very easily become a nightmare. Continuous integration is great and it’s one of the very first things I do when starting a new project.</p>
<p>I have seen many examples where <strong>teams stop paying attention to broken builds</strong>. This usually happens when the CI process becomes a big, hairy beast. This defeats the main goal of CI: <strong>broken builds should never be ignored and fixing them should be the top priority for the team</strong>. To ensure that it happens, the CI process should be kept short, sweet and simple. If tests take too long to run, are unreliable or can’t help pin-point the problem, the team will quickly stop paying attention to the broken builds or even worse, finger-pointing like “<em>the other project broke the build</em>” will ensue in dysfunctional teams.</p>
<p>Continuous integration is mainly focused on development teams. <strong>It’s possible to have CI with a manual release process</strong>. In our case, we had CI but the binaries and corresponding configuration files were manually copied to staging and production environments. In contrast, <strong>the continuous delivery automates the release process end-to-end</strong>. To achieve that goal, it uses a pipeline with distinct stages and their associated processes.</p>
<p>A <strong>continuous delivery pipeline</strong> is a manifestation of your release process for getting a new release out of the door. According to <a href="http://martinfowler.com/bliki/DeploymentPipeline.html">Martin Fowler</a>:</p>
<blockquote>
<p>One of the challenges of an automated build and test environment is you want your build to be fast, so that you can get fast feedback, but comprehensive tests take a long time to run. <strong>A deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing</strong>. Deployment pipelines are a central part of ContinuousDelivery.</p>
</blockquote>
<p>A typical CD pipeline might look like the following:</p>
<p><img src="https://codeahoy.com/img/blogs/cd_pipeline.png" alt="Continuous Delivery Pipeline" /></p>
<p>A crucial part that determines the success of CD pipelines is well-written <strong><a href="http://www.extremeprogramming.org/rules/functionaltests.html">acceptance tests</a></strong> that are run in the later stages of the pipeline for “<em>more through probing</em>”. They ensure that the software satisfies user requirements and specifications. Acceptance tests should not be exposed to internal system details and should treat it like a <strong>black-box</strong>. Our acceptance tests consisted of input that a real user would provide and the tests received and checked the system’s output to verify that it matches expectation.</p>
<p><strong>Transition from one stage to the next in the CD pipeline could be automatic or manual</strong>. Manual doesn’t imply copying artifacts manually to the next stage. It means that someone needs to signal the system that it’s okay to <strong>transition to the next stage usually at a push of a button</strong>.</p>
<p>Continuous delivery <strong>pipelines are modeled after delivery processes</strong>. There’s no right way: one pipeline may look very different from another. For example, in an <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">SOA</a> project with many stand-alone components, we decided that a single pipeline for all the components was the best solution. Another project required individual pipelines for each component (microservice). The <strong>integration pipeline</strong>, shown in the picture below.</p>
<p><img src="https://codeahoy.com/img/blogs/integration_pipeline.png" alt="Integration Pipeline" /></p>
<p>Implementing a good continuous delivery pipeline can be a <em>daunting task</em> but yields great benefits if done right. In my opinion, the best way is to carefully study your deployment process, understand all the dependencies, get buy-in from the team and start with something small and simple.</p>
<h2 id="continuous-delivery-vs-continuous-deployment">Continuous Delivery vs Continuous Deployment</h2>
<p>In continuous <strong>delivery</strong>, a human makes the final call to deploy the release to production environment. Typically, the release to production happens several changes later or at some fixed date.</p>
<p>Continuous <strong>deployment</strong> evolves the continuous <strong>delivery</strong> process one step further: every change that happens to pass automated tests is <strong>deployed to production automatically</strong>. Continuous deployment may not be right for every project and even though it sounds great in theory, and I’m sure it is, I’ve not really tried it in a commercial project.</p>
<p>Here’s a nice picture comparing the continuous delivery and deployment processes taken from <a href="http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment">Yassal Sundman’s blog</a>:</p>
<p><img src="https://codeahoy.com/img/blogs/continuous-delivery-deployment-sm.jpg" alt="Continuous Delivery vs Continuous Deployment" /></p>
<p>As far as continuous delivery tools are concerned, I don’t have a personal preference. I’ve recently started using <a href="https://aws.amazon.com/codepipeline/">AWS CodePipeline</a> (along with <a href="https://aws.amazon.com/codedeploy/">AWS CodeDeploy</a>) to automate the delivery process on the AWS cloud and I’m generally satisfied with it.</p>
The Law of Demeter - Writing Shy Code2016-06-17T00:00:00+00:00https://codeahoy.com/2016/06/17/the-law-of-demeter---writing-shy-code<p>In all my years of building server-side applications, I have come to believe that the single most important aspect that determines the long term success of these projects isn’t the speed of algorithms or the fancy frameworks. <em>Nope</em>. It’s the complexity of the code. <strong>Unmanaged complexity has a profound effect on the maintainability of large projects</strong>. Applications that are difficult to understand aren’t amenable to refactoring. Introducing <strong>new features become a slow and painful process, increasing the crucial time to market</strong>. I’ve seen complicated systems where developers were petrified of making even small changes in the fear that they might inadvertently break some other part. Or the spaghetti code that is only understood by a single individual or a handful of developers who get a free pass on anything because the project will be doomed if they quit.</p>
<!--more-->
<p>There are multiple factors that contribute to code complexity. <strong>One important factor is the amount of <a href="https://en.wikipedia.org/wiki/Coupling_(computer_programming)">coupling</a> or interdependencies between the application’s modules</strong>. Let’s walkthrough a trivial example. Suppose there’s a server that allows users to connect. When users connect and authenticate, they are wrapped in a ‘User’ class:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Represents a user</span>
<span class="kd">class</span> <span class="nc">User</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">username</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">id</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">Socket</span> <span class="n">socket</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">disconnect</span><span class="o">()</span> <span class="o">{</span>
<span class="n">socket</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
<span class="o">}</span>
<span class="c1">// Other methods that operate on the socket.</span>
<span class="o">}</span>
</code></pre></div></div>
<p>If we want to send a message a user, we would do something like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Another class</span>
<span class="kd">class</span> <span class="nc">Message</span> <span class="o">{</span>
<span class="c1">// Send "Hello." string to a User</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">sayHello</span><span class="o">(</span><span class="nc">User</span> <span class="n">user</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Socket</span> <span class="n">s</span> <span class="o">=</span> <span class="n">user</span><span class="o">.</span><span class="na">socket</span><span class="o">;</span> <span class="c1">// Get the user's socket</span>
<span class="nc">OutputStream</span> <span class="n">outputStream</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="na">getOutputStream</span><span class="o">();</span>
<span class="nc">PrintWriter</span> <span class="n">out</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PrintWriter</span><span class="o">(</span><span class="n">outputStream</span><span class="o">);</span>
<span class="n">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello."</span><span class="o">);</span> <span class="c1">// send the message to the socket.</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The <em>User</em> class has an obvious flaw: it failed to encapsulate the <em>socket</em> object and leaked it to the world. If we want to change this implementation in the future (e.g. use an asynchronous socket library), we’ll have to make changes in many places.</p>
<p>However, the <em>sayHello(…)</em> method of the <em>Message</em> class isn’t entirely innocent. <strong>It sinned in the manner in which it interacted with the <em>socket</em> object</strong>. It <strong>obtained access to an independent “third-party” object</strong> (<em>socket</em>) and used it directly. The example might be contrived, but I have seen this pattern far too many times in “real-world” applications. The good news is that <strong>this can be easily detected using a technique with a fancy name: <a href="http://www.ccs.neu.edu/research/demeter/papers/law-of-demeter/oopsla88-law-of-demeter.pdf">The Law of Demeter</a></strong>. The “law” (the term itself is a misnomer. It’s rather a technique or a guideline) can be <a href="https://en.wikipedia.org/wiki/Law_of_Demeter">summarized</a> as:</p>
<blockquote>
<ul>
<li>Each unit should have only <strong>limited knowledge about other units</strong>: only units “closely” related to the current unit.</li>
<li>Each unit should only talk to its friends; <strong>don’t talk to strangers</strong>.</li>
<li><strong>Only talk to your immediate friends</strong>.</li>
</ul>
<p>The fundamental notion is that <strong>a given object should assume as little as possible about the structure or properties of anything else</strong> (including its subcomponents), in accordance with the principle of “information hiding”.</p>
</blockquote>
<p>In short, tight coupling between logically independent modules violates the Law of Demeter. Although, it is a side effect of poor encapsulation, <strong>the law discourages direct access and use of third-party objects</strong>. Here’s a good <a href="http://pmd.github.io/pmd-5.1.3/rules/java/coupling.html">example</a> to illustrate the Law of Demeter:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Foo</span> <span class="o">{</span>
<span class="cm">/**
* This example will result in two violations.
*/</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">example</span><span class="o">(</span><span class="nc">Bar</span> <span class="n">b</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// this method call is ok, as b is a parameter of "example"</span>
<span class="no">C</span> <span class="n">c</span> <span class="o">=</span> <span class="n">b</span><span class="o">.</span><span class="na">getC</span><span class="o">();</span>
<span class="c1">// this method call is a violation, as we are using c, which we got from B.</span>
<span class="c1">// We should ask b directly instead, e.g. "b.doItOnC();"</span>
<span class="n">c</span><span class="o">.</span><span class="na">doIt</span><span class="o">();</span>
<span class="c1">// this is also a violation, just expressed differently as a method chain without temporary variables.</span>
<span class="n">b</span><span class="o">.</span><span class="na">getC</span><span class="o">().</span><span class="na">doIt</span><span class="o">();</span>
<span class="c1">// a constructor call, not a method call.</span>
<span class="no">D</span> <span class="n">d</span> <span class="o">=</span> <span class="k">new</span> <span class="no">D</span><span class="o">();</span>
<span class="c1">// this method call is ok, because we have create the new instance of D locally.</span>
<span class="n">d</span><span class="o">.</span><span class="na">doSomethingElse</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now scroll back up and look at the <code class="language-plaintext highlighter-rouge">sayHello(...)</code> method. It violates the Law of Demeter by talking to a stranger: the <em>socket</em> object. Even though <strong>the method itself is pretty much helpless, it helps us detect tight-coupling</strong>. The problem starts with the <em>User</em> class that failed to hide its internal details. So let’s fix it:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">User</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">username</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">id</span><span class="o">;</span>
<span class="c1">// Make the field private to hide it from the world.</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">Socket</span> <span class="n">socket</span><span class="o">;</span>
<span class="c1">// Simplified implementation to encapsulate messaging functionality</span>
<span class="kt">void</span> <span class="nf">sendMessage</span><span class="o">(</span><span class="nc">String</span> <span class="n">message</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">OutputStream</span> <span class="n">outputStream</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="na">getOutputStream</span><span class="o">();</span>
<span class="nc">PrintWriter</span> <span class="n">out</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">PrintWriter</span><span class="o">(</span><span class="n">outputStream</span><span class="o">);</span>
<span class="n">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">message</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now we can fix the <code class="language-plaintext highlighter-rouge">sayHello(...)</code> method to stop relying on the <em>socket</em> object:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Message</span> <span class="o">{</span>
<span class="c1">// Doesn't violate the Law of Demeter anymore.</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">sayHello</span><span class="o">(</span><span class="nc">User</span> <span class="n">user</span><span class="o">)</span> <span class="o">{</span>
<span class="n">user</span><span class="o">.</span><span class="na">sendMessage</span><span class="o">(</span><span class="s">"Hello."</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>(Coupling) Problem solved. In their book <a href="https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">The Pragmatic Programmer</a>, <a href="https://twitter.com/pragmaticandy">Andrew</a> and Dave suggest writing <strong>“shy” code that doesn’t interact with too many things</strong>.</p>
<p>I keep an eye out for the Law of Demeter violations when writing or reviewing code and <strong>refactor code to reduce unnecessary coupling where it makes sense</strong>. This could be automated with <a href="http://pmd.github.io/">source code analyzers</a>. I haven’t personally used it myself so take my advice with a grain of salt. I’ll update this post if I use it myself.</p>
Remote Software Development - Lessons Learned2016-06-12T00:00:00+00:00https://codeahoy.com/2016/06/12/remote-software-development---lessons-learned<p>Previously, I talked about the ill-fated <a href="http://www.codeahoy.com/2016/04/21/when-to-rewrite-from-scratch-autopsy-of-a-failed-software/">rewrite of our core product</a>. The ‘<a href="https://en.wikipedia.org/wiki/Second-system_effect">second system</a>’, although better and faster than its predecessor, was <em>rejected</em> by the customer. But the story has a silver lining - one thing that we got right in the process that empowered us to grow and thrive: <strong>we built a highly productive remote development team</strong>.</p>
<p>In this post, I’ll share my experiences: how through trial and error (and with luck), we finally got the right formula and made it work for us.</p>
<p><!--more--></p>
<p>Part of the original system team was one of the best <a href="http://www.joelonsoftware.com/items/2009/09/23.html">duct-tape developers</a> I’ve ever worked with. We called him ‘<em>The Code Machine</em>’. He didn’t write clean code, but he wrote it fast as hell and made sure it worked. The original system, done with a gun to our heads, was a big, complex, monolith application and the <strong>boundaries between its modules were ill defined</strong>. Computer scientists refer to this type of code as <a href="https://en.wikipedia.org/wiki/Coupling_(computer_programming)">tightly-coupled</a> and the system is referred to as <a href="https://en.wikipedia.org/wiki/Orthogonality_(programming)">non-orthogonal</a>.</p>
<p>The ‘second system’ had a more grand vision and we needed to grow the team to build it right and on schedule. We hired another developer, a bright, young CS graduate to work on the system. One problem with the original system quickly became evident to me: because the code was tightly-coupled (the auto-generated UML class diagrams resembled a spider’s cobweb), the <strong>new developer couldn’t work independently: changing one part of the system indirectly impacted other parts</strong>. While they were not bickering, the two developers couldn’t get out of each other’s way and the progress was slow. Parallel to all this, <strong>the upper management decided to give ‘remote software development’ a try</strong>. Someone on the board recommended a company from Hyderabad, India and before I even knew it, I was interviewing a bunch of remote Java developers.</p>
<p>The non-orthogonality problem was further magnified when the remote developer joined the team. Because the system was tightly-coupled, he was going to have to learn it all before he could become productive. Somehow, we isolated a few ‘modules’ so that he could start writing features without spending months learning the entire system. But his changes frequently conflicted with the work of the local team and <strong>integration became a nightmare, requiring everyone’s involvement</strong>. This issue aside, I had other concerns as well. It appeared that the remote developer was just ‘covering his ass’ and leaving a paper trail. I hate bureaucracy and don’t work that way. I later learned, that the offshore company had a ‘different’ culture than the one I was trying to build.</p>
<p>Let me stop here to summarize the situation: we had a <strong>complex system</strong> with huge overlap of module responsibilities. Changes rippled through the whole system and affected the entire team. We had a <strong>remote developer in India</strong>, outside of our comfort (time)zone, and integration often forced everyone to work odd hours. To make matters worse, the remote developer had <strong>low morale</strong> and the offshore company we were contracting with, was ripe with dysfunctional politics.</p>
<p>It was then that I decided to slow ourselves down and think about how can we avoid all this in the new design. Since we didn’t have a dedicated Q/A, I assigned that responsibility temporarily to the new developer so he could be productive (it was dick move, but he was allowed time to learn and practice his coding skills and he ended up getting <a href="http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=320">SCJP</a> during that period). The remote developer got re-assigned to a new R&D project that the CEO was cooking up.</p>
<p>When the things calmed down, we realized that three things must happen if we want to scale our development:</p>
<ol>
<li>The new system must be orthogonal. It should be modular and modules should communicate with other only using well defined messages, the public API. I wanted the modules to be 100% blackbox so they could be developed and tested independently… <a href="http://codeahoy.com/2016/05/06/good-abstractions-have-fewer-leaks/">abstractions that do not leak</a>.</li>
<li>Each module must be assigned to a single developer or a team who will 100% own it. I wanted to get around the <a href="https://en.wikipedia.org/wiki/Brooks%E2%80%99_law">Brook’s Law</a> by minimizing communication between local and remote developers as much as possible.</li>
<li>We must do something about the culture and fix the working conditions for our remote developers. We needed the right people who would take full ownership.</li>
</ol>
<h2 id="1-building-orthogonal-system">1. Building Orthogonal System</h2>
<p>In two-dimensional geometry, two lines are orthogonal if they intersect each other at right angles. This affords a kind of <strong>independence</strong>, where you could move along one of the lines without affecting your position projected onto the other line. This simply means that in an orthogonal system, you could easily change the ‘networking layer’ without affecting the ‘database layer’ or the ‘UI layer’. In <a href="https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">Pragmatic Programmer</a>, Andrew and David used an excellent analogy to describe a non-orthogonal system:</p>
<blockquote>
<p>You’re on a helicopter tour of the Grand Canyon when the pilot, who made the
obvious mistake of eating fish for lunch, suddenly groans and faints. Fortunately,
he left you hovering 100 feet above the ground. You rationalize that the collective
pitch lever controls overall lift, so lowering it slightly will start a gentle descent
to the ground. However, when you try it, you discover that life isn’t that simple.
The helicopter’s nose drops, and you start to spiral down to the left. Suddenly you
discover that you’re flying a system where every control input has secondary
effects. Lower the left-hand lever and you need to add compensating backward
movement to the right-hand stick and push the right pedal. But then each of
these changes affects all of the other controls again. Suddenly you’re juggling an
unbelievably complex system, where every change impacts all the other inputs.
Your workload is phenomenal: your hands and feet are constantly moving, trying
to balance all the interacting forces.</p>
<p>Helicopter controls are decidedly not orthogonal.</p>
</blockquote>
<p>We took the layered approach to system design similar to the <a href="https://en.wikipedia.org/wiki/OSI_model">OSI model</a>. The layers we built had following qualities:</p>
<ul>
<li>Each layer had a single major responsibility.</li>
<li>Layers exposed a well-defined API. Communication with other layers was done only using the API.</li>
<li>Layers (with the exception of business logic) communicated with only one other layer.</li>
</ul>
<p>Here’s what the design of the design looked like:</p>
<p><img src="https://codeahoy.com/img/blogs/orthogonal_system.png" alt="orthogonal system" /></p>
<p>In addition to the layers, there were several smaller subsystems not shown in the diagram above such as the ‘Admin UI’ tool and simulators for testing the system end-to-end.</p>
<h2 id="2-circumventing-brooks-law">2. Circumventing Brook’s Law</h2>
<p>Brook’s Law sucks. It basically says that the communication overhead increases as the number of people involved in a project increases. Communication with the remote team that is outside +/- 5 hours of your timezone becomes a burden. We had full 12 hours time difference. So <strong>we decided to extend the principal of single responsibility to developers</strong> and decided that each layer shall have a single owner or a small sub-team located in the same timezone. Initially, I was little hesitant silos that might affect team work and create <a href="https://en.wikipedia.org/wiki/Local_optimum">local optimums</a>, but we had a great team and it never became an issue. The upper management was concerned about ‘<strong>individual owners getting hit by a bus</strong>’ and wanted some level of duplicity. To achieve that, we decided that we’ll keep the system as simple as possible, perform regular code reviews and document it well. Our approach to documentation was simple:</p>
<ul>
<li>For every layer or module, I came up with a <strong>high-level product document</strong>. The goal was to describe the <em>what and why</em> (not how) in simple and concise terms. The document had <strong>a vision that described what the module would do in one sentence and why we were building it, a 10,000 foot overview, core requirements and a list of the things that the module shall not do</strong>.</li>
<li>The developers created design documents. They were a brief description of how the project was organized into sub-packages, followed by <a href="https://en.wikipedia.org/wiki/Interaction_overview_diagram">interaction diagrams</a>. On average, this document was mostly 3-6 diagrams with very little text.</li>
</ul>
<p>The documentation was kept in the GitHub wiki along with the source code.</p>
<p>So we had divided the system in a manner that individual pieces could be assigned independently to remote teams and the instructions were clear enough to minimize communication. We wanted to assign ownership and get out of their way. But there was a big problem: our remote developer wasn’t motivated enough to take ownership. Growing the team over there was out of the question since we had no control over the environment or the culture.</p>
<h2 id="3-getting-the-right-people-on-the-bus">3. Getting the Right People On the Bus</h2>
<p>As the remote developer was finally starting to become somewhat productive, I had concerns about his morale. I reached the conclusion that the things cannot be improved and we terminated the contract.</p>
<p>I was so disheartened at the state of remote software development that I was ready to write it off, but a fellow CTO for a different company told me that <strong>they’re getting great results on oDesk</strong>. I tried to experiment a little and asked for $1500/month budget that was approved. I found an independent consultant and contracted him on a part-time basis to work on our web admin tool. <strong>The guy was a winner</strong>. Whenever I assigned him a new task, he asked questions until he fully understood. He would then go away and send me regular updates on the progress. His releases were on time and his code was pristine. He didn’t program in Java, but luckily had (equally smart) friends who did. It’s a long story but he helped us grow the team and we hired four full time developers and he eventually setup his own office to provide them a space to work from.</p>
<p>I visited the remote team once or twice a year. These visits helped me understand the issues these guys faced and the face-to-face whiteboard meetings enabled them to understand more complex designs easily.</p>
<h2 id="summary">Summary</h2>
<p>I purposely told the story of how my company built a remote software development team instead of giving recommendations up front in the hope that you would draw your own conclusions. In retrospect, we went through trial and error and luckily found the right people that made it work. But despite all of that, it was a strenuous journey. If the remote developers <strong>require constant babysitting and you cannot measure their productivity, it’s probably not working</strong>. If you could afford it, <strong>steer clear of remote developers working +/- 3 hours outside of your time zone</strong>. If you decide to take the offshore route, here are the key takeaways from our experience:</p>
<ul>
<li><strong>Divide the system into independent layers and minimize the overlap between the layers</strong>. Keep the number of layers that have to communicate with one another to a minimum.</li>
<li>Go to extreme lengths to hire the right people; avoid offshore sweatshops or hired guns. Understand that it will require extra time and effort to make them productive. When you find the right people, <strong>treat them with the same respect as you would extend to local developers, make them feel like they are part of the family, and make sure they are being compensated fairly</strong>. Don’t make them stay up late nights to attend meetings. For example, instead of making four guys stay up late to talk to me, I would schedule the weekly meeting during my evening, their morning.</li>
<li><strong>Provide clear requirements</strong>. I cannot stress this enough. Be as concise and as crystal clear as possible. If the requirements aren’t clear, the remote developers might get blocked waiting for answers or even worse, deviate from the requirements and build something entirely different.</li>
</ul>
IDEs and Productivity2016-06-10T00:00:00+00:00https://codeahoy.com/2016/06/10/ides-and-developer-productivity<p>I used to be neutral on the choice and even the use of an <a href="https://en.wikipedia.org/wiki/Integrated_development_environment">IDE</a> for writing code. In university, I learned and used <a href="https://en.wikipedia.org/wiki/Vim_(text_editor)">Vim</a> for assignments. When I started my first job, I switched to a <a href="https://netbeans.org/">popular Java IDE</a> because everyone at work was using it and it featured a nice debugger. Other than debugging and basic auto-completion, I didn’t learn the IDE much in terms of its features. But that changed one day when I ran into an issue compiling a Java project. I went over to the developer’s desk who had made the most recent commit. He opened up the project in his IDE and in <strong>just a few keystrokes, he had the entire <a href="https://blog.jetbrains.com/idea/2010/05/maven-dependencies-diagram/">maven dependency graph</a> displayed on his screen. Just like magic</strong>. Few more keystrokes and he was able to locate the root cause. I was <strong>surprised</strong> because after months of using the IDE, I had no idea it had that feature. Two things happened:</p>
<!--more-->
<ul>
<li>I realized that up until that point, I was using the IDE like a text editor and missing out on many of its powerful features.</li>
<li>I became convinced that <strong>knowing how to effectively use an IDE has a profound impact on developer productivity</strong>. This is especially true for complex enterprise applications, where superior knowledge of IDE can provide a significant boost in productivity.</li>
</ul>
<p>I started out with Vim and loved it. When I first tried Eclipse/NetBeans for Java development, I was overwhelmed by the number things that were <strong>cluttered</strong> on the screen.</p>
<p><img src="https://codeahoy.com/img/blogs/eclipse-clutter.jpg" alt="Eclipse Clutter" /></p>
<p>The second thing that put me off was that I had to use the mouse a lot for navigation. But I saw the value in refactoring, code-completion and generating boiler-plate code so I decided to bite the bullet and switched to it from Vim. In retrospect, it was a good decision.</p>
<p>Many developers who become proficient in text editors like Vim or <a href="https://en.wikipedia.org/wiki/Emacs">Emacs</a> don’t switch to an IDE. I worked with a developer who wrote code exclusively in Emacs. When I hired him, he told that he’ll use nothing but Emacs. I was sure that in time I’ll be able to ‘show him the light’ and that he’ll see the value in using an IDE. But after watching him write code in Emacs, I realized that he was obviously an expert in Emacs and <strong>had more to lose than gain by starting all over again in an IDE</strong>. I didn’t make any attempts to convince him to give up Emacs for an IDE. (On the contrary, he got me thinking that I should perhaps ditch my IDE and switch to Emacs.)</p>
<p>So if you are already using Vim or Emacs and are productive in it, I won’t try to convince you to switch to an IDE. But if you sitting on the fence and don’t have a preference, <strong>I’d strongly recommend picking up a good IDE and learning it really well</strong>. The choice of the IDE will depend on the language you are are programming in. I have been using <strong><a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> for Java development and highly recommend it</strong>. It’s extremely powerful and I feel like I can’t live without the <em>alt-enter</em> shortcut anymore. It <a href="http://martinfowler.com/bliki/PostIntelliJ.html">impressed Martin Fowler</a> after developers in his company voluntarily switched to it:</p>
<blockquote>
<p>The biggest endorsement of IntelliJ came from ThoughtWorks developers. <strong>If anyone suggested a standard IDE for ThoughtWorks projects we needed tear-gas to control the riots</strong>. There were JBuilder zealots, textpad zealots, slickedit zealots - don’t even get me started on the emacs zealots.</p>
<p><strong>Within six months nearly everyone was using IntelliJ. Voluntarily and eagerly</strong>.</p>
</blockquote>
<p>But don’t just use IntelliJ like any other IDE. <strong>Make an effort to learn and understand its features to enhance the development process</strong>. I’d also suggest learning keyboard shortcuts and <strong>start relying on the mouse less and less until you don’t need to use it all when writing code</strong>. I haven’t completely got there myself, but it is possible to do 100% mouse free development in IntelliJ. Watch this presentation by <a href="https://blog.jetbrains.com/idea/author/hhariri/">Hadi Hariri</a> in which he demonstrates how to ditch the mouse and use IntelliJ only with the keyboard: <a href="https://www.youtube.com/watch?v=eq3KiAH4IBI">IntelliJ IDEA Tips and Tricks Full Version</a>.</p>
<h2 id="notes">Notes:</h2>
<ul>
<li><a href="https://blog.jetbrains.com/idea/2016/03/enjoying-java-and-being-more-productive-with-intellij-idea/">Enjoying Java and Being More Productive with IntelliJ IDEA</a></li>
<li>For Python development, I use <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> which is developed by the same guys as IntelliJ IDEA.</li>
<li>I replaced Sublime Text with <a href="https://atom.io/">Atom</a> by GitHub on my machines.</li>
<li>Mark Seemann makes some strong points <a href="http://blog.ploeh.dk/2013/02/04/BewareofProductivityTools/">against the use of productivity tools</a> (including IDEs). You be the judge.</li>
</ul>
Write Less Code2016-06-03T00:00:00+00:00https://codeahoy.com/2016/06/03/write-less-code<p>Not too long ago, I sat down to ‘clean up’ a project that I inherited. I was given the reins of the refactoring efforts because the project has had several bugs in production. It was stuck in a vicious cycle where fixing old bugs would introduce new ones. So I dived into the source code one weekend and the problem soon became evident: <strong>the project was a big, hairy mess</strong>. I use the word <em>big</em> because there was lots of unnecessary, redundant and tightly coupled code. By <em>hairy mess</em>, I don’t mean that the code looked amateur or was full of shortcuts. In fact, the problem was quite the opposite. <strong>There was too much magic</strong> and everywhere I looked, I saw clever and grandiose design practices that had no relationship with the actual problem that the project was built to solve. Things like reflection, aspect oriented programming, custom annotations were all present. The project was an over-engineered beast. To put it into perspective, after the refactoring was over, the module was reduced to less than half of its original size.</p>
<!--more-->
<p>I’m sure the developers who wrote the project did so with the best intentions, but their clever tricks turned against them. They spent a lot of time on periodic maintenance and fixing bugs. The clients were unhappy that the software was full of bugs. The developers felt like shit because everyone was always complaining about the project. But who’s to blame for their misery, for the long hours they had to work to fix the bugs and get no satisfaction out of their jobs? <em>No one else to blame other than the developers themselves</em>. One of my favorite bloggers, Jeff Atwood, wrote that the <a href="https://blog.codinghorror.com/the-best-code-is-no-code-at-all/" rel="nofollow">best code is no code at all</a>:</p>
<blockquote>
<p>It’s painful for most software developers to acknowledge this, because they love code so much, <strong>but the best code is no code at all</strong>. Every new line of code you willingly bring into the world is code that has to be debugged, code that has to be read and understood, code that has to be supported. Every time you write new code, you should do so reluctantly, under duress, because you completely exhausted all your other options. <strong>Code is only our enemy</strong> because there are so many of us programmers writing so damn much of it. If you can’t get away with no code, the next best thing is to <strong>start with brevity</strong>.</p>
</blockquote>
<p>Jeff’s point is undeniable. As developers, we have an itch to come up with clever solutions that we think will make us look professional or help us learn a new tool or technology. We build complex layers to solve simple problems and justify them as being “actually necessary”. But we must realize that <em>the more code we write, the more magic we apply, the more opportunities and doors we leave open for the bugs to creep in</em>. These bugs will come back and haunt us or our successors in the form of overtime required to fix them on time. I’m obviously not talking about using slick tricks to reduce the number of lines of code. Rather we should ask ourselves whether we need to write all that code to solve the actual problem. I’ve seen a couple of custom ORMs and <strong>handmade thread pools</strong> in my career which brings me to another point:</p>
<blockquote>
<p>Don’t reinvent the wheel. Pretty please.</p>
</blockquote>
<p>But don’t just stop there. Think whether that fancy framework is needed at all. A project I worked on used <em>Hibernate</em> along with the complementary <em>DAO</em>’s and <em>DTO</em>’s to execute <strong>one</strong> simple, straight-forward query. Another project had a comprehensive event handling system for a filter that used the <a href="https://docs.oracle.com/javase/tutorial/reflect/" rel="nofollow">reflection API</a> to find and invoke the handler class based on the event type. It was an “ingenious” solution and it took me a while to figure out that the unused methods marked by the IDE were actually invoked using reflection. The icing on the cake: the system handled just one type of event. About five classes worth of code could have been condensed into a simple <code class="language-plaintext highlighter-rouge">if</code> statement:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">event</span><span class="o">.</span><span class="na">type</span> <span class="o">==</span> <span class="no">THE_ONE_TYPE_THIS_SYSTEM_CAN_HANDLE</span><span class="o">)</span> <span class="o">{</span>
<span class="n">process_the_event_and_return_result</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The best code is no code at all and the fastest code is the code that never gets executed. Our goal should be to keep our solutions as simple as possible and stay away from our natural tendencies to over-engineer, use clever tricks and design patterns until it can be proven that they are absolutely necessary to solve the problem. <strong>Complexity</strong> is our worst <strong>enemy</strong>. Unnecessary complexity, even more so because most of the time, <strong><a href="https://codeahoy.com/2017/08/19/yagni-cargo-cult-and-overengineering-the-planes-wont-land-just-because-you-built-a-runway-in-your-backyard/">you aren’t gonna need it</a></strong>.</p>
<p>I’m going to end this post with an excellent piece of advice from Jeff:</p>
<blockquote>
<p>If you love writing code– really, truly love to write code– you’ll love it enough to write as little of it as possible.</p>
</blockquote>
<p>Please share this article if you liked it. It will help the site grow and every share means a lot. Social sharing buttons are below. Thank you!</p>
Why Do Developers Love Music so Much? How to Concentrate in Noisy Open Offices2016-05-28T00:00:00+00:00https://codeahoy.com/2016/05/28/why-do-developers-love-music-so-much<blockquote>
<p>There are a million ways to lose a work day, but not even a single way to get one back - <em>Peopleware</em></p>
</blockquote>
<p>I used to like open floor plans until I had to work in one. On my first day of work at a company with an <strong>open office</strong> plan, I walked through a sea of conjoined desks to my spot and noticed on the way that over three quarters of the staff in the section had their headphones on. I thought to myself that these guys must love their music. It wasn’t until I settled in and tried to read my emails that I noticed how loud the office was. It then striked me that the poor souls were making a hopeless attempt to <strong>block out the noise with their headphones so they could concentrate on their tasks</strong>. Not long after, I was reaching into my laptop bag trying to look for my headphones and at the same time trying to find some music on the Youtube making a vain attempt to get in the zone.</p>
<!--more-->
<p>At my very first job which was at a startup, I had a semi-private office which I shared with a senior developer. The second office could fit two developers or be used as a meeting area when it was available. The outside area was big enough for 4, perhaps 5 developers. The Operations and the I.T. staff sat in a different section that was separated from the engineering offices by a corridor and couple of doors. The engineering offices were generally <strong>very quiet</strong> and there were hardly any distractions to developers.</p>
<p>The office layout looked like this:</p>
<p><img src="https://codeahoy.com/img/blogs/starscriber-offices.png" alt="Starscriber engineering offices" class="center-image" /></p>
<p>It was perfect (except that the windows didn’t have much of a view). We had just the right balance of space and the number of people using it.</p>
<p>Getting back to open floor plans, I think that the biggest problem is not so much with the concept, but with the density of people crammed into a small space. With so many people, there will be many visual and audible distractions like phone calls, conversations, skype calls, FedEx delivery guys walking around etc. This makes it impossible for developers to focus especially when they are trying to concentrate on a hard problem. Symptoms of such environment include developers looking for quiet corners in the office or ‘working from home’ in order to finish their projects on time.</p>
<p>I’m not saying that listening to music is bad - I listen to it myself when I’m doing some trivial task like writing a report or when I’m trying to relax. But I do have an issue when developers <strong>must use music in order to concentrate on their work</strong>. In the <a href="http://www.amazon.com/exec/obidos/ASIN/0932633439/">Peopleware</a> chapter “Bring Back the Door”, there’s a section on an interesting study conducted at the Cornell University on the effects of listening to music while performing programming tasks:</p>
<blockquote>
<p>During the 1960s, researchers at Cornell University conducted a series of tests on the effects of working with music. They polled a group of computer science students and divided the students into two groups, those who liked to have music in the background while they worked (studied) and those who did not. Then they put half of each group together in a silent room, and the other half of each group in a different room equipped with earphones and a musical selection. Participants in both rooms were given a Fortran programming problem to work out from specification. To no one’s surprise, participants in the two rooms performed about the same in speed and accuracy of programming […]</p>
<p>The Cornell experiment, however, contained a <strong>hidden wild card</strong>. The specification required that an output data stream be formed through a series of manipulations on numbers in the input data stream. For example, participants had to shift each number two digits to the left and then divide by one hundred and so on, perhaps
completing a dozen operations in total. Although the specification never said it, the net effect of all the operations was that each output number was necessarily equal to its input number. Some people realized this and others did not. Of those who figured it out, the overwhelming majority came from the quiet room.</p>
<p>Many of the everyday tasks performed by professional workers are done in the serial processing center of the left brain. Music will not interfere particularly with this work, since it’s the brain’s holistic right side that digests music. But not all of the work is centered in the left brain. There is that occasional breakthrough that makes you say “Ahah!” and steers you toward an ingenious bypass that may save months or years of work. The creative leap involves right-brain function. <strong>If the right brain, is busy listening to 1001 Strings on Muzak, the opportunity for a creative leap is lost</strong>.</p>
</blockquote>
<p>Great points.</p>
<p>Taking an action to change the working environment isn’t easy. Most employees and managers who understand that their work environment is counter-productive don’t take an action because they feel that any sort of change is beyond their control or that any attempt to do so would be useless. Unfortunately, this is true. It cost money and some leases even forbid any sort of restructuring or changes to the office layout. But it’s 100% worth it. I have won few battles on this front and lost many. It took <em>over 2 years</em> of convincing and couple of resignations for the stake holders of a software development firm I helped start to move out of a very noisy ‘shared office’ into a quiet one with lots of private and semi-private offices. This “Policy of Default” is illustrated in Peopleware:</p>
<blockquote>
<p>A California company that I consult for is very much concerned about being responsive to its people. Last
year, the company’s management conducted a survey in which all programmers (more than a thousand) were asked to list the best and the worst aspects of their jobs. The manager who ran the survey was very excited
about the changes the company had undertaken. He told me that the number two problem was poor communication
with upper management. Having learned that from the survey, the company set up quality circles, gripe sessions, and other communication programs. I listened politely as he described them in detail. When he was done, I asked <strong>what the number one problem was. “The environment,” he replied. “People were upset about the noise.” I asked what steps the company had taken to remedy that problem. “Oh, we couldn’t do anything about that,” he said.
“That’s outside our control</strong>.”</p>
</blockquote>
<p>Except noise and interruptions, I have no major beef with open office floor plans. And these things are inherent properties of big open floor plans. Noise makes it difficult to concentrate and interruptions forces developers to context switch and getting back into what they were doing takes time: time that is wasted. Open floor plans are counter-productive and have established a culture of underperformance and overtime where developers regularly stay late or work late evenings from home to finish their tasks. Productivity falls and everyone suffers in the long run. May be it’s time to “Bring Back the Door” and provide a quiet, distraction-free environment for the developers to exercise their creativity.</p>
<p><img src="/img/developers_music.png" alt="developer listening music to block out sound and noise" class="center-image" /></p>
Avoid Singletons to Write Testable Code2016-05-27T00:00:00+00:00https://codeahoy.com/2016/05/27/avoid-singletons-to-write-testable-code<p>Often times there is a need to share a single object of a class throughout the code base. For example, we might want to store all online users in one central registry or share a central queue amongst all producer and consumer objects. We want to:</p>
<ul>
<li>ensure that exactly <em>one</em> object of a class exists.</li>
<li>provide a way to get that object.</li>
</ul>
<!--more-->
<p>This is a <strong>valid and a very common requirement</strong> but is often equated and related to a design pattern called <a href="https://sourcemaking.com/design_patterns/singleton">Singleton</a>. While they provide a quick and easy solution, <strong>singletons are considered bad because they make unit testing and debugging difficult</strong>. <a href="https://www.linkedin.com/in/brianbutton">Brian Button</a> has made some valid <a href="https://blogs.msdn.microsoft.com/scottdensmore/2004/05/25/why-singletons-are-evil/">arguments</a> against singletons:</p>
<blockquote>
<p>[Singletons] provide a well-known point of access to some service in your application so that you don’t have to pass around a reference to that service. How is that different from a global variable? (remember, globals are bad, right???) What ends up happening is that the <strong>dependencies in your design are hidden inside the code, and not visible by examining the interfaces of your classes and methods</strong>. You have to inspect the code to understand exactly what other objects your class uses.</p>
</blockquote>
<blockquote>
<p>One of the underlying properties that makes code testable is that it is loosely coupled to its surroundings. This property allows you to substitute alternate implementations for collaborators during testing to <strong>achieve specific testing goals (think mock objects). Singletons tightly couple you to the exact type of the singleton object</strong>, removing the opportunity to use polymorphism to substitute an alternative.</p>
</blockquote>
<p>He’s absolutely right. It’s very difficult to write unit tests for code that uses singletons because it is generally tightly coupled with the singleton instance, which makes it hard to control the creation of singleton or mock it.</p>
<p>Recently I came across a project that made very liberal use of singletons. When I asked the developer about it, he said: “<em>I know singletons are bad. But I needed a single instance of objects in all these cases and had no choice but to use the singleton.</em>” Wrong. Having a single instance is a valid requirement, but <strong>singletons are not the only solution</strong>. A cleaner alternative would be to create objects in one place like the <code class="language-plaintext highlighter-rouge">main()</code> method and pass them to other classes that need them using their constructors. <strong>This concept is called <a href="https://en.wikipedia.org/wiki/Dependency_injection">‘Dependency Injection’</a></strong>.</p>
<p>To illustrate these concepts, let’s look at few example. Suppose we want to store all online users in a central registry and provide a way to find out whether a user is online or not by the username. If we use singletons, the code would look like:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SomeClassUsingSingleton</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isUserOnline</span><span class="o">(</span><span class="nc">String</span> <span class="n">username</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Obtain singleton instance directly</span>
<span class="nc">UserRegistry</span> <span class="n">userRegistry</span> <span class="o">=</span> <span class="nc">UserRegistry</span><span class="o">.</span><span class="na">getInstance</span><span class="o">();</span>
<span class="k">return</span> <span class="n">userRegistry</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">username</span><span class="o">)</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">?</span> <span class="kc">true</span> <span class="o">:</span> <span class="kc">false</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Let’s rewrite the above code to use dependency injection instead of singletons. In this example, <code class="language-plaintext highlighter-rouge">UserRegistry</code> object is created in the <code class="language-plaintext highlighter-rouge">main()</code> method and passed to the class via its constructor:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SomeClassUsingDependencyInjection</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">UserRegistry</span> <span class="n">userRegistry</span><span class="o">;</span>
<span class="c1">// Creators of the class pass an instance of UserRegistry</span>
<span class="kd">public</span> <span class="nf">SomeClass</span><span class="o">(</span><span class="nc">UserRegistry</span> <span class="n">userRegistry</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">userRegistry</span> <span class="o">=</span> <span class="n">userRegistry</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isUserOnline</span><span class="o">(</span><span class="nc">String</span> <span class="n">username</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">userRegistry</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">username</span><span class="o">)</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">?</span> <span class="kc">true</span> <span class="o">:</span> <span class="kc">false</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Problem solved. However initializing all objects in the <code class="language-plaintext highlighter-rouge">main()</code> method and passing them where they are needed is a mechanical task and increases the number of arguments that declared in constructors. Dependency injection (DI) frameworks and containers like <a href="https://github.com/google/guice">Google’s Guice</a> or <a href="http://docs.spring.io/autorepo/docs/spring/3.2.x/spring-framework-reference/html/beans.html">Spring</a> are designed to <em>automate</em> this task. <strong>You just declare <em>dependencies</em> on objects where you need them and the framework automatically provides or <em>injects</em> them</strong>. It’s that simple. Let’s rewrite the above example using Spring:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SomeClassUsingSpringDI</span> <span class="o">{</span>
<span class="nd">@Autowire</span> <span class="c1">//Will be injected automatically by Spring</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">UserRegistry</span> <span class="n">userRegistry</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">SomeClass</span><span class="o">()</span> <span class="o">{</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isUserOnline</span><span class="o">(</span><span class="nc">String</span> <span class="n">username</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">userRegistry</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">username</span><span class="o">)</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">?</span> <span class="kc">true</span> <span class="o">:</span> <span class="kc">false</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And the <code class="language-plaintext highlighter-rouge">UserRegistry</code> class is declared as a Spring component:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Component</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">UserRegistry</span> <span class="o">{</span>
<span class="o">...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We can also have multiple implementations of the <code class="language-plaintext highlighter-rouge">UserRegistry</code> such as one that uses an in-memory data structure to store users and another one that uses an external cache, and tell the DI framework to pick and inject the right one at run-time.</p>
<p><strong>To write testable code, we must separate object creation from the business logic</strong> and singletons, by their nature, prevent this by providing a global and a <code class="language-plaintext highlighter-rouge">static</code> way of creating and obtaining an instance of their classes. They make it impossible to mock and isolate methods that depend on them for testing. The <code class="language-plaintext highlighter-rouge">new</code> operator is no different and same arguments apply. Dependency injection frameworks centralize object creation and separates it from the business logic making it possible to isolate methods and write good unit tests. Projects of all sizes can benefit from DI frameworks. I normally include a DI framework form the very start because it’s tough to resist the singleton temptation half-way through the project when you really need it and don’t have time to mess around setting up DI.</p>
Effective Coding Standards2016-05-22T00:00:00+00:00https://codeahoy.com/2016/05/22/effective-coding-standards<p>Coding standards are a set of guidelines, best practices, programming styles and conventions that developers adhere to when writing source code for a project. All big software companies have them. Here are few guidelines from the <a href="https://www.kernel.org/doc/Documentation/CodingStyle">‘Linux kernel coding style’</a>:</p>
<blockquote>
<p><strong>a.</strong> Tabs are 8 characters, and thus indentations are also 8 characters.</p>
<p><strong>b.</strong> The limit on the length of lines is 80 columns and this is a strongly
preferred limit.</p>
<p><strong>c.</strong> The preferred form for allocating a zeroed array is the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>p = kcalloc(n, sizeof(...), ...);
</code></pre></div> </div>
<p>Both forms check for overflow on the allocation size n * sizeof(…),
and return NULL if that occurred.</p>
</blockquote>
<!--more-->
<p>Recently, I came across a blog post from <a href="https://twitter.com/rjrodger">Richard Rodger</a>. In ‘<a href="http://www.richardrodger.com/2012/11/03/why-i-have-given-up-on-coding-standards/#.V0FHiJMrJsM">Why I Have Given Up on Coding Standards</a>’, he writes:</p>
<blockquote>
<p>Every developer knows you should have a one, exact, coding standard in your company. Every developer also knows you have to fight to get your rules into the company standard. Every developer secretly despairs when starting a new job, <strong>afraid of the crazy coding standard some power-mad architect has dictated</strong>.</p>
<p><strong>It’s better to throw coding standards out and allow free expression</strong>. The small win you get from increased conformity does not move the needle. Coding standards are technical ass-covering.</p>
</blockquote>
<p><em>Oh boy</em>. While I <strong>disagree</strong> with Richard that coding standards should be abandoned, I share his pain. I briefly worked with a nut job of a “senior developer” who came in as the lead for a project we’d been working on for 6 months. He was an academic who had just finished his PhD and had little experience working on real world projects. He spent first couple of weeks writing “coding standards” in <em>total isolation</em> like he was some kind of a God and we were lowly beings who just weren’t good enough. His coding standards document was full of his personal opinions and promoted some insane form of coding style. The control freak <em>demanded</em> that we update the source code we had already written to reflect his standards. I have never witnessed team morale hit rock bottom so fast. Needless to say, I had a very brief stay at that job.</p>
<p>Another example that I can think of: a manager who insisted on being part of every major code review. During the reviews, he would flag formatting issues, that were almost always a matter of his own preference, as “errors”. The worst part of the story is that <em>he hadn’t written down his coding standards anywhere!</em> I guess he thought developers would learn his style through osmosis. Sometimes, it felt as if he made up rules on the fly. As I mentioned in my post on conducting <a href="http://codeahoy.com/2016/04/03/effective-code-reviews/">effective code reviews</a>, it is useless arguing over formatting issues if you don’t have coding standards.</p>
<p>The point is that coding standards are often misunderstood by naive managers and control freaks who misuse them in one way or the other such that it achieves nothing (<em>no one follows them</em>) or causes friction within the team and hurts morale. Many software developers become bitter and start hating coding standards. Coding standards aren’t the problem. Like any other tool, they become harmful when used incorrectly or in the wrong hands. Coding standards that suck have the following attributes:</p>
<ul>
<li>full of author’s own opinions and personal coding style. Coding standards are not personal agendas.</li>
<li>huge focus on style and formatting issues and is often vague.</li>
<li>recommendations disguised as standards. I have made this <a href="https://github.com/starscriber/coding-standards/wiki">mistake</a>. <strong>Standards must be treated like rules and hence must be enforceable</strong>.</li>
</ul>
<p>Good software developers and architects understand that coding style is very personal varies from individual to individual. They write coding standards that respect developers’ freedom and allow them to express themselves. They do not attempt to mechanize the whole process, rather they focus on a few well-known practices that are widely accepted or plain common sense. And before any standard is put into practice, they get buy-in from the team, if the team wasn’t already involved in formulating the standards. Here are <strong>few examples of good coding standards</strong> related to formatting:</p>
<ul>
<li>No more than one statement per line.</li>
<li>Line length should not exceed 80 or 100 characters.</li>
<li>Test class must start with the name of the class they are testing followed by ‘Test’. E.g. <code class="language-plaintext highlighter-rouge">ServerConfigurationTest</code>.</li>
<li>One character variable names should only be used in loops or for temporary variables.</li>
</ul>
<p>All of these could be easily justified in a black-and-white manner without the enforcer appearing like a dictator. On the other extreme, here are some so called “standards” that will rub developers the wrong way and prompt un-necessary debates:</p>
<ul>
<li>Class names must not end in <code class="language-plaintext highlighter-rouge">-er</code>. <em>[Personal Opinion]</em></li>
<li>Don’t use <code class="language-plaintext highlighter-rouge">static</code> fields or methods. <em>[Personal Opinion]</em></li>
<li>Aim for low Cohesion and High Coupling. <em>[Recommendation. Cannot be enforced.]</em></li>
<li>Use Test Driven Development. <em>[Recommendation. Cannot be enforced.]</em></li>
</ul>
<p>There is a grey area between common sense guidelines and personal preferences such as whether to put braces on the same line or the next. Standardize if you must, but try to keep items in the grey area (generally formatting issues) to a bare minimum.</p>
<h2 id="effective-coding-standards">Effective Coding Standards</h2>
<p>Let’s ask the question: why exactly do we need coding standards and what benefits do they offer? <a href="https://msdn.microsoft.com/en-us/library/aa291591(v=vs.71).aspx">Most</a> <a href="https://www.sitepoint.com/coding-standards/">articles</a> <a href="https://www.smashingmagazine.com/2012/10/why-coding-style-matters/">I</a> found online draw a direct relationship between coding standards and software maintainability. While there is absolutely no doubt that source code that adheres to good standards is more readable and reflects harmony, there is another side of coding standards which is often overlooked at the expense of too much attention on aesthetics. <strong>Effective coding standards focus on techniques that highlight problems and make bugs stand-out and visible to everyone</strong>. <a href="http://www.joelonsoftware.com/articles/Wrong.html">Joel said it better in 2005</a>:</p>
<blockquote>
<p>Look for <strong>coding conventions that make wrong code look wrong</strong>.</p>
</blockquote>
<p>In Java programming, having the following standards will help catch bugs early on and increase software quality:</p>
<ul>
<li>Whenever your override <code class="language-plaintext highlighter-rouge">equals()</code> method, you must also override the <code class="language-plaintext highlighter-rouge">hashCode()</code> method.</li>
<li>Do not compare strings using <code class="language-plaintext highlighter-rouge">==</code> or <code class="language-plaintext highlighter-rouge">!=</code>.</li>
<li>Do not ignore exceptions that you caught.</li>
<li>Do not catch broad exception classes like <code class="language-plaintext highlighter-rouge">Exception</code> or <code class="language-plaintext highlighter-rouge">RuntimeException</code>.</li>
</ul>
<p>To recap, effective coding standards:</p>
<ol>
<li>are short, simple and concise rules. They <em>do not attempt to cover and processify everything</em> and leave plenty of room for developers to exercise their own creativity.</li>
<li>strike the right balance between formatting issues and issues that “<em>make the wrong code look wrong</em>”.</li>
<li>are black and white instead of vague suggestions or recommendations.</li>
</ol>
<p>Coding standards, when used for the right reasons and in the right manner, offer many benefits. They make source code more readable and the software project more maintainable. They also help catch bugs and mistakes that are disguised as seemingly harmless code. While they might not catch all possible bugs, I’ll take something over nothing any day.</p>
<h2 id="automate-the-process-of-checking-code-standards">Automate the Process of Checking Code Standards</h2>
<p>Once you have effective coding standards, you should <strong>automate the process</strong> of checking source code’s adherence to standards. It will save a lot of time in peer reviews and catch everything that humans might miss. There is an abundance of style checking tools available for all major <a href="https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis">programming languages</a>. For our Java projects, we use a popular tool called <a href="http://checkstyle.sourceforge.net/">Checkstyle</a> which checks source code for style and design problems. It provides several <em><a href="http://checkstyle.sourceforge.net/checks.html">Checks</a></em> and by default, checks code against <a href="http://www.oracle.com/technetwork/java/codeconvtoc-136057.html">Sun’s conventions</a>. Or you could choose <a href="https://google.github.io/styleguide/javaguide.html">Google’s coding standards</a>, which I recommend. If you use IntelliJ IDEA (also highly recommended), there’s a <a href="https://plugins.jetbrains.com/plugin/1065">checkstyle plugin</a> that shows results right inside the IDE.</p>
<p><img src="https://codeahoy.com/img/checkstyle-intellij.png" alt="IntelliJ Checkstyle" /></p>
<p>If you are starting fresh and looking for coding standards, start by searching online for well-known standards for your programming language. Start small and remember that there is more than <em>one right way</em> to style the code. There might already be a standard available that you could borrow. For Java, I personally like <a href="http://google.github.io/styleguide/javaguide.html">Google’s coding standards</a> that I adopted with slight modifications to indentation rules. Google also have coding standards for many <a href="https://github.com/google/styleguide">other languages</a>. <strong>Your coding standards should check for both style issues and design problems</strong>. Once you have standards, make sure that they are <strong>adopted by the team and automated</strong>. However, code that deviates from standards shouldn’t be considered erroneous. It should simply be marked as out of compliance and deviations must be reviewed and fixed.</p>
<h2 id="suggested-reading">Suggested Reading</h2>
<ul>
<li><a href="https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis">List of tools for static code analysis</a></li>
<li><a href="http://paul-m-jones.com/archives/34">Why Coding Standards Matter (blog)</a></li>
<li><a href="http://findbugs.sourceforge.net/">FindBugs (Java)</a></li>
<li><a href="http://pmd.github.io/">PMD</a></li>
</ul>
7 Deadly Sins of Mobile Websites2016-05-15T00:00:00+00:00https://codeahoy.com/2016/05/15/7-deadly-sins-of-mobile-websites<p><em>A mobile website is designed and optimized for browsing on a smartphone</em>. They come in various flavours: sometimes there are different desktop and mobile versions, other times they are the same. The focus of this article is on those websites that are specifically designed for “<em>enhancing the mobile user experience</em>”, although more often than not, it has the exact opposite effect.</p>
<p><img src="https://codeahoy.com/img/7deadly/1.png" alt="BBC mobile Site" />
<em>A mobile site</em></p>
<!--more-->
<p>The sins presented in this article are so common that I’m sure readers must be able to relate to them. The most common complaints people have - or the 7 Deadly Sins of Mobile Websites - are <em>drumroll</em>:</p>
<ol>
<li>Slow to load</li>
<li>Cluttered with text (Happy talk, Me Talk)</li>
<li>Crappy navigation (Small buttons)</li>
<li>Different content/theme from the Desktop version? (<em>“Honey, where did the categories go?”</em>)</li>
<li>Popups</li>
<li>Auto Redirect</li>
<li>Advertisements and Banners</li>
</ol>
<h2 id="the-1st-deadly-sin-slow-to-load">The 1st deadly sin: Slow to Load</h2>
<p>It’s a <a href="http://blog.kissmetrics.com/loading-time/">well-known phenomenon</a> that website loading time is a crucial factor in determining the site’s success. Websites that are slow by virtue of loading high resolution images (even worse, whole carousel full of images), useless piles of Javascript crap, will <a href="http://gigaom.com/2011/07/19/consumers-losing-patience-with-the-slow-mobile-web/">lead to abandonment</a>.</p>
<p><img src="https://codeahoy.com/img/7deadly/2.png" alt="Wait... wait... wait for it..." />
<em>Wait… wait… wait for it…</em></p>
<p>I have decided not to sound like a broken record. Instead I will leave you with this rather fine point to ponder: A percentage of users browsing mobile websites are using slower connections like EDGE or 3G, not the fast Wifi we have in our homes and offices. So while a website may take 4 seconds to load on desktop, it might take longer than that to load on the mobile.</p>
<h2 id="the-2nd-deadly-sin-cluttered-with-text">The 2nd deadly sin: Cluttered with Text</h2>
<p>Websites guilty of this vice try to cram too much un-necessary text to look busy, put <a href="http://en.wikipedia.org/wiki/Happy_talk">happy talk</a> (<em>“Welcome to my sweet, lovely website”</em>) or what I like to call “me” talk (<em>“We are very happy to receive the elusive Greatness-Trophy thanks to the hard work and dedication of our CEO”</em>). The average user who arrived on Acme Widget’s website to check hours of operation don’t give a fat f*ck about their head of marketing’s epiphanies.</p>
<p><img src="https://codeahoy.com/img/7deadly/3.png" alt="Time to bring out the Magnifier" />
<em>Time to bring out the Magnifier</em></p>
<p>In real life, we like people who are articulate. They speak fluently, coherently and get their thoughts across in a concise manner. Why should websites be any different?</p>
<p><img src="https://codeahoy.com/img/7deadly/4.jpg" alt="This looks like dog's hairy behind!" />
<em>This looks like dog’s hairy behind!</em></p>
<h2 id="the-3rd-deadly-sin-crappy-navigation">The 3rd deadly sin: Crappy Navigation</h2>
<p>Hidden menus, hover objects, small buttons, small radio buttons, buttons that are too close to each other, the list goes on. Imagine, you’re travelling in a municipal bus which is jerking you around like no roller coaster would, while you are trying to <em>upvote</em> the new kitten video. But you can’t land your finger on the damn button. Every time you try, it opens the link besides that button - every link left or right seems to work and no matter how hard you try. And who do you blame for your misery: your. fat. fingers.</p>
<p><img src="https://codeahoy.com/img/7deadly/5.jpg" alt="Will I get it to land on the right link this time?" />
<em>Will I get it to land on the right link this time?</em></p>
<h2 id="the-4th-deadly-sin-different-contenttheme-from-the-desktop-version">The 4th deadly sin: Different Content/Theme from the Desktop Version</h2>
<p>Often times, the mobile website looks so different from the desktop version that I start wondering if I’m the right website. If the mobile website has a very different layout and navigation, it’s asking <strong>users to go through another learning curve</strong>. No thanks.</p>
<p><img src="https://codeahoy.com/img/7deadly/6.jpg" alt="Desktop vs Mobile version. Is this the same site? Where did the content go?" />
<em>Desktop vs Mobile version. Is this the same site? Where did the content go?</em></p>
<h2 id="the-5th-deadly-sin-popups">The 5th deadly sin: Popups</h2>
<p>Ah, the joy of having a huge box in front of my face telling me how good the App is and I should get it from the App Store. The irony: I already have the steaming pile of sh*t app installed and much rather use the website. How about I just leave instead?</p>
<p><img src="https://codeahoy.com/img/7deadly/7.jpg" alt="The content is behind there." />
<em>The content is behind there. Somewhere.</em></p>
<p><a href="http://econsultancy.com/ca/blog/62084-10-mobile-websites-that-suffer-from-chronic-pop-up-syndrome">Chris Lake makes some compelling points</a> against the Popup Syndrome, summarized in this fantastic statement:</p>
<blockquote>
<p>A link should be a promise: you click one to be taken to a specific page. That’s just how it is, and it’s what every web user expects (unless programmed to expect something different, e.g. Forbes, which I no longer visit). Websites that lead you down the garden path before fulfilling the promise only serve to disappoint users.</p>
</blockquote>
<p>Need I say more? Popups are so bad on mobile websites they should be banned altogether and erased from existence.</p>
<h2 id="the-6th-deadly-sin-auto-redirect">The 6th deadly sin: Auto Redirect</h2>
<p>At first, it sounds like a novel idea: redirect users to mobile optimized or iPad optimized website based on the type of device/browser they are using. If done right and you have a great mobile website, it works great. The problem is when the mobile website sucks, <strong>users who just want to use the desktop version instead will get agitated if they don’t get what they want</strong>. I become very annoyed whenever sites ask me to make the <em>very important decision</em> to choose between <em>“The Mobile Version”</em>, <em>“The Desktop Version”</em>, or <em>“Get the App”</em> every single time.</p>
<p>But what makes <strong>auto redirect an absolute sin is faulty redirects</strong>. As Yoshikiyo Kato writes in this <a href="http://blog.nxcgroup.com/2013/google-to-start-penalizing-mobile-redirects/">post</a>:</p>
<blockquote>
<p>A faulty redirect is when a desktop page redirects smartphone users to an irrelevant page on the smartphone-optimized website. A typical example is when all pages on the desktop site redirect smartphone users to the homepage of the smartphone-optimized site.</p>
</blockquote>
<p>For example, in the figure below, the redirects shown as red arrows are considered faulty.</p>
<p><img src="https://codeahoy.com/img/7deadly/8.png" alt="Faulty redirects" /></p>
<p>In fact, faulty redirection is so bad that there are reports suggesting that <a href="http://blog.nxcgroup.com/2013/google-to-start-penalizing-mobile-redirects/">Google penalizes websites</a> guilty of this sin in terms of their page rank.</p>
<h2 id="the-7th-deadly-sin-advertisement--banner-ads">The 7th deadly sin: Advertisement & Banner Ads</h2>
<p>How could anyone ever even think about jamming in bunch of advertisement even with that much limited screen real-estate? Some people think putting banners on websites is a wise idea. News Flash - Banner ads are trouble. This is 2016, not 1999.</p>
<p><a href="http://www.digitaltrends.com/web/banner-ads-suck-say-guys-who-invented-banner-ads/">Banner ads suck</a> say guys who invented banner ads. Nuff said.</p>
<p><img src="https://codeahoy.com/img/7deadly/9.jpg" alt="Last time we had little success with one banner. This time, let's double up the banners." />
<em>Last time we had little success with one banner. This time, let’s double up the banners.</em></p>
<p><img src="https://codeahoy.com/img/7deadly/9.jpg" alt="Overlay ad gone wrong" />
<em>Double check ads to make sure they don’t appear like these on smartphones.</em></p>
<p>Well there you have it, <em>The 7 deadly sins of mobile websites</em>. I hope you found this list useful. If you are guilty of committing these sins, don’t feel bad. As <a href="https://en.wikipedia.org/wiki/Robert_Zoellick">Robert Zoellick</a> said:</p>
<blockquote>
<p>All of us make mistakes. The key is to acknowledge them, learn, and move on. The real sin is ignoring mistakes, or worse, seeking to hide them.</p>
</blockquote>
<h3 id="if-i-missed-anything-or-a-sin-let-me-know-in-the-comments-section">If I missed anything or a sin, let me know in the comments section.</h3>
Software Estimates are not Targets2016-05-14T00:00:00+00:00https://codeahoy.com/2016/05/14/software-estimates-are-not-targets<p>Software estimation is a hard problem. So much so that in 2012, when <a href="https://twitter.com/WoodyZuill">Woody Zuill</a> tweeted his <a href="http://zuill.us/WoodyZuill/2012/12/10/no-estimate-programming-series-intro-post/">blog post</a> with the hashtag <a href="https://twitter.com/hashtag/noestimates?src=rela">#NoEstimates</a>, he set the software development community on fire. Everyone from discontented developers to seasoned software veterans flocked to the discussion on Twitter on either side of the debate. (You can read more about the #NoEstimates movement <a href="http://ronjeffries.com/xprog/articles/the-noestimates-movement/">here</a>.) In this post, let’s try to answer the question whether we need estimates and then look at what software estimates are and more importantly what they aren’t.</p>
<p><a href="https://en.wikipedia.org/wiki/Tom_DeMarco">Tom Demarco</a> gave the following tongue-in-cheek definition of a software estimate:</p>
<blockquote>
<p>“An estimate is the most optimistic prediction that has a non-zero probability of coming true.”</p>
</blockquote>
<p>Which brings me to my next point.</p>
<p><!--more--></p>
<h2 id="do-we-need-estimates">Do We Need Estimates?</h2>
<p>The short answer is <em>yes, we do</em>.</p>
<p>We need estimates for the following reasons:</p>
<ul>
<li>To know when something will be done.</li>
<li>To provide a release date.</li>
<li>To allocate cost, resources and people.</li>
<li>To make go/no-go calls: pursue the development or kill the proposal.</li>
</ul>
<p><strong>We need estimates to make decisions</strong> and <strong>estimates form the foundation of plans</strong>. Good estimates allow us to make sound decisions and to chart a course of action.</p>
<h2 id="what-is-an-estimate">What is an Estimate?</h2>
<p>Google gives the following <a href="https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=define+estimate">definition</a> of an estimate:</p>
<blockquote>
<ol>
<li>roughly calculate or judge the value, number, quantity, or extent of.</li>
<li>an approximate calculation or judgment of the value, number, quantity, or extent of something.</li>
</ol>
</blockquote>
<p>Strictly speaking, the dictionary’s definition is correct but the term has very different connotations in the software world. Estimates are frequently <strong>confused with organizational targets and plans. An organizational target is a desirable outcome that benefits the organization in one way or the other. It is an objective that the organization seeks</strong>. When you hear statements that sound similar to the following, immediately know that the speaker is talking about the organization’s targets, not estimates:</p>
<ul>
<li><em>“The goal is to have the new feature released by May 19th all users.”</em></li>
<li><em>“We must have the product ready by the end of this month for our show and tell.”</em></li>
<li><em>“We need this feature released in time for Christmas.”</em></li>
<li><em>“We must get the module updated ASAP to align with the new regulation.”</em></li>
</ul>
<p>While estimates are related to targets, the two are are entirely different beasts. <strong>A target is a goal which requires a sound plan to achieve it. In turn plans require estimates. But estimation doesn’t need to look at a target or a plan. Estimates should be completely unbiased</strong>. If it requires 6 weeks to update a module to use DynamoDB instead of Cassandra, it will take 6 weeks irrespective of the target which could be 4 weeks or 6 months. Good plans incorporate estimates to figure out how to meet the target. If the target is to have the DynamoDB upgrade complete in 4.5 weeks instead of 6 weeks, the plan might include additional resources to speed up the process.</p>
<blockquote>
<p>An estimate is NOT a target. <strong>Estimates</strong> are unbiased; <strong>Plans</strong> are heavily biased and rely on estimates to figure out how to meet the <strong>target</strong>.</p>
</blockquote>
<p>I wish I had known the difference early on in my career. Many times I got requests along the following lines:</p>
<blockquote>
<p>“<em>Umer</em>, we need this product ready in 2 months or we’ll lose the bid to the other vendor.” - Account Manager</p>
</blockquote>
<p>I used to get all worked up at what I felt were unrealistic deadlines and estimates. In hindsight, these non-technical managers were just sharing business commitments and targets. And there’s nothing wrong with it. <em>(In some cases though, I’d have appreciated if they had consulted the team before making any commitments but that’s another story.)</em> Without knowing the difference between estimates and plans, I committed to the imposed deadlines. Even if the product was done on time, there were negative side-effects. The team was overworked, the work was of low-quality and <a href="http://codeahoy.com/2016/04/27/do-not-let-technical-debt-get-out-of-control/">technical debt was accrued</a>. I accepted targets as estimates and as a substitute for a plan. In his book <a href="http://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351">Software Estimation: Demystifying the Black Art</a>, which is a definitive work on the subject, Steve McConnel provides a more productive way to deal with such requests:</p>
<blockquote>
<p>EXECUTIVE: How long do you think this project will take? We need to have this
software ready in 3 months for a trade show. I can’t give you any more team
members, so you’ll have to do the work with your current staff. Here’s a list of
the features we’ll need.</p>
<p>PROJECT LEAD: Let me make sure I understand what you’re asking for. Is it more
important for us to deliver 100% of these features, or is it more important to
have something ready for the trade show?</p>
<p>EXECUTIVE: We have to have something ready for the trade show. We’d like to
have 100% of those features if possible.</p>
<p>PROJECT LEAD: I want to be sure I follow through on your priorities as best I can.
If it turns out that we can’t deliver 100% of the features by the trade show,
should we be ready to ship what we’ve got at trade show time, or should we plan
to slip the ship date beyond the trade show?</p>
<p>EXECUTIVE: We have to have something for the trade show, so if push comes to
shove, we have to ship something, even if it isn’t 100% of what we want.</p>
<p>PROJECT LEAD: <em>OK, I’ll come up with a plan for delivering as many features as
we can in the next 3 months</em>.</p>
</blockquote>
<p>(In this example, the actual estimate to finish the project was 5 months.)</p>
<p>Steve gives the following definition of a “Good Estimate” in his book:</p>
<blockquote>
<p>A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets.</p>
</blockquote>
<p>100% agreed. The key take aways from this post are:</p>
<ul>
<li>Estimates are a crucial part of any plan, and a bad plan is better than no plan at all.</li>
<li>Estimates ≠ Targets.</li>
<li>Estimates are unbiased.</li>
<li>Plans are heavily biased and incorporate estimates to meet targets.</li>
</ul>
<p>Software estimation is tough. There are many unknowns and uncertainties. It’s like driving on the <a href="https://en.wikipedia.org/wiki/San_Francisco%E2%80%93Oakland_Bay_Bridge">Bay Bridge</a> to get to San Francisco in dense fog and broken headlights, only to receive a phone call from the hosts that the party has moved to the opposite side. Software estimation is tough, but it’s certainly not magic. This post touched upon the definition of software estimates and the relationship between estimates, plans and targets. Later posts will talk about how to get better at estimating. In the meantime, go buy a copy of <a href="http://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351">Steve’s book</a>. You will not regret it.</p>
Top new Java features in Java 8 and Beyond2016-05-10T00:00:00+00:00https://codeahoy.com/2016/05/10/5-10-java-features-many-developers-havent-heard-of<p>In this post, we’ll look at some of the latest features introduced in Java 10. Let’s go.</p>
<!--more-->
<p>Here’s a list of features covered in this post so you could skip to the one you don’t know about or skip this post entirely if you know them all :-)</p>
<ol>
<li><a href="#1">The try-with-resources Statement</a></li>
<li><a href="#2">Catch Multiple Exceptions in a Single <code class="language-plaintext highlighter-rouge">catch</code> Block</a></li>
<li><a href="#3">Underscores in Numeric Literals</a></li>
<li><a href="#4">Default Methods (in Interface)</a></li>
<li><a href="#5">Parallel Sorting of Large Arrays</a></li>
<li><a href="#6">Optional Return Values</a></li>
<li><a href="#7">Strings in <code class="language-plaintext highlighter-rouge">switch</code> statements</a></li>
<li><a href="#8">The Diamond Operator <code class="language-plaintext highlighter-rouge"><></code></a></li>
<li><a href="#9">Annotations Everywhere</a></li>
<li><a href="#10">Varargs</a></li>
</ol>
<p>A word of caution: There’s no harm in doing things the old fashioned way if it is working. It is much better than rushing to use a feature that is not fully understood. But knowledge is power.</p>
<p>Without further ado, let’s get started.</p>
<h2 id="-1-the-try-with-resources-statement"><a name="1"></a> 1. The try-with-resources Statement</h2>
<p>Prior to Java 7, working with <a href="https://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html"><code class="language-plaintext highlighter-rouge">InputStream</code></a> (and other similar APIs) produced ugly looking code. Here’s an egregious example.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">InputStream</span> <span class="n">is</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FileInputStream</span><span class="o">(</span><span class="s">"unreadme.txt"</span><span class="o">);</span>
<span class="k">try</span> <span class="o">{</span>
<span class="c1">// Read the file</span>
<span class="o">}</span> <span class="k">catch</span><span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Handle if an exception occurs while reading the file</span>
<span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">if</span><span class="o">(</span><span class="n">stream</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="n">stream</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">catch</span><span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Handle if an exception occurs while closing the file</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>There’s a lot of noise in the code above and twin <code class="language-plaintext highlighter-rouge">try-catch</code> statements. It is unnecessarily verbose, even by Java standards.</p>
<p>The <em>try-with-resources</em> statement addresses this issue. You declare any object that implements <a href="https://docs.oracle.com/javase/8/docs/api/java/io/Closeable.html"><code class="language-plaintext highlighter-rouge">java.lang.Closeable</code></a> at the start of the block. When the block exits, Java automatically closes the object by calling its <code class="language-plaintext highlighter-rouge">close()</code> method. Let’s rewrite the above example using <em>try-with-resources</em>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span><span class="o">(</span><span class="nc">FileInputStream</span> <span class="n">is</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FileInputStream</span><span class="o">(</span><span class="s">"unreadme.txt"</span><span class="o">))</span> <span class="o">{</span>
<span class="c1">// do something with the file</span>
<span class="o">}</span> <span class="k">catch</span><span class="o">(</span><span class="nc">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">//...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Much cleaner. You can even specify multiple <code class="language-plaintext highlighter-rouge">Closeable</code> objects.</p>
<h2 id="-2-catch-multiple-exceptions-in-a-single-catch-block"><a name="2"></a> 2. Catch Multiple Exceptions in a Single <code class="language-plaintext highlighter-rouge">catch</code> Block</h2>
<p>Since Java 7, multiple exceptions can be caught in a single <code class="language-plaintext highlighter-rouge">catch</code> block which makes the code less verbose and avoids duplication. Here’s an example of the multi-catch block:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">{</span>
<span class="c1">// Code that throws multiple exceptions</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IndexOutOfBoundsException</span> <span class="o">|</span> <span class="nc">IOException</span> <span class="n">ex</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"err"</span><span class="o">,</span> <span class="n">ex</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Instead of the usual:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">{</span>
<span class="c1">// Code that throws multiple exceptions</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IndexOutOfBoundsException</span> <span class="n">oobe</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"trouble traversing"</span><span class="o">,</span> <span class="n">oobe</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IOException</span> <span class="n">ioe</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"problem reading file"</span><span class="o">,</span> <span class="n">ioe</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Also, don’t do this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="o">{</span>
<span class="c1">// Code that throws multiple exceptions</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"an error occurred"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Catching <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/Exception.html"><code class="language-plaintext highlighter-rouge">Exception</code></a> is generally a bad idea. The <code class="language-plaintext highlighter-rouge">catch</code> block in the example above will catch all exceptions that are thrown in the <code class="language-plaintext highlighter-rouge">try</code> body including the ones that it cannot handle. This prevents upper methods in the stack from handling the exception properly.</p>
<p>The only catch is that the exceptions in multi-catch must be disjoint. See this <a href="http://stackoverflow.com/questions/8393004/in-a-java-7-multicatch-block-what-is-the-type-of-the-caught-exception">Stack Overflow answer</a> for more details.</p>
<h2 id="-3-underscores-in-numeric-literals"><a name="3"></a> 3. Underscores in Numeric Literals</h2>
<p>Starting from Java 7, you can use underscores in numeric literals to make your code more readable. The underscores can appear <strong>anywhere between the digits</strong>, except at the start or at the end of literal. Here are some examples taken from <a href="https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html">Oracle’s tutorial</a>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">long</span> <span class="n">creditCardNumber</span> <span class="o">=</span> <span class="mi">1234_5678_9012_3456L</span><span class="o">;</span>
<span class="kt">long</span> <span class="n">socialSecurityNumber</span> <span class="o">=</span> <span class="mi">999_99_9999L</span><span class="o">;</span>
<span class="kt">long</span> <span class="n">phoneNumber</span> <span class="o">=</span> <span class="mi">123_456_7890L</span><span class="o">;</span>
<span class="kt">long</span> <span class="n">data</span> <span class="o">=</span> <span class="mb">0b11010010_01101001_10010100_10010010</span><span class="o">;</span>
<span class="kt">byte</span> <span class="n">nybbles</span> <span class="o">=</span> <span class="mb">0b0010_0101</span><span class="o">;</span>
<span class="kt">long</span> <span class="n">maxLong</span> <span class="o">=</span> <span class="mh">0x7fff_ffff_ffff_ffff</span><span class="no">L</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">x6</span> <span class="o">=</span> <span class="mh">0x5_2</span><span class="o">;</span>
</code></pre></div></div>
<p>Although, I’m not sure if it would ever make sense to store credit cards or social security numbers in your code, <strong>using underscores certainly make reading hex and binary variables a lot more convenient.</strong></p>
<h2 id="-4-default-methods-in-interface"><a name="4"></a> 4. Default Methods (in Interface)</h2>
<p>Since Java 8, you can include <strong>method bodies <a href="https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html">to interfaces</a></strong> which wasn’t allowed in the previous versions of Java. These methods are known as the <strong>default methods</strong> because they are <strong>automatically</strong> included in classes that implement the interface. Default methods were <strong>added primarily for backwards compatibility reasons and you should <a href="https://zeroturnaround.com/rebellabs/how-your-addiction-to-java-8-default-methods-may-make-pandas-sad-and-your-teammates-angry/">use them judiciously</a></strong>.</p>
<p>This blog has a good <a href="http://zeroturnaround.com/rebellabs/java-8-explained-default-methods/">overview of the subject</a>.</p>
<h2 id="5-parallel-sorting-of-large-arrays"><a name="5"></a>5. Parallel Sorting of Large Arrays</h2>
<p>This one’s my favorite. It allows larger arrays to be <strong>sorted faster</strong>. It works by dividing a large array into several smaller subarrays and then sorting each subarray concurrently or in parallel. The results are merged back together to form the answer. So instead of,</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Array</span><span class="o">.</span><span class="na">sort</span><span class="o">(</span><span class="n">someArray</span><span class="o">);</span> <span class="c1">// Sorts arrays sequentially</span>
</code></pre></div></div>
<p>Starting with Java 8, you could use:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Arrays</span><span class="o">.</span><span class="na">parallelSort</span><span class="o">(</span><span class="n">someArray</span><span class="o">);</span> <span class="c1">// Parallel Sorting</span>
</code></pre></div></div>
<p>The analysis done for this <a href="http://www.drdobbs.com/jvm/parallel-array-operations-in-java-8/240166287">article</a> of Dr. Dobb’s got 4x better performance:</p>
<blockquote>
<p>I loaded the raw integer data from an image into an array, which ended up at <strong>46,083,360 bytes in size</strong> (this will vary depending on the image you use). The serial sort method took almost 3,000 milliseconds to sort the array on my quad-core laptop, while the parallel sort methods took a maximum of about 700 milliseconds. It’s not often that a new language release <strong>updates the performance of a class by a factor of 4x</strong>.</p>
</blockquote>
<p>4x!? I’d be very pleased with anything over 2x.</p>
<h2 id="6-optional-return-values"><a name="6"></a>6. Optional Return Values</h2>
<p>Java 8 has introduced a container object called <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html"><code class="language-plaintext highlighter-rouge">Optional</code></a> for wrapping objects that may not be present or <code class="language-plaintext highlighter-rouge">null</code>. A method can wrap its return type in <code class="language-plaintext highlighter-rouge">Optional</code>. Let’s look at an example:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="nc">Optional</span><span class="o"><</span><span class="nc">User</span><span class="o">></span> <span class="nf">getUser</span><span class="o">(</span><span class="nc">String</span> <span class="n">id</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// If the user id is NOT FOUND, return null</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And here’s how to call the method:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Optional</span><span class="o"><</span><span class="nc">User</span><span class="o">></span> <span class="n">optional</span> <span class="o">=</span> <span class="n">getUser</span><span class="o">(</span><span class="s">"jhal1"</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">optional</span><span class="o">.</span><span class="na">isPresent</span><span class="o">())</span> <span class="o">{</span>
<span class="c1">// User found. Get the value</span>
<span class="nc">User</span> <span class="n">user</span> <span class="o">=</span> <span class="n">optional</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="c1">// No user found</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In case you are wondering, what’s wrong with just returning the <code class="language-plaintext highlighter-rouge">null</code> value? Callers aren’t always aware that method may return <code class="language-plaintext highlighter-rouge">null</code> and do not always check for it. This happens frequently and is one of the reasons why the <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/NullPointerException.html"><code class="language-plaintext highlighter-rouge">NullPointerException</code></a>s are so plentiful. So, <a href="http://www.oracle.com/technetwork/articles/java/java8-optional-2175753.html">if you are tired of null pointer exceptions</a>, use <code class="language-plaintext highlighter-rouge">Optional</code>.</p>
<h2 id="7-strings-in-switch-statements"><a name="7"></a>7. Strings in <code class="language-plaintext highlighter-rouge">switch</code> statements</h2>
<p>I almost forgot that Java has a <code class="language-plaintext highlighter-rouge">switch-case</code> statement. Prior to Java 7, <code class="language-plaintext highlighter-rouge">switch-case</code> statement only worked with integer types (except <code class="language-plaintext highlighter-rouge">long</code>) and <code class="language-plaintext highlighter-rouge">enums</code>. Java 7 introduced the ability to use a <code class="language-plaintext highlighter-rouge">String</code> object as the expression. Here’s how it looks:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">String</span> <span class="n">car</span> <span class="o">=</span> <span class="n">getCar</span><span class="o">();</span>
<span class="k">switch</span><span class="o">(</span><span class="n">car</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="s">"Corvette"</span><span class="o">:</span>
<span class="c1">//Handle Corvette</span>
<span class="k">break</span><span class="o">;</span>
<span class="k">case</span> <span class="s">"AC Cobra"</span><span class="o">:</span>
<span class="c1">//Handle AC Cobra</span>
<span class="k">break</span><span class="o">;</span>
<span class="k">case</span> <span class="s">"McLaren F1"</span><span class="o">:</span>
<span class="c1">//Handle McLaren F1</span>
<span class="k">break</span><span class="o">;</span>
<span class="k">default</span><span class="o">:</span>
<span class="c1">//Handle "Car not Found" error</span>
<span class="k">break</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Which is equivalent to the more cluttered:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">(</span><span class="n">car</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"Corvette"</span><span class="o">))</span> <span class="o">{</span>
<span class="c1">//Handle Corvette</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="n">car</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"AC Cobra"</span><span class="o">))</span> <span class="o">{</span>
<span class="c1">//Handle AC Cobra</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="n">car</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"McLaren F1"</span><span class="o">))</span> <span class="o">{</span>
<span class="c1">//Handle McLaren F1</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="c1">//Handle "Car not Found" error</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="8-the-diamond-operator-"><a name="8"></a>8. The Diamond Operator <code class="language-plaintext highlighter-rouge"><></code></h2>
<p>Diamond operators were introduced to make the use of generics a little less verbose. Take a look at this example:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Map</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">>></span> <span class="n">aMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">>>();</span>
</code></pre></div></div>
<p>The parameter types are duplicated on the left and right sides of the expression. Since Java 7, you can omit the type definitions on the right side of assignment expressions with a diamond operator, <code class="language-plaintext highlighter-rouge"><></code>. The above statement could be rewritten as:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Map</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">List</span><span class="o"><</span><span class="nc">String</span><span class="o">>></span> <span class="n">aMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashMap</span><span class="o"><>();</span>
</code></pre></div></div>
<p>When the compiler encounters the diamond operator (<>), it <strong>infers the generic type arguments from the context</strong>.</p>
<h2 id="9-annotations-everywhere"><a name="9"></a>9. Annotations Everywhere</h2>
<p>Thanks to Java 8, annotations can be retrofitted <a href="http://docs.oracle.com/javase/tutorial/java/annotations/basics.html">almost anywhere</a> in your code. Great, because that’s <a href="http://www.annotatiomania.com/">just what we needed</a>. I’m fine with annotations, but I’ve seen some developers going overboard, trying to do <strong>too much magic</strong> with annotations. Too much of annotations, just like too much of anything, are bad. You’re probably better off not knowing that this feature even exists. End of my rant.</p>
<h2 id="10-varargs"><a name="10"></a>10. Varargs</h2>
<p><a href="http://docs.oracle.com/javase/1.5.0/docs/guide/language/varargs.html">Varargs</a> are useful for passing an arbitrary number of parameters to a method. Such as <a href="http://docs.oracle.com/javase/8/docs/api/java/lang/String.html#format-java.lang.String-java.lang.Object...-"><code class="language-plaintext highlighter-rouge">String.format(String format, Object...args)</code></a>. Joshua Bloch in <a href="http://www.amazon.com/Effective-Java-2nd-Joshua-Bloch/dp/0321356683">Effective Java</a> recommends using varargs judiciously:</p>
<blockquote>
<p>varargs are effective in circumstances where you really do want a method with a variable number of arguments. Varargs were designed for printf, which was added to the platform in release 1.5, and for the core reflection facility (Item 53), which was retrofitted to take advantage of varargs in that release. Both printf and reflection benefit enormously from varargs. You can retrofit an existing method that takes an array as its final parameter to take a varargs parameter instead with no effect on existing clients. <strong>But just because you can doesn’t mean that you should!</strong></p>
</blockquote>
<p>That’s sound advice. Just because there’s a feature available, doesn’t mean you have to use it.</p>
The char Type in Java is Broken?2016-05-08T00:00:00+00:00https://codeahoy.com/2016/05/08/the-char-type-in-java-is-broken<p>If I may be so brash, it is my opinion that the <code class="language-plaintext highlighter-rouge">char</code> type in Java is dangerous and should be avoided if you are going to use Unicode characters. <code class="language-plaintext highlighter-rouge">char</code> is used for representing characters (e.g. ‘a’, ‘b’, ‘c’) and has been supported in Java since it was released about 20 years ago. When Java first came out, the world was a simpler place. Windows 95 was the latest, greatest operating system, world’s <a href="https://en.wikipedia.org/wiki/Motorola_StarTAC">first flip phone</a> was just put on sale, and Unicode had less than <em>40,000</em> characters, all of which fit perfectly into the 16-bit space that <code class="language-plaintext highlighter-rouge">char</code> provides. But things have changed drastically. Unicode has outgrown the 16-bit space and now requires 21 bits for all of its <em>120,737</em> characters.</p>
<!--more-->
<p>Java has supported Unicode since its first release and <strong>strings are internally represented using <a href="https://en.wikipedia.org/wiki/UTF-16">UTF-16</a> encoding</strong>. UTF-16 is a <em>variable length</em> encoding scheme. For characters that can fit into the 16 bits space, it uses 2 bytes to represent them. For all other characters, it uses 4 bytes. This is great. All possible Unicode characters in existence plus a lot more (1 million more) could be represented using UTF-16 and thus as Strings in Java.</p>
<p>But <code class="language-plaintext highlighter-rouge">char</code> is a different story altogether. Let’s look at its definition from the <a href="https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html">official source</a>:</p>
<blockquote>
<p>char: The char data type is a single <strong>16-bit Unicode character</strong>. It has a minimum value of ‘\u0000’ (or 0) and a maximum value of ‘\uffff’ (or 65,535 inclusive).</p>
</blockquote>
<p><em>“16-bit Unicode character”?</em> I guess <a href="http://www.joelonsoftware.com/articles/Unicode.html" rel="nofollow">Joel</a> was right:</p>
<blockquote>
<p><strong>Some people are under the misconception that Unicode is simply a 16-bit code where each character takes 16 bits and therefore there are 65,536 possible characters. This is not, actually, correct</strong>. It is the single most common myth about Unicode, so if you thought that, don’t feel bad.</p>
</blockquote>
<p>There is no such thing as “16-bit Unicode character”. Please read <a href="http://www.joelonsoftware.com/articles/Unicode.html" rel="nofollow">Joel’s article</a> if you don’t understand the last statement.</p>
<p><code class="language-plaintext highlighter-rouge">char</code> uses 16 bits to store Unicode characters that fall in the 0 - 65,535 which isn’t enough to store all Unicode characters anymore. You might think: <em>Gee, 65,535 is plenty already. I’ll never use that many</em>. That’s true. But your users will. And when they send you a character that requires more than 16 bits, like these emojis 👦👩, <strong>the <code class="language-plaintext highlighter-rouge">char</code> methods like <code class="language-plaintext highlighter-rouge">someString.charAt(0)</code> or <code class="language-plaintext highlighter-rouge">someString.substring(0,1)</code> will break and give you only half the code point. And the worst part is that the compiler won’t even complain</strong>. Recently, a fellow developer told me that their “North American users” started complaining that the chat nicknames and messages “aren’t displaying properly”. After a lot of grief, they found the issue and had to undo all <code class="language-plaintext highlighter-rouge">char</code> manipulation in their software to handle emojis and other cool characters. (Use <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/String.html#codePointAt(int)"><code class="language-plaintext highlighter-rouge">codePointAt(index)</code></a> instead which returns an int that will fit all Unicode characters in existence.)</p>
<p>I have heard people say things like: <em>“if internationalization isn’t a concern, you’d probably be fine using <code class="language-plaintext highlighter-rouge">char</code>”</em> or <em>“don’t worry about it unless your program is going to be released in China or Japan”</em>.</p>
<p>First, I rarely come across applications where internationalization isn’t a concern anymore. My last three jobs all required internationalization at their core. Second, <strong>emojis characters are supported by all popular applications these days</strong>. Unicode isn’t just about internationalization anymore.</p>
<p>To be fair to <code class="language-plaintext highlighter-rouge">char</code>, it will work fine most of the time for many applications. It isn’t broken but it has a flaw which could ‘break’ your application silently and make your users see garbled text. May be, a UTF-16 character type from Oracle is the answer. Or at least a runtime exception when compiler detects that something bad is about to happen in the interim. Until then, we should probably avoid the <code class="language-plaintext highlighter-rouge">char</code> type.</p>
<p>Even its official <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/Character.html">JavaDocs</a> don’t sound all that convincing to me:</p>
<blockquote>
<p>The char data type (and therefore the value that a Character object encapsulates) are based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. <strong>The Unicode Standard has since been changed to allow for characters whose representation requires more than 16 bits</strong>. The range of legal code points is now U+0000 to U+10FFFF, known as Unicode scalar value. (Refer to the definition of the U+n notation in the Unicode Standard.)</p>
</blockquote>
<p>🤷</p>
Minimum Viable Product - Lessons for Software Teams2016-05-07T00:00:00+00:00https://codeahoy.com/2016/05/07/minimum-viable-product-lessons-for-software-teams<p>The concept of the minimum viable product or the MVP was popularized by Eric Reis in his book <a href="http://www.amazon.com/Lean-Startup-Entrepreneurs-Continuous-Innovation/dp/0307887898">The Lean Startup</a>. He defines it as:</p>
<blockquote>
<p>The minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.</p>
</blockquote>
<p>Startups fail for many reasons but one of the <strong>biggest reasons is that they build products that their customers do not want</strong>. By that time, it’s too late. They have already spent months or even years of their lives building a great product - a grand vision - that the market isn’t, unfortunately, willing to purchase. MVP is a strategy to avoid exactly that scenario: <strong>building something that the customers do not want</strong>.</p>
<!--more-->
<p>Eric <a href="http://www.inc.com/lee-clifford-julie-schlosser/lean-startup-eric-ries-testing-your-product.html">cites</a> the story of Zappos. They didn’t start out with a grand vision of building a cool website, distribution and call centers. Nope. <strong>Zappos started by testing a simple hypothesis: are customers willing to buy shoes online?</strong></p>
<blockquote>
<p><strong>They went to local shoe store, took pictures of each of their products and put them online. If anyone bought shoes from them [at this early stage], they planned to go to the store, buy the shoes and mail them to the customer</strong>. There was no big business behind it; there was a website and a hope that they’ll get so many orders that it will get annoying to do all the purchasing and shipping manually. <strong>It was all to test their big idea</strong>.</p>
</blockquote>
<p>MVP is a great way to test the actual usage and assumptions as opposed to conventional market research that includes researching online, surveys etc. which often provide misleading results.</p>
<p><strong>Software teams can learn a lot from MVP and apply it to build products that meet their client needs</strong>. When I first heard of MVP, I thought of it as a <strong>rebranding of ‘Continuous Delivery (CD)’</strong> which, along with the practices of incremental and iterative development, has been around for a long time. But there is a huge difference and it lies in how software developers understand and perceive these concepts. <a href="http://istqbexamcertification.com/what-is-incremental-model-advantages-disadvantages-and-when-to-use-it/">This article</a> captures the definition I hear from most developers when I ask them about incremental development:</p>
<blockquote>
<p>In incremental model the whole requirement is divided into various builds … more easily managed modules. Each module passes through the requirements design, implementation and testing phases. A working version of software is produced during the first module, <strong>so you have working software early on during the software life cycle</strong>. Each subsequent release of the module adds function to the previous release. The process continues till the complete system is achieved.</p>
</blockquote>
<p>The article even has an illustration showing how the Mona Lisa would be built incrementally or piece-by-piece:</p>
<p><img src="https://codeahoy.com/img/Incremental-mona-lisa.jpg" alt="Incremental Mona Lisa.jpg" /></p>
<p>The problem is that <em>“working software early on in the life cycle”</em> doesn’t happen in real-life. Modules take up a lot of time; developers start arguing about technologies and methodologies, various architectures and database systems are evaluated and the complete painting takes up a long time. When it gets released, the customer turns around and demands that Mona Lisa’s dress should be of red color instead of deep forest green!</p>
<p>Traditional methodologies guide the software development process to ensure that the <strong>product gets built right</strong>, but the MVP answers the bigger question: <strong>are we building the right product?</strong> While Agile does promote <em>evolutionary development</em> where results are <a href="https://en.wikipedia.org/wiki/Agile_software_development">demonstrated to stakeholders after each iteration</a>, a MVP isn’t just show and tell. It is a real product that gets released to the users. Software teams could build a MVP using Agile/Scrum, <em>incrementally</em> and deliver it <em>continuously</em>.</p>
<p>Because a MVP gets released to the customers, it requires support from the organization, clients and other departments. Building a MVP isn’t easy. On the one hand, you want to limit it to few essential or key features, but on the other hand, you want to deliver something that your customers will find useful so they can start using it. You must find the right balance.</p>
<p>Releasing the product only when it is almost or absolutely ready is a mistake I have seen far too often. Usual software development goes like this: software teams talk to their clients and stakeholders to get their requirements (<em>Everyone does that. Right?</em>). They create a product vision (or write functional specifications) and show it to the stakeholders. Once the stakeholders agree on the requirements and the product vision, the team set out to build the product. The product is divided into several layers or several subs-systems, if following SOA or microservices architecture. Each layer is built piece-by-piece, incrementally and iteratively. Even the CI/CD pipeline is there since week one. Finally, when everything is ready, the product is ‘released’ to the customer. Then the customer <strong>flips and ask for changes or new features</strong>. It happens almost always. Depending on how well the team captured and modeled their requirements, the rework could take weeks or months!</p>
<p>I made this mistake when I architected a telecommunications project using Service Oriented Architecture (SOA). While the team followed iterative approach to individual services and did continuos integration, <strong>we didn’t implement the front-end that our customer could use and provide feedback on early on</strong>. We used Agile and it worked great for the database layer, the business layer, the sniffers, the messaging layers and the load balancers. When the product was complete, we released it to our big client for a pilot with few thousand users. That same week, I received a phone call from the CEO asking for very different functionality, in an apologetic tone. It turned out that <strong>once the client used the product, they changed their mind</strong>. The big release date was one or two months away and we had to do major rework to adopt the change.</p>
<p>What can we learn from this story? Build the MVP. While I’m not against refactoring and adopting to change, in this case, we could have avoided extra effort if we had built a MVP. We didn’t built in a vacuum. We prototyped but <strong>prototypes only help so much</strong>. In the example I gave, the product’s goal was to support mobile menus that allowed people to send messages and transfer funds. We prototyped and mocked the menus in <a href="https://balsamiq.com/">Balsamiq</a> and even went as far as creating a simple HTML page that allowed the client to interact with the menu options. At regular intervals, we’d send client videos of partial functionality in action. But <strong>prototypes and videos aren’t remotely as interesting as the actual product. Clients looked at those and said everything looks great. But once they got the actual product on their phones, they got creative</strong> and came up with new ideas. If I were doing the project all over again, the first thing I’d do is build the front-end (menus), use a simple database (even SQLite instead of Cassandra) and make the product work on real devices and give it to the client. Plumbing to make it work across multiple sites and NoSQL databases would come later.</p>
<p>A little while ago, I was working on a backend with REST based microservices architecture. There were several clients with different needs and there was a lot of unknown. A common complaint from the developers who worked on earlier projects was that <strong>the clients always change their minds after the release which makes their products messy since they accrue <a href="http://codeahoy.com/2016/04/27/do-not-let-technical-debt-get-out-of-control/">technical debt</a></strong>.
I had a Déjà vu. So we figured out the most urgent needs and defined a milestone that would deliver a working product. It’ll have fewer services than our grand vision, but everything will work end-to-end. It won’t be perfect or complete, but it will have just the right functionality they need to get up and running and provide us the feedback. And we will grow and evolve it over time.</p>
<p>The closest reference to MVP I could find in software development is a concept called ‘Tracer Bullets’ that was first used by Andy and Dave in their book the <a href="http://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">The Pragmatic Programmer</a>:</p>
<blockquote>
<p>We once undertook a complex client-server database marketing project … The servers were a
range of relational and specialized databases. The client GUI, written in Object Pascal,
used a set of C libraries to provide an interface to the servers … <strong>There were many unknowns and many different environments</strong>, and no one was too sure how the GUI should behave.</p>
<p>This was a great opportunity to use tracer code. We developed the framework for the front
end, libraries for representing the queries, and a structure for converting a stored query into
a database-specific query. Then we put it all together and checked that it worked. For that
initial build, all we could do was submit a query that listed all the rows in a table, but it
proved that the UI could talk to the libraries, the libraries could serialize and unserialize a
query, and the server could generate SQL from the result. <strong>Over the following months we
gradually fleshed out this basic structure, adding new functionality by augmenting each
component of the tracer code in parallel</strong>.</p>
</blockquote>
<p>So when you are setting out on a journey to build a new product and have lots of unknowns and assumptions, build a MVP - or use Tracer Bullets. Make sure to <strong>build the right thing</strong>. You might have seen this image a thousand times already, but this is exactly what MVP will avoid:</p>
<p><img src="https://codeahoy.com/img/software-requirements.png" alt="Software Requirements" /></p>
<h2 id="notes">Notes</h2>
<p>John Mayo-Smith <a href="http://www.informationweek.com/two-ways-to-build-a-pyramid/d/d-id/1012280?">illustrates</a> two approaches to building a pyramid:</p>
<ol>
<li>
<p>Build it gayer by layer
<img src="https://codeahoy.com/img/flat_pyramid.gif" alt="Flat Pyramid" /></p>
</li>
<li>
<p>Start with a smaller pyramid and keep growing it.
Just like MVP. Delivering a “smaller product” that captures essential features that the customer could actually use and deliver feedback on.</p>
</li>
</ol>
<p><img src="https://codeahoy.com/img/growing_pyramid.gif" alt="Growing Pyramid" /></p>
Good Abstractions Have Fewer Leaks2016-05-06T00:00:00+00:00https://codeahoy.com/2016/05/06/good-abstractions-have-fewer-leaks<blockquote>
<p>Abstraction is one of the greatest visionary tools ever invented by human beings to imagine, decipher, and depict the world. - Jerry Saltz</p>
</blockquote>
<p>Abstraction are all around us. We abstract things to hide details making it easier to see the “big picture” and help cope with the complexity. When I push down on the gas pedal, a lot of magic happens under the hood to move my car. But I don’t have to know any of that. I only ever need to know that pushing on the pedal increases the speed and releasing it will decrease it. The gas pedal is a <strong>simple interface and that’s what makes it so great</strong>. It has been with us for a very long time and it’s here to <a href="https://forums.teslamotors.com/forum/forums/what-term-should-we-use-gas-pedal">stay</a>.</p>
<!--more-->
<p>In software, like any other engineering discipline, abstractions are everywhere: the protocols, the frameworks, the libraries, the game engines, the file systems, even the programming languages we use everyday are abstractions of <a href="https://en.wikipedia.org/wiki/Low-level_programming_language">low-level languages</a>. It can be argued that everything in computer science is inevitably an abstraction of something more complicated under the hood. The higher-level abstractions provide useful functionality in the form of a black-box, <strong>hiding all the complexity and implementation details</strong>.</p>
<p>Except that it’s not always so black and white. <strong>Software abstractions are never perfect</strong> and, arguably, are far from perfect. Abstractions are deficient when you have to <strong>understand the low-level details or peek under the covers to understand what’s going</strong>. Consider Java’s garbage collection process. It would have been great if I never had to think about how it works - after all, it was supposed to work like magic. But in reality, when a bug was reported that <strong>“requests are taking a lot longer to process”</strong>, I had to go in and figure out that the garbage collection was slowing things down. One cannot optimize performance and avoid penalties without learning how the garbage collector works (at least not for high-throughput, low-latency applications). Joel Spolsky called this the <a href="http://www.joelonsoftware.com/articles/LeakyAbstractions.html">law of leaky abstractions</a>:</p>
<blockquote>
<p>All non-trivial abstractions, to some degree, are leaky.</p>
<p>Abstractions fail. Sometimes a little, sometimes a lot. There’s leakage. Things go wrong. It happens all over the place when you have abstractions.</p>
</blockquote>
<p><img src="https://codeahoy.com/img/leaky_abstraction.jpg" alt="Leaky Pipe" /></p>
<p>Joel is right. All abstractions in software are leaky: they attempt to hide away the complexity but the underlying details are not complete hidden. However, <strong>the truth is that we can’t live without abstractions.</strong> Without abstractions and modules, I can’t even begin to comprehend how any large software project could be maintained. Without Java’s garbage collection, my code will be cluttered with <code class="language-plaintext highlighter-rouge">delete</code> statements and memory leaks will be all over the place. The flexibility offered by the the garbage collector works 90% of the time. It leaks because the underlying function of automatic memory management is very complex and highly irregular to which there isn’t a general purpose solution. To its credit, garbage collection provides ways to tweak performance depending on the specific needs.</p>
<p>I don’t think Joel meant that we should abandon abstractions altogether. I guess the point was to embrace the fact that all abstractions are leaky. <strong>Edge cases will often require developers to get their hands dirty by understanding the framework to optimize performance or to troubleshoot. And that’s perfectly fine.</strong> Good abstractions and frameworks reduce the need for someone to understand the inner workings. Bad abstractions leak a lot and expose their users to all the details. When I heard of Remote Procedure Call (RPC), I loved the concept. It’d allow me to call a method on a remote server as easily as I would a local method. Great. There was a learning curve with <a href="https://thrift.apache.org/">Apache Thrift</a> and it was well worth it. The framework, <a href="http://www.valuedlessons.com/2008/06/my-experience-with-message-passing.html">albeit leaky</a>, saved me from writing my own client/server and dealing with the communication complexity (although it was later replaced with REST). Similarly, compared to C++ string library which Joel cites as being leaky, strings in Java are lot less leaky (probably because they are a native feature of the language.)</p>
<p>Leaky abstraction aside, over the years, I have seen both good and terrible abstractions masqueraded as APIs. Not in distant memory, I had to fix an “abstraction” <strong>that failed to hide the functionality</strong> it provided and had very tight coupling with the applications that used it. 30% of the module’s logic was scattered outside of it, weaved throughout other applications using java annotations. Another time I was reviewing some code and came across some very complex logic. It turned out that the developer that created an “abstraction” over Hibernate to provide an even more general and completely useless solution. Hibernate is already an abstraction over SQL! <strong>Adding another layer on top of Hibernate made things even more complex than they needed to be</strong>. As <a href="https://en.wikipedia.org/wiki/David_Wheeler_(British_computer_scientist)">David J. Wheeler</a> is quoted:</p>
<blockquote>
<p>All problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection.</p>
</blockquote>
<p>Abstraction are all around us and <strong>without abstractions we’d be doomed.</strong> After all, <a href="https://en.wikiquote.org/wiki/Carl_Sagan">if you wish to make an apple pie from scratch, you’d first have to invent the universe</a>. Abstractions provide us the flexibility to work with something very complicated. Imagine the complexity of dealing with blobs of bits and bytes on magnetic platters, or on semiconductor chips, instead of files and databases! Understanding what goes on under the hood from time to time is a good trade-off considering that most of the time, the flexibility offered by a good abstraction would be sufficient.</p>
<p>As software developers, our goal should be to build less leaky abstractions. I also mentioned a few examples of bad and over abstractions that complicate things even more and should be avoided at all costs. In the next post or so, I’ll talk about modular software and how abstractions help software development and when they become a pain.</p>
Software Rot, Entropy and the Broken Window Theory2016-05-02T00:00:00+00:00https://codeahoy.com/2016/05/02/software-rot-entropy-and-the-broken-window-theory<p><img src="https://codeahoy.com/img/broken_windows.jpg" alt="Broken Windows" /></p>
<blockquote>
<p>“Complexity is the business we are in and complexity is what limits us.” - Fred Brooks, The Mythical Man-Month</p>
</blockquote>
<p>Software projects go through many modifications over their lifetime. As they evolve, the code grows in size and complexity creeps in. Software developers spend a large portion of their time <strong>maintaining</strong> existing software either by adding new functionality or fixing bugs. Often times, they are forced to take <strong>shortcuts</strong> to meet deadlines. Developers add new functionality in a ‘quick and dirty’ manner and apply duct-tape to defects. While the organization meets its short-term goal of getting the software out of the door quickly, the code quality suffers and deteriorates. After a while, things start to get really bad. The software becomes so complex and buggy, that it is virtually impossible to maintain. Fixing a bug would introduces more bugs and modifying one part of the software would break several others.</p>
<!--more-->
<p>Let’s look at a related concept called <strong>software entropy</strong>. Entropy is the amount of <strong>disorder</strong> in a system. It is a physical phenomenon but <a href="http://www.amazon.com/Object-Oriented-Software-Engineering-Approach/dp/0201544350">Ivar Jacobson et al</a> used it to describe the disorder in a software system:</p>
<blockquote>
<p>The second law of thermodynamics, in principle, states that a closed system’s disorder cannot be reduced, it can only remain unchanged or increased. A measure of this disorder is entropy. This law also seems plausible for software systems; <strong>as a system is modified, its disorder, or [software] entropy, always increases</strong>. This is called Software Entropy.</p>
</blockquote>
<p>When the ‘disorder’ or the <em>software entropy</em> increases, it leads to <strong><a href="https://en.wikipedia.org/wiki/Software_rot">software or code rot</a></strong>. The system ends up becoming so complex and disorganized that it is too costly or impossible to maintain. People get frustrated and consider major refactoring or, in some cases, <a href="http://codeahoy.com/2016/04/21/when-to-rewrite-from-scratch-autopsy-of-a-failed-software/">rewriting from scratch</a>. These arduous solutions fix the problem in the short-term but the software will <strong>rot again if the team doesn’t adopt a plan for keeping future complexity under control</strong>.</p>
<p>While there are many factors that lead to software rot, the most important ones, according to <a href="https://en.wikipedia.org/wiki/Andy_Hunt_(author)">Andrew Hunt</a> and <a href="https://en.wikipedia.org/wiki/Dave_Thomas_(programmer)">Dave Thomas</a>, are the <strong>psychology and the team culture</strong>. In their book, <a href="http://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">The Pragmatic Programmer</a>, they argue that <strong>software entropy is contagious and if not controlled, becomes an epidemic</strong>.</p>
<blockquote>
<p>In inner cities, some buildings are beautiful and clean, while others are rotting hulks. Why? Researchers in the field of crime and urban decay discovered a fascinating trigger mechanism, one that very quickly turns a clean, intact, inhabited building into a smashed and abandoned derelict .</p>
<p>A broken window.</p>
<p><strong>One broken window, left unrepaired for any substantial length of time, instills in the inhabitants of the building a sense of abandonment — a sense that the powers that be don’t care about the building. So another window gets broken</strong>. People start littering. Graffiti appears. Serious structural damage begins. In a relatively short space of time, the building becomes damaged beyond the owner’s desire to fix it, and the sense of abandonment becomes reality.</p>
</blockquote>
<p>Broken window theory was proposed by criminologists <a href="https://en.wikipedia.org/wiki/James_Q._Wilson">James Wilson</a> and <a href="https://en.wikipedia.org/wiki/George_L._Kelling">George Kelling</a> and had an enormous impact on police policy throughout the 1990s. While the broken window theory was been <a href="http://www.smithsonianmag.com/smart-news/sorry-malcolm-gladwell-nycs-drop-in-crime-not-due-to-broken-window-theory-12636297/?no-ist">widely</a> <a href="http://chronicle.uchicago.edu/060330/brokenwindow.shtml">criticized</a>, I think it makes sense. When shortcuts are taken poor design decisions are made, it sends a signal that no one cares or that no one is in charge. Since its a feature of the environment, even good software developers <em>might</em> fall for it. The solution according to Andrew and Dave is simple:</p>
<blockquote>
<p><strong>Don’t leave “broken windows”</strong> (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered. If there is insufficient time to fix it properly, then board it up. <strong>Perhaps you can comment out the offending code, or display a “Not Implemented” message, or substitute dummy data instead. Take some action to prevent further damage and to show that you’re on top of the situation</strong>.</p>
</blockquote>
<p>A simple comment around ugly code stating that its ugly and needs to get fixed soon is better than nothing. Code can quickly rot once windows start breaking. Even a mere perception of disorder could result in total chaos. Don’t leave broken windows, fix them as soon as you could. Entropy will creep in - don’t let it win.</p>
Do Experienced Programmers Use Google Frequently?2016-04-30T00:00:00+00:00https://codeahoy.com/2016/04/30/do-experienced-programmers-use-google-frequently<p>Software developers, especially those who are new to the field, often <a href="http://two-wrongs.com/how-much-does-an-experienced-programmer-use-google">ask</a> this question or at least <a href="http://www.hanselman.com/blog/AmIReallyADeveloperOrJustAGoodGoogler.aspx">wonder</a> whether they are good developers or just good at googling up solutions.</p>
<h3 id="do-experienced-programmers-use-google-frequently">“Do experienced programmers use Google frequently?”</h3>
<p>The resounding answer is <strong>YES, experienced (and good) programmers use Google… a lot</strong>. In fact, one might argue they <strong>use it more than the beginners</strong>. Using Google doesn’t make them bad programmers or imply that they cannot code without Google. In fact, truth is quite the opposite: Google is an essential part of their software development toolkit and they know when and how to use it.</p>
<!--more-->
<p>A big reason to use Google is that it is hard to remember all those minor details and nuances especially when you are programming in multiple languages and using dozens of frameworks. As Einstein said:</p>
<blockquote>
<p>“Never memorize something that you can look up.” - Albert Einstein</p>
</blockquote>
<h5>
<a href="https://twitter.com/intent/tweet?text=HNever+memorize+something+that+you+can+look+up.%E2%80%9D+-+Albert+Einstein+%0D%0Ahttps%3A%2F%2Fbit.ly%2F2wt3vsU+" data-size="large">
==> Tweet This Quote <== </a>
</h5>
<p>Aside from that, good programmers also know that they cannot be the first one to have encountered a problem. They use Google to <strong>research</strong> possible solutions, carefully evaluating the results and consciously separating the wheat from the chaff; they don’t <strong>blindly follow or copy-paste</strong> any solution they come across. Expert programmers are also paranoid, living in self-doubt and questioning their competence. Whenever their spidey senses start tingling, they know they may be going the wrong hole; they rely on Google on validate their logic.</p>
<p>Going by the definition, I would be considered an experienced programmer. Recently, I had to write web server using <a href="http://netty.io/">Netty</a> in Java to handle persistent sockets from mobile games. I had never used Netty before. Here are my Google searches I did:</p>
<p class="message">
<strong>1.</strong> netty tutorial
<strong>2.</strong> netty maven dependency
<strong>3.</strong> netty bytebuf to string
<strong>4.</strong> netty bytebuf release
<strong>5.</strong> netty 4 changes
<strong>6.</strong> setOption(“child.bufferFactory”) netty 4<br />
<strong>7.</strong> ByteBuf netty
<strong>8.</strong> opensource projects using netty framework<br />
<strong>9.</strong> netty 4 examples
<strong>10.</strong> netty 4 adding json encoder
<strong>11.</strong> netty channel pipeline
<strong>12.</strong> netty 4 messagetomessage encoder
<strong>13.</strong> netty serverbootstrap childhandler
<strong>14.</strong> ByteBuf netty
<strong>15.</strong> lengthfieldbasedframedecoder netty 4
<strong>16.</strong> netty 4 client examples
<strong>17.</strong> netty 4 bytebuf to bytebuffer
<strong>18.</strong> netty 4 endianness
<strong>19.</strong> netty channelhandlercontext
<strong>20.</strong> netty channelhandlercontext thread safe
<strong>21.</strong> netty user authentication
<strong>22.</strong> netty heartbeat handling
<strong>23.</strong> load test netty with 10k concurrent sockets
</p>
<p>I wrote 255 lines of code that included a working server and a client. I queried google 23 times mostly landing on StackOverflow, Netty 4 website, GitHub, and JavaDocs. If you do the math, that averages out to <strong>1 query every 10 lines of code</strong>! I had no idea. Let me know in the <strong>comments what your average is</strong>.</p>
<p>So sit back, relax and remember that <strong>Google is software developer’s best friend</strong>.</p>
<p><em>How often do you use Google when programming? Do you have any Google power tips that you want to share with others? Just leave a comment below.</em></p>
Do Not Let Technical Debt Get Out of Control2016-04-27T00:00:00+00:00https://codeahoy.com/2016/04/27/do-not-let-technical-debt-get-out-of-control<p>Technical debt is a useful metaphor for describing the consequences of adding new functionality to a system in a <strong>quick and dirty</strong> manner to get something out of the door faster. The proper way would have resulted in a much cleaner design and implementation, but would also have taken much longer. <a href="http://martinfowler.com/bliki/TechnicalDebt.html">Martin Fowler</a> calls technical debt a wonderful metaphor:</p>
<blockquote>
<p>Technical Debt is a <strong>wonderful metaphor</strong> developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the <strong>extra effort that we have to do in future development because of the quick and dirty design choice</strong>.</p>
</blockquote>
<p>Taking on technical debt should be a <strong>strategic decision</strong> where all stakeholders must understand the consequences and risks involved. Like most financial debts, it should not be taken recklessly and interest <strong>payments must be paid on time to avoid penalties</strong>.</p>
<p><!--more--></p>
<p>While technical debt has negative connotations, it is an <strong>unavoidable reality</strong> for many software projects. In her <a href="http://www.amazon.com/Practical-Object-Oriented-Design-Ruby-Addison-Wesley/dp/0321721330">book</a> on Practical Object-oriented Design in Ruby: An Agile Primer, <a href="http://www.sandimetz.com/">Sandi Metz</a> wrote:</p>
<blockquote>
<p>Sometimes the value of having the feature right now is so great that it outweighs any future increase in costs. If lack of a feature will force you out of business today it doesn’t matter how much it will cost to deal with the code tomorrow; you must do the best you can in the time you have.</p>
</blockquote>
<p>At Starscriber, we accrued technical debt from time to time to take advantage of new business opportunities and tried to pay it off as soon as the dust settled. But we didn’t always succeed. There were at least two projects where the debt got out of control. Implementing (or hacking, would be a better word) new features was a complex and <strong>painful process</strong> for everyone involved: developers, testers and operations teams. The change requests didn’t stopped coming and we made the mistake of <strong>not acknowledging that we are accumulating way too much technical debt and letting it drag on for way too long</strong>. The result? It became a huge burden and required a lot of effort just to keep the system running.</p>
<p>However, in my opinion the biggest casualty wasn’t the productivity; it was the team culture and the morale. <strong>People got demotivated since they took no pride in their work</strong>. They couldn’t: no creativity or learning was involved, other than finding creative ways to ‘hack’ and make it work. And since the project required <strong>tribal knowledge</strong> to understand all the hacks, we had to keep the ‘demotivated’ team together until we could take it no more and had to <strong>halt new development to do major refactoring and testing</strong>. Some teams consider <a href="http://codeahoy.com/2016/04/21/when-to-rewrite-from-scratch-autopsy-of-a-failed-software/">rewrite from scratch when facing this predicament, which is almost always a big mistake</a>.</p>
<p>The lessons to be learned are:</p>
<h3 id="1-acknowledge">1. Acknowledge</h3>
<p>Technical debt is inevitable and will become a major problem if ignored. We got sidetracked in the pursuit of pleasing a big client and didn’t acknowledge or realize that we are accruing big time technical debt which later caused bugs and slowed down our ability to add new features.</p>
<h3 id="2-decide-strategically-understand-short-term-vs-long-term-consequences">2. Decide Strategically: Understand Short-Term vs Long-Term Consequences</h3>
<p>In your career, it will often make sense to ship a subpar system to gain market advantage. In fact, it would be a mistake not to. In our last <a href="http://www.paperistic.com/">startup</a>, we made the opposite mistake and didn’t deliver a <em>minimum viable product</em> on time. As a result, we lost out on crucial early feedback. The <strong>trick is that all stakeholders must understand and strategically prioritize</strong> the tradeoffs of speed vs quality. Non-technical stakeholders often don’t understand the concept and <a href="http://www.ontechnicaldebt.com/blog/steve-mcconnell-on-categorizing-managing-technical-debt/">Steve McConnell</a> has some good advice on how to educate them:</p>
<blockquote>
<p><strong>Technical staff should build on the technical debt metaphor as a way to talk to business staff to explain that if they take on short-term technical debt, they will need to pay it off or else it will end up costing the business on the long-term</strong>. For example: “If we spend X weeks working on this particular infrastructure area, it will allow us to add features A, B, and C. Although the work itself does not show immediate benefit, it will open the door for other work that will produce business benefits later on.” Now this becomes a productive discussion and we have a reason for the business to engage.</p>
</blockquote>
<h3 id="3-dont-let-it-get-out-of-control-pay-the-debt">3. Don’t Let it Get out of Control: Pay the Debt</h3>
<p>The longer you wait, the higher interest payments get. If you let technical debt accrue and not pay if off, future development will stall, code quality will suffer, tribal knowledge will be required to understand all the hacks and people working on the project will become demotivated. Have a plan for paying off the technical debt to avoid massive interest payments in the future. It should be a part of your normal development process.</p>
<p>Taking on technical debt is risky business: it gives you the short-term benefits, but you’ll have to pay the debt back with interest in the future. Interests will keep on accruing and the more you delay paying the debt off, the higher the interest payments are going to be. Dealing with technical debt is not always easy: <strong>upper management often don’t see the value</strong> since it doesn’t result in new features; <strong>Things break and you might even have to throw some code out</strong>. Embark on the journey and have faith that you are doing the right thing and that your system will be in a better shape than it was before.</p>
<p>In the <a href="http://codeahoy.com/2016/05/02/software-rot-entropy-and-the-broken-window-theory/">next post or so</a>, we’ll look at Software Entropy. Please like and follow us on <a href="https://www.facebook.com/codeahoy">Facebook</a> and <a href="http://twitter.com/codeahoy">Twitter</a> to stay up-to-date.</p>
What is HTTP/2?2016-04-23T00:00:00+00:00https://codeahoy.com/2016/04/23/what-is-http2<p>There are many reasons to feel excited about HTTP/2. It is the first major update of the HTTP protocol in <strong>16 years!</strong> It was long overdue as the web dramatically evolved over the years. HTTP/2 is aimed at making the web faster and overcome many shortcomings of HTTP/1.1. HTTP/2 brings advancements to speed, efficiency, standardization and security.</p>
<h2 id="http11-simple-protocol-complex-workarounds">HTTP/1.1: Simple Protocol, Complex Workarounds</h2>
<p>HTTP was <a href="https://www.w3.org/Protocols/HTTP/AsImplemented.html">envisioned</a> as a simple, application level, <strong>request-response</strong> protocol. Clients connect to servers and make HTTP requests asking for resource. Servers send response back and terminate the connection.</p>
<p>HTTP/1.1 was released in 1999 to keep up with the performance demands of 90’s <a href="https://gizmodo.com/23-ancient-web-sites-that-are-still-alive-5960831">brochure-esque websites</a>. Its major improvement over its predecessor, the HTTP/1.0, was that it allowed a <em>connection to be reused</em> to send multiple requests and responses. This improves performance by eliminating connection setup overhead for each request.</p>
<!--more-->
<p>In 2004, Web 2.0 ushered in the new era of rich user experience and collaboration. It allowed users to interact with websites, leave comments, indulge in social sharing and enjoy many new features. <a href="https://en.wikipedia.org/wiki/Web_2.0#Technologies">New technologies</a> emerged and many Web 2.0 websites like Wikipedia, Flickr and YouTube went on to become huge successes.</p>
<p>While all this was going on, the protocol that ran it all, HTTP/1.1, was having trouble keeping up. The websites were growing significantly in size and complexity and it wasn’t designed to handle that. So smart people did what they do best: they invented workarounds, so-called best practices, to overcome HTTP/1.1 limitations. Hacks like <a href="http://stackoverflow.com/questions/14810890/what-are-the-disadvantages-of-using-http-pipelining">request pipelining</a>, <a href="https://www.maxcdn.com/one/visual-glossary/domain-sharding-2/">domain sharding</a>, <a href="https://developer.apple.com/library/iad/documentation/NetworkingInternet/Conceptual/SafariImageDeliveryBestPractices/ReducingHTTPRequestswithSprites/ReducingHTTPRequestswithSprites.html">sprite sheets</a> and <a href="http://www.websiteoptimization.com/speed/tweak/inline-images/">data inlining</a> were used to <strong>optimize performance</strong>. This added complexity on top of HTTP/1.1 and introduced regressions like unnecessary downloads, poor caching etc. Lack of standards meant that web developers had to deal with different browsers and versions. Ugh! I remember countless hours I sunk into tweaking websites to look great on both Firefox and Internet Explorer 7. It worked, but it was messy. And painful.</p>
<h2 id="the-journey-to-http2-a-spdy-stepping-stone">The Journey to HTTP/2: A SPDY Stepping Stone</h2>
<p>With no new development of HTTP/1.1 on the radar, Google took matters into their own very capable hands and started <a href="https://en.wikipedia.org/wiki/SPDY"><strong>SPDY</strong> (pronounced Speedy)</a>:</p>
<blockquote>
<p>SPDY is a replacement for HTTP, designed to speed up transfers of web pages, by eliminating much of the overhead associated with HTTP. SPDY supports <strong>out-of-order responses, header compression, server-side push, and other optimizations</strong> that give it an edge over HTTP when it comes to speed.</p>
</blockquote>
<p>In reality, SPDY <strong>didn’t really replace HTTP/1.1</strong> (and neither does HTTP/2 as we’ll later see). Augmented would be the right word. SPDY sat on top of HTTP/1.1 and heavily modified the data transfer formats and connection handling.</p>
<p>Google released SPDY in 2010 in Chrome 6 and soon deployed SPDY across all Google services. Word spread and SPDY soon gained traction and support from the community and vendors like Mozilla, Nginx, Microsoft and Facebook. Internet Engineering Task Force (IEFT), responsible for HTTP standards, seized the opportunity and <a href="https://tools.ietf.org/html/rfc7540">published HTTP/2 standards in 2015</a> deriving heavily from SPDY; SPDY’s fingerprints are all over HTTP/2.</p>
<p>Google has announced that it will <a href="http://blog.chromium.org/2016/02/transitioning-from-spdy-to-http2.html">stop supporting SPDY on May 15th, 2016</a>. Adios SPDY, you will be remembered as an important stepping stone on the journey to HTTP/2.</p>
<h2 id="http2-features">HTTP/2 Features</h2>
<p>HTTP/2 is here. Every modern web browser now supports HTTP/2. All major cloud and CDN vendors support it. The adoption among websites isn’t significant but it’s growing. As of April 24, 2016, <a href="http://w3techs.com/technologies/details/ce-http2/all/all">7.2% of all websites use HTTP/2</a>.</p>
<p>HTTP/2 brings many improvements to HTTP/1.1 and the biggest ones you need to know are:</p>
<h3 id="1-multiplexing">1. Multiplexing</h3>
<p>Under HTTP/1.1, connections were persistent which allowed multiple requests to be sent or <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">pipelined</a> over the same TCP connection. While it improved performance by reducing connection establishment overhead, it wasn’t a silver bullet. Even though multiple requests could be sent over the same connection, the responses must arrive synchronously in the same order as they were requested. This means an expensive resources (e.g. loading a large image file) will block a lightweight response if requested in the wrong order. This phenomenon is known as the <a href="https://en.wikipedia.org/wiki/Head-of-line_blocking">head-of-line (HOL) blocking</a>. In fact, pipelining was so poorly supported by web servers that many web browsers simply disabled it.</p>
<p>HTTP/2 allows multiplexing that solves HOL issue by allowing <strong>responses to be arrived out of order</strong>, thus eliminating the need to open multiple connections.</p>
<h3 id="2-compression--metadata-to-reduce-header-overhead">2. Compression & Metadata to Reduce Header Overhead</h3>
<p>Each and every request and response under HTTP/1.1 has a header typically between 200 bytes to 2KB in size. The first issue is that the <strong>headers are not permitted to be compressed</strong>. Another issues with HTTP/1.1 headers is that they contain a lot of <strong>redundant information</strong> that is exchanged several hundred times as browsers make as many requests to load a webpage. Static headers like Accept* and User-Agent only need to be exchanged once.</p>
<p>HTTP/2 fixes both these problems by compressing and eliminating unnecessary headers. <em>High five.</em></p>
<h3 id="3-server-push">3. Server Push</h3>
<p>HTTP/2 server can send data to a client before the client even asks for it. To understand why it is beneficial, let’s understand how a webpage is loaded under HTTP/1.1: web browser requests a web page, waits for it to be downloaded, parses it to find all linked assets such CSS and JavaScript and then make <em>separate requests</em> to download these asserts.</p>
<p>HTTP/2 Server Push allows the server to send these files to the browser proactively with the first request knowing that it will need them to display the page. I’m sure there’s a way to tell the server to stop being so proactive if the web browser already has files in its cache.</p>
<h3 id="4-encryption-is-not-really-optional">4. Encryption is <em>Not Really</em> Optional</h3>
<p>People debated for a long time whether or not encryption (HTTPS) should be made <strong>mandatory</strong> in HTTP/2. In the end, the standards folks <a href="http://http2.github.io/faq/#does-http2-require-encryption">decided not to make it mandatory</a>, with a fair bit of warning:</p>
<blockquote>
<p>After extensive discussion, the Working Group did not have consensus to require the use of encryption (e.g., TLS) for the new protocol. However, some implementations have stated that they will only support HTTP/2 when it is used over an encrypted connection, and currently no browser supports HTTP/2 unencrypted.</p>
</blockquote>
<p>So while the use of TLS is not imposed by the standard, browser vendors have made it a requirement. <strong>Google Chrome and Mozilla Firefox have pledged to not support HTTP/2 without HTTPS</strong>.</p>
<h3 id="5-binary-encoding">5. Binary Encoding</h3>
<p>Unlike HTTP/1.1, HTTP/2 uses binary encoding. Without getting into binary vs text protocols debate, this means that HTTP/2 will be more efficient to parse and compact on the wire but will no longer be human readable. When I was learning HTTP, one of the first things I did was made a request by hand and looked at the response along with all the headers as it arrived in all its glory. Unfortunately, this won’t be possible in HTTP/2 - at least not without specialized tools.</p>
<h3 id="5-backwards-compatibility">5. Backwards Compatibility</h3>
<p><strong>HTTP/2 is backwards compatible</strong> with HTTP/1.1. This means if you want to upgrade to HTTP/2, you could do so without changing anything. The upgrade will be equally seamless to your users. Don’t forget to undo HTTP/1.1 performance optimizations (aka best practices) as they no longer provide the same benefits.</p>
<h2 id="http2---future-is-fast-and-complex">HTTP/2 - Future is Fast… And Complex</h2>
<p>People have <a href="https://queue.acm.org/detail.cfm?id=2716278">criticized</a> HTTP/2 for being Google’s idea, optimized for their needs and having needless complexity (things like flow controls to prevent DOS attacks). One has to agree that HTTP/2 is undoubtedly more complex than its predecessors, but the complexity is a necessary evil to keep up with today’s needs. An average webpage in 2016 is the same <a href="https://mobiforge.com/research-analysis/the-web-is-doom?r=1">size as the original DOOM shareware binary</a>. Sure Google benefits from HTTP/2, but don’t we all? It is time to move beyond HTTP/1.1 and I believe HTTP/2 is the answer.</p>
When to Rewrite from Scratch - Autopsy of a Failed Software2016-04-21T00:00:00+00:00https://codeahoy.com/2016/04/21/when-to-rewrite-from-scratch---autopsy-of-a-failed-software<p>It was winter of 2012. I was working as a software developer in a small team at a start-up. We had just released the first version of our software to a real corporate customer. The development finished right on schedule. When we launched, I was over the the moon and very proud. It was extremely satisfying to watch the system process couple of million of unique users a day and send out tens of millions of SMS messages. By summer, the company had real revenue. I got promoted to software manager. We hired new guys. The company was poised for growth. Life was great. <strong>And then we made a huge blunder and decided to rewrite the software. From scratch.</strong></p>
<p><!--more--></p>
<h2 id="why-we-felt-that-rewrite-from-scratch-was-needed">Why We Felt That Rewrite from Scratch Was Needed?</h2>
<p>We had written the original system with a gun to our heads. We had to race to the finish line and incurred technical debt. We weren’t having long design discussions or review meetings - we didn’t have time for such things. We would finish a feature as quickly as we can, get it tested and release it to the customer. We had a shared office (from TRTech) and I remember new software developers at other companies getting into lengthy design and recurring architecture debates over design patterns, something we couldn’t afford to do.</p>
<p>Despite agile-on-steroids design, the original system wasn’t badly written and generally was well structured. There was some spaghetti code that carried over from company’s previous proof of concept attempts that we left untouched because it was working and we had no time. But instead of thinking about incremental improvements, we <em>convinced</em> ourselves that we need to rewrite from scratch because:</p>
<ul>
<li>the old code was bad and hard to maintain.</li>
<li>the “monolith java architecture” was inadequate for our future need of supporting a very large operator with 60 million mobile users and multi-site deployments.</li>
<li>I <em>wanted</em> to try out new, shinny technologies like Apache Cassandra, Virtualization, Binary Protocols, Service Oriented Architecture, etc.</li>
</ul>
<p>We convinced the entire organization and the board and sadly, we got our wish.</p>
<h2 id="the-rewrite-journey">The Rewrite Journey</h2>
<p>The development officially began in spring of 2012 and we set end of January, 2013 as the release date. Because the vision was so grand, we needed even more people. I hired consultants and couple of remote developers in India. However, we didn’t fully anticipate the need to maintain the original system in parallel with new development and underestimated customer demands. Remember I said in the beginning we had a real customer? The customer was one one of the biggest mobile operators in South America and once our system had adoption from its users, they started making demands for changes and new features. So we had to continue updating the original system, albeit half-heartedly because we were digging its grave. We dodged new feature requests from the customer as much as we can because we were going to throw the old one away anyways. This contributed to delays and we missed our January deadline. In fact, we missed it by 8 whole months!</p>
<p>But let’s skip to the end. When the project was finally finished, it looked great and met all the requirements. Load tests showed that it can easily support over 100 million users. The configuration was centralized and it had a beautiful UI tool to look at charts and graphs. It was time to go and kill the old system and replace it with the new one… <strong>until the customer said “no” to upgrade</strong>. It turned out that the original system had gained wide adoption and their users had started relying on it. They wanted absolutely no risks. Long story short, after months of back and forth, we got nowhere. The project was officially doomed.</p>
<h2 id="lessons-learnt">Lessons Learnt</h2>
<ul>
<li>You should almost never, ever rewrite from scratch. We rewrote for all the wrong reasons. While parts of code were bad, we could have easily fixed them with refactoring if we had taken time to read and understand the source code that was written by other people. We had genuine concerns about the scalability and performance of the architecture to support more sophisticated business logic, but we could have introduced these changes incrementally.</li>
<li>Systems rewritten from scratch offer no new value to the user. To the engineering team, new technology and buzzwords may sound cool but they are <strong>meaningless to customers</strong> if they don’t offer new features that the customers need.</li>
<li>We <strong>missed real opportunities</strong> while we were focused on the rewrite. We had a very basic ‘Web Tool’ that the customer used to look at charts and reports. As they became more involved, they started asking for additional features such as real-time charts, access-levels, etc. Because we weren’t interested in the old code and had no time anyways, we either rejected new requests or did a bad job. As a result, the customer stopped using the tool and insisted on reports by email. Another lost opportunity was an opportunity to build a robust Analytics platform that was <em>badly</em> needed.</li>
<li>I underestimated the effort of maintaining the old system while the new one is in development. We estimated 3-5 requests a month and got 3 times as many.</li>
<li>We thought our code was harder to read and maintain since we didn’t use proper design patterns and practices that other developers spent days discussing. It turned out that most professional code I have seen in larger organizations is 2x time worst than that we had. So we were dead wrong about that.</li>
</ul>
<h2 id="when-is-rewrite-the-answer">When Is Rewrite the Answer?</h2>
<p><a href="http://www.joelonsoftware.com/articles/fog0000000069.html">Joel Spolsky made strong arguments against rewrite</a> and suggests that one should never do it. I’m not so sure about it. Sometimes incremental improvements and refactoring are very difficult and the only way to <strong>understand</strong> the code is to rewrite it. Plus software developers love to write code and create new things - it’s boring to read someone else’s code and try to understand their code and their ‘mental abstractions’. But good programmers are also good maintainers.</p>
<p>If you want to rewrite, do it for the right reasons and plan properly for the following:</p>
<ul>
<li>The old code will still need to be maintained, in some cases, long after you release the new version. Maintaining two versions of code will require huge efforts and you need to ask yourself if you have enough time and resources to justify that based on the size of the project.</li>
<li>Think about losing other opportunities and prioritize.</li>
<li>Rewriting a big system is more risky than smaller ones. Ask yourself if you can incrementally rewrite. We switched to a new database, became a ‘Service Oriented Architecture’ and changed our protocols to binary, all at the same time. We could have introduced each of these changes incrementally.</li>
<li>Consider the developers’ bias. When developers want to learn a new technology or language, they want to write some code in it. While I’m not against it and it’s a sign of a good environment and culture, you should take this into consideration and weigh it against risks and opportunities.</li>
</ul>
<p>Michael Meadows made <a href="http://programmers.stackexchange.com/questions/6268/when-is-a-big-rewrite-the-answer">excellent observations</a> on when “BIG” rewrite becomes necessary:</p>
<blockquote>
<p><strong>Technical</strong></p>
<ul>
<li>The coupling of components is so high that changes to a single component cannot be isolated from other components. A redesign of a single component results in a cascade of changes not only to adjacent components, but indirectly to all components.</li>
<li>The technology stack is so complicated that future state design necessitates multiple infrastructure changes. This would be necessary in a complete rewrite as well, but if it’s required in an incremental redesign, then you lose that advantage.</li>
<li>Redesigning a component results in a complete rewrite of that component anyway, because the existing design is so fubar that there’s nothing worth saving. Again, you lose the advantage if this is the case.</li>
</ul>
<p><strong>Political</strong></p>
<ul>
<li>The sponsors cannot be made to understand that an incremental redesign requires a long-term commitment to the project. Inevitably, most organizations lose the appetite for the continuing budget drain that an incremental redesign creates. This loss of appetite is inevitable for a rewrite as well, but the sponsors will be more inclined to continue, because they don’t want to be split between a partially complete new system and a partially obsolete old system.</li>
<li>The users of the system are too attached with their “current screens.” If this is the case, you won’t have the license to improve a vital part of the system (the front-end). A redesign lets you circumvent this problem, since they’re starting with something new. They’ll still insist on getting “the same screens,” but you have a little more ammunition to push back.
Keep in mind that the total cost of redesigning incrementally is always higher than doing a complete rewrite, but the impact to the organization is usually smaller. In my opinion, if you can justify a rewrite, and you have superstar developers, then do it.</li>
</ul>
</blockquote>
<p>Abandoning working projects is dangerous and we wasted an enormous amount of money and time duplicating working functionality we already had, rejected new features, irritated the customer and delayed ourselves by years. If you are embarking on a rewrite journey, all the power to you, but make sure you do it for the right reasons, understand the risks and plan for it.</p>
Git Stash - Saving Your Changes2016-04-18T00:00:00+00:00https://codeahoy.com/2016/04/18/10-git-stash-explained<p>Let’s say you are in the middle of implementing a new feature. You’re half way through your changes and the code is in a messy state. You get a message that there’s an urgent issue that requires you to switch gears and work on it immediately. You don’t want to commit your half baked changes but also don’t want to lose your work because you want to revisit it at a later time? What do you do?</p>
<!--more-->
<p>The answer to this problem is the <code class="language-plaintext highlighter-rouge">git stash</code> command.</p>
<p>Running <code class="language-plaintext highlighter-rouge">git stash</code> will take the changes you’ve made to tracked files in the working directory as well as staged changes and saves them to a stack. You can reapply these changes from the stack at any time. After stashing, you’ll end up with a clean working directory and can freely switch branches and work on something else. Let’s walk through a complete example.</p>
<p>Let’s say we’re in the middle of editing <code class="language-plaintext highlighter-rouge">file.txt</code> when we get a Slack message to switch to something else right away. We’ll use <code class="language-plaintext highlighter-rouge">git stash</code> to save our changes.</p>
<pre class="prettyprint lang-sh">
$ echo "Improvement 1 of 3" >> file.txt
$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
modified: file.txt
$ git stash save "Partial improvement to file.txt"
$ git status
# On branch master
nothing to commit, working directory clean
</pre>
<p>After stashing, you’re free to do whatever you like. You can switch to a different branch, make your changes, commit and push them to the remote repo. After you’re done, you’re ready to resume working on <code class="language-plaintext highlighter-rouge">file.txt</code> which you stashed away. To reapply, you will use the <code class="language-plaintext highlighter-rouge">git stash apply</code> command.</p>
<pre class="prettyprint lang-sh">
$ git stash apply
</pre>
<p>Assuming you don’t run into merge issues, you should get your partial changes back. If not, git will throw a merge conflict error if it can’t reapply your changes safely. We’ll look at how to resolve merge conflicts in the next chapter.</p>
<p>You can stash multiple times. Each time you run <code class="language-plaintext highlighter-rouge">git stash</code> it will save a new stash on the stack. To see a list of all the stashes stored on the stack, use the <code class="language-plaintext highlighter-rouge">git stash list</code> command.</p>
<pre class="prettyprint lang-sh">
$ git stash list
stash@{0}: WIP on master: d724198 partial improvement 2
stash@{1}: WIP on master: d724198 bug fix for Unity
stash@{2}: WIP on master: c9a03f4 added partial improvement 1
</pre>
<p>The stashes are ordered by most recent to newest (hence it is a stack.) <code class="language-plaintext highlighter-rouge">stash@{0}</code> is the most recent and <code class="language-plaintext highlighter-rouge">stash@{2}</code> is the oldest in the example above. To restore the very first stash you saved:</p>
<pre class="prettyprint lang-sh">
$ git stash apply stash@{2}
</pre>
<p>A stash could be applied to any branch not just the same branch it was saved from. Also note that stash will ignore ‘un-tracked’ files. If you added a new file, you must first add it to the index using <code class="language-plaintext highlighter-rouge">git add</code> before stashing.</p>
<h3 id="merge-conflicts">Merge conflicts</h3>
<p>There are times when <code class="language-plaintext highlighter-rouge">git stash apply</code> won’t work and throw a merge conflict. E.g.</p>
<pre class="prettyprint lang-sh">
$ git stash apply
error: The following untracked working tree files would be overwritten by merge:
README.md
Please move or remove them before you merge.
Aborting
</pre>
<p>The easiest way to get out of merge conflicts is to apply your stash to a new branch. To do this you can use <code class="language-plaintext highlighter-rouge">git stash branch <new branchname></code>. This will create a new branch, check out the commit when you stashed your changes, reapply your stash on top of it.</p>
<pre class="prettyprint lang-sh">
$ git stash branch temp_restore
</pre>
<p>That’s pretty much it. Here are some bonus tips for git stash:</p>
<ul>
<li>To save your stash with a message or give it a name you can use the following syntax: <code class="language-plaintext highlighter-rouge">git stash save <message></code>. For example: <code class="language-plaintext highlighter-rouge">git stash save "feature orca-654"</code>.</li>
<li>By default, stash doesn’t save untracked files. You could either stage them or save with the <code class="language-plaintext highlighter-rouge">-u</code> switch e.g. <code class="language-plaintext highlighter-rouge">git stash save -u</code></li>
<li>To delete the stash after it has been applied, you can use the <code class="language-plaintext highlighter-rouge">git stash pop</code> command e.g. <code class="language-plaintext highlighter-rouge">git stash pop stash@{2}</code> will apply <code class="language-plaintext highlighter-rouge">stash@{2}</code> and delete it from the stack.</li>
<li>To delete a stash without applying it, use <code class="language-plaintext highlighter-rouge">git stash drop</code> e.g. <code class="language-plaintext highlighter-rouge">git stash drop stash@{0}</code></li>
<li>Use <code class="language-plaintext highlighter-rouge">git stash show</code> to see a summary of diffs.</li>
<li>To delete all your stashes, use the <code class="language-plaintext highlighter-rouge">git stash clear</code> command. Be careful because this is dangerous command. You will lose your stashes forever.</li>
</ul>
<p>That’s all. If you enjoyed this article, please share it with your friends. The links to share on Facebook, Twitter, LinkedIn are below.</p>
What's the difference between git fetch vs git pull?2016-04-18T00:00:00+00:00https://codeahoy.com/2016/04/18/10-git-pull-vs-git-fetch-(and-stashing)<p>Git has two types of repositories: local and remote. The local repository is on your computer and has all the files, commit history etc. Remote repositories are usually hosted on a central server or on the Internet.</p>
<p>Downloading data from the remote repo to local is an essential part of working with git.</p>
<p>Both <strong>git pull</strong> and <strong>git fetch</strong> are used to download data from remote repository. These two commands have important differences and similarities. Let’s explore them in more detail.</p>
<!--more-->
<h2 id="the-main-difference-between-git-fetch-and-git-pull">The main difference between <code class="language-plaintext highlighter-rouge">git fetch</code> and <code class="language-plaintext highlighter-rouge">git pull</code></h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git fetch origin
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">git fetch</code> <strong>only</strong> downloads the latest data from the remote repository. It does <strong>not merge</strong> any of this new data into the current branch or changes the working files. Fetch is <em>safe</em> because it only downloads new changes (since you last synced with the remote.) It doesn’t create conflicts or interfere with the work in progress. Developers use <code class="language-plaintext highlighter-rouge">fetch</code> to find out if there have been new changes, review changes before merging or sometimes to track someone else’s feature branch.</p>
<hr />
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git pull origin master
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">git pull</code> in contrast <em>not only</em> downloads the latest data, but it also <strong>automatically merges</strong> it into your current branch and updates the working files automatically. It doesn’t give you a chance to review the changes before merging, and as a consequence, ‘merge conflicts’ can and do occur. One important thing to keep in mind is that it will merge <em>only</em> into the current working branch. Other branches will stay unaffected.</p>
<p>Here’s a diagram to illustrate the difference between git fetch and git pull.</p>
<p><img src="https://codeahoy.com/img/git-pull-vs-fetch.png" alt="git pull and fetch" /></p>
<p>I have covered the main difference between <code class="language-plaintext highlighter-rouge">git fetch</code> and <code class="language-plaintext highlighter-rouge">get pull</code> above. But if you want <strong>more details</strong>, read on.</p>
<h2 id="git-fetch-explained-in-detail"><code class="language-plaintext highlighter-rouge">git fetch</code> explained in detail</h2>
<p>As we’ve seen, <code class="language-plaintext highlighter-rouge">git fetch</code> <strong>only downloads</strong> latest changes into the local repository, and <strong>does not merge</strong> into the current branch. It downloads fresh changes that other developers have pushed to the remote repo since the last fetch and allows you to review and merge manually at a later time using <code class="language-plaintext highlighter-rouge">git merge</code>. Because it doesn’t change your working directory or the staging area, it is entirely safe, and you can run it as often as you want.</p>
<p>You may be thinking where are the changes are stored after a fetch since they are not merged into the working file? The answer is that they are stored in your local repository in what is called <a href="https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches">remote tracking branches</a>. A remote tracking branch is a local copy (or mirror) of a remote branch, e.g. <code class="language-plaintext highlighter-rouge">origin/master</code>. You’ll need to run <code class="language-plaintext highlighter-rouge">git branch -a</code> to see all local and remote branches. After doing a fetch, if you want to check out what has changed, you can do:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">git log origin/master ^master</code>: to get a list of all commits that are in remote master but not in your local branch.</li>
<li><code class="language-plaintext highlighter-rouge">git diff ..origin</code>: to see the diff.</li>
<li><code class="language-plaintext highlighter-rouge">git checkout origin/master</code>: to checkout the remote master branch and see what files have changed.</li>
</ul>
<p>Once you have reviewed the changes and are ready to merge, you can switch back to the master branch and run <code class="language-plaintext highlighter-rouge">git merge</code>. It will merge changes from the remote branch into local.</p>
<p>Here’s an example. Let’s switch to develop branch and do <code class="language-plaintext highlighter-rouge">git fetch</code>. To keep it simple, I’ll omit the output.</p>
<pre class="prettyprint lang-sh">
$ git checkout develop
$ git fetch
</pre>
<p>Now let’s see the list of commits that are in remote develop but not in local.</p>
<pre class="prettyprint lang-sh">
$ git log origin/develop ^develop
commit 6123ef537f0dac5410f409a8dfc2719491e13fc9 (origin/master, origin/HEAD)
Author: Umer Mansoor <...@gmail.com>
Date: Sat Feb 1 08:11:52 2020 -0800
fixed toc
commit 7143fccddce97405b05f51facf9e1560301027ab
Author: Umer Mansoor <...@gmail.com>
Date: Sat Feb 1 08:10:01 2020 -0800
Update README.md
</pre>
<p>When you are ready to merge, simply run:</p>
<pre class="prettyprint lang-sh">
$ git merge
Updating 89aaded..5926bf5
Fast-forward
README.md | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
</pre>
<h2 id="git-pull-example"><code class="language-plaintext highlighter-rouge">git pull</code> example</h2>
<p>The <code class="language-plaintext highlighter-rouge">git pull</code> command downloads from the remote repository to the local repository and <strong>automatically merges</strong> those changes into the <em>current branch</em>.</p>
<pre class="prettyprint lang-sh">
$ git checkout master
...
$ git pull
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From https://github.com/repo/arepo
7548eeb..8599bfe master -> origin/master
Updating 7548eeb..8599bfe
Fast-forward
README.md | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 README.md
</pre>
<p>You can see that in the output above, the changes were downloaded from the remote master and then merged using fast-forward method into the local master branch.</p>
<h4 id="summary">Summary</h4>
<p>In summary, <code class="language-plaintext highlighter-rouge">pull</code> and <code class="language-plaintext highlighter-rouge">fetch</code> are similar in the sense that they both download latest changes from the remote repo to local. The <strong>difference</strong> is that <code class="language-plaintext highlighter-rouge">pull</code> automatically merges changes into the current branch while <code class="language-plaintext highlighter-rouge">fetch</code> doesn’t. If you are interested, here are some more commands and details that I didn’t cover in the post to keep it simple.</p>
<ul>
<li>Under the hood, <code class="language-plaintext highlighter-rouge">git pull</code> is equivalent to running <code class="language-plaintext highlighter-rouge">git fetch origin HEAD</code> followed by <code class="language-plaintext highlighter-rouge">git merge HEAD</code>.</li>
<li>To run <code class="language-plaintext highlighter-rouge">git pull</code> in verbose mode, add the verbose switch i.e. <code class="language-plaintext highlighter-rouge">git pull --verbose</code></li>
<li>To put all your changes on top of what everyone else has committed, you can pull using the rebase flag i.e. <code class="language-plaintext highlighter-rouge">git pull --rebase origin</code>.</li>
</ul>
<p>If you’re interest, check out my next tutorial on <a href="https://codeahoy.com/2016/04/18/10-git-stash-explained/">git stash</a>. Stashing is a useful tool in git that allows users to save their partially complete changes and reapply them at a later time.</p>
The Problem With Code Coverage Metrics2016-04-16T00:00:00+00:00https://codeahoy.com/2016/04/16/do-not-misuse-code-coverage<p>Code coverage is a valuable metric. Software developers write tests for the code they have written and run code coverage analysis which gives them an assessment of how much of their code is covered by tests or more importantly what parts of the code are untested.</p>
<p>Some organizations and managers make high level of code coverage mandatory for their teams. “<strong>90% code coverage and no less!</strong>”, says the manager. It becomes another metric in management’s arsenal to <em>assess</em> code quality and (god forbid) team’s performance. It’s a <strong>big mistake</strong> to interpret and use code coverage in this way. Code coverage doesn’t say anything about the quality of the code or the tests. It is very easy to get high code coverage with <em>low quality</em> testing. Code coverage number will not say that some parameter was not checked for <code class="language-plaintext highlighter-rouge">null</code> value, that the contract required <code class="language-plaintext highlighter-rouge">String</code>s to be <code class="language-plaintext highlighter-rouge">trim()</code>ed before use or that even though all lines of code were hit, some particular sequence wasn’t tested. Nope. The tests might be meaningless and brittle masking real issues, but who cares, as long as there is coverage.</p>
<!--more-->
<p>Let me get this straight again: <em>code coverage is a valuable metric</em>. But when management turns it into a goal and becomes fixated on it to measure quality and performance, things rapidly disintegrate. It might have to do with our <a href="http://www.amazon.com/exec/obidos/ASIN/0932633366/ref=nosim/joelonsoftware">psychology</a> or our nature, but we humans optimize our performance according to how we are being measured and become distracted from what really matters. Testing requires thoughtfulness and careful design. Scott Bain, author at Sustainable Test Driven Development <a href="http://www.sustainabletdd.com/2011/12/lies-damned-lies-and-code-coverage.html">explained</a> it better:</p>
<blockquote>
<p>If developers are writing unit tests because “the boss says so” then they have no real professional or personal motivation driving the activity. They’re doing it because they have to, not because they want to. Thus, they will put in whatever effort they have to in order to increase their code coverage to the required level and not one bit more. <strong>It becomes a “tedious thing I have to do to before I can check in my code, period.”</strong></p>
</blockquote>
<p>In his excellent article “<a href="http://www.exampler.com/testing-com/writings/coverage.pdf">How to Misuse Code Coverage</a>”, Brian Marick discovered the same thing. Organization that mandated a code coverage percentage got just the percentage they wanted. Which is scary because it might be a sign that people are gaming the system to meet the target.</p>
<blockquote>
<p>Perhaps this might hint at the answer: when I talk about coverage to organizations that use
85%, say, as a shipping gate, I sometimes ask how many people have gotten substantially
higher, perhaps 90%. There’s usually a few who have, <strong>but everyone else is clustered right
around 85%. Are we to believe that those other people just happened to hit 85% and were
unable to find any other tests worth writing? Or did they write a first set of tests, take their
coverage results, bang away at the program until they got just over 85%</strong>, and then heave a
sigh of relief at having finished a not-very-fun job?</p>
</blockquote>
<p>Even if we take the positive outlook that people will make an honest effort to meet enforced code coverage percentage, it’s still counter-productive to have them obsess over the code coverage as a number rather than focusing on the quality, sufficiency and maintainability of tests.</p>
<h2 id="but-my-team-has-had-several-bugs-escape-to-production">“But my team has had several bugs escape to production!”</h2>
<p>Mandating a code coverage number is not the answer. If anything, you’ll make matters worse. Writing good tests is a skill. As a manager, it is your job is to figure out exactly what is going wrong.</p>
<p>I managed a team that consistently had issues with bugs. Luckily, we were catching these bugs in Q/A but because the development team was offshore and in a different timezone, the feedback loop was becoming a burden. The local software development manager was out of ideas as well. I started my investigation one evening. The first thing I looked at was their code coverage. Not stellar at 76% but not so bad either. I scratched my head and dug into the source code and started looking at the tests. And I didn’t have to dig deeper to find the problem: the tests were <em>crap</em>. I remember a small test that alone got 40% coverage! It was a <a href="http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/">dirty hybrid of unit and integration test </a> that tested a high level event processing method with the right parameters. It didn’t even check what would happen when one of the parameters is missing or has the wrong value.</p>
<p>In their defense, they had assigned a “junior developer” to write the tests while the “senior guys” wrote actual code. How did we fix it? We didn’t ask the team to increase code coverage, in fact, we didn’t even mention it. We started educating them on how to write good unit tests. We encouraged them to watch <a href="https://www.youtube.com/watch?v=wEhu57pih5w">videos</a> during office hours etc. After initial resistance and passive-aggressiveness, they saw the light and realized that a lot of errors Q/A were discovering, they could find themselves using proper unit tests techniques. More importantly, they realized that for them to professionally grow, they must learn to write proper tests and <em>write themselves</em>. The bug count went down significantly.</p>
<h2 id="the-google-approach">The Google approach</h2>
<p>I like the Google <a href="https://docs.google.com/presentation/d/1god5fDDd1aP6PwhPodOnAZSPpD80lqYDrHhuhyD7Tvg/edit#slide=id.g3f5c82004_99_130">approach</a>. They <strong>strive</strong> for 85% code coverage but it is not <strong>“set in stone”</strong>. Their code coverage results over a month are shown in the graph below.</p>
<p><img src="https://codeahoy.com/img/coveragegoogle.png" alt="Google Code Coverage" /></p>
<p>Pretty Impressive.</p>
<h2 id="summary">Summary</h2>
<p>Managers should expect their team to have high coverage but they must not turn it into a target. Metrics do not make good code. The ultimate goal is to have fewer bugs that escape into the production.</p>
Is it OK to make mistakes at work?2016-04-14T00:00:00+00:00https://codeahoy.com/2016/04/14/mistakes-at-work-are-not-sins<blockquote>
<p>I have not failed. I have just found 10,000 ways that won’t work. -Thomas Edison</p>
</blockquote>
<p>Software development is an activity that requires serious brain power. And it’s only natural that software developers make mistakes along the way. However, many organizations and managers stigmatize failures. It’s considered a <em>sin</em> to make an error on the job. A sin that is not going to be forgotten any time soon.</p>
<p>We are taught from an early age that mistakes are bad. Students are marked by the number of mistakes and the society looks down on failures. So why should managers allow it? The answer is simple: <strong>people will make mistakes in any activity that requires imagination and creativity</strong>. That’s how they learn and improve. <a href="http://www.amazon.com/Peopleware-Productive-Projects-Second-Edition/dp/0932633439">DeMarco and Lister</a> explained it better:</p>
<blockquote>
<p>Fostering an atmosphere that doesn’t allow for error simply
makes people defensive. They don’t try things that may turn out
badly. You encourage this defensiveness when you try to systematize
the process, when you impose rigid methodologies so that staff
members are not allowed to make any of the key strategic decisions
lest they make them incorrectly. The average level of technology
may be modestly improved by any steps you take to inhibit error.
<strong>The team sociology, however, can suffer grievously</strong>.</p>
</blockquote>
<!--more-->
<p>The last sentence is the key: <em>The team sociology, however, <del>can</del> will suffer grievously</em>. As a software manager, your primary day job is to make it possible for your team to do work. When people become defensive, they lose motivation to do good work. Whenever a ‘bug’ is reported by customer or a client, instead of focusing their efforts to objectively locate the bug, employees spend time and energy on ‘covering their asses’ and collecting data to ‘prove’ that there has to be something “wrong with the other system.”</p>
<p>The biggest fear managers have is that mistakes made will cost their company money or damage customer relationship. While this is true, most mistakes made in the workplace don’t damage company’s reputation. Managers should identify <strong>few</strong> key customer facing areas where mistakes will be catastrophic and put additional oversight and checks in places. Keep this list very short and allow employees the freedom to make mistakes in other areas.</p>
<p>Another common irrational fear is that by allowing mistakes, people will make ‘stupid’ or ‘repeated’ mistakes. I guess I have have been lucky to have managed very few ‘bad apples’. (I could only think of 2 bad employees in the last 4 years.) Majority of software developers are smart, talented and proud people who value quality work, learning and professional development.</p>
<p>In his autobiography, <a href="http://www.amazon.com/Against-Odds-Autobiography-Business-Icons/dp/1587990148">Against the Odds</a>, James Dyson writes:</p>
<blockquote>
<p>I made <strong>5127 prototypes</strong> of my vacuum before I got it right. There were <strong>5126 failures</strong>. But I learned from each one. That’s how I came up with a solution. So I don’t mind failure. I’ve always thought that schoolchildren should be marked by the number of failures they’ve had. The child who tries strange things and experiences lots of failures to get there is probably more creative…
We’re taught to do things the right way. <strong>But if you want to discover something that other people haven’t, you need to do things the wrong way</strong>.</p>
</blockquote>
<p>I couldn’t agree more. Mistakes are part of a healthy team culture. Unless the job is dead simple and requires no imagination and creativity, mistakes will be made. I made a mistake in 2012 that cost my company as much as $5,000. A few ‘billable’ events were lost due to a Redis library issue. I had just convinced my bosses to let me replace the old MySQL database with shinny new Redis. I was devastated. We found the bug after 3 sleepless nights. I was very angry at myself for screwing up. But my boss shrugged it off, thanked me sincerely for fixing the problem and recognized my efforts publicly. Quoting <a href="https://en.wikipedia.org/wiki/John_Wooden">someone</a>, he said to me: “<em>If you’re not making mistakes, then you’re not trying anything new.</em>”</p>
<p>Mistakes should be embraced and celebrated. Software development is a difficult activity and software developers will make mistakes. They shouldn’t be crucified for making <em>honest</em> mistakes. It’s part of the learning. Great managers don’t cheat their employees of personal growth and development opportunities.</p>
Generating Sessions Ids2016-04-13T00:00:00+00:00https://codeahoy.com/2016/04/13/generating-session-ids<p>Session Id’s are <strong>unique, short-lived numbers</strong> that servers assign to users when they log in (or visit) so they can remember (or track) users for the duration of their sessions. Servers use session Id’s to remember users because the underlying protocol, HTTP, is <a href="https://en.wikipedia.org/wiki/Stateless_protocol">stateless</a>. Once they receive session Id from the server, users send it back in the following requests to identify themselves. For example, when you login to a website, the server assigns you a session Id and sends it to your browser wrapped in a <a href="https://en.wikipedia.org/wiki/HTTP_cookie">cookie</a>. The browser <strong>automatically sends the cookie back</strong> in the subsequent requests so the server knows who is making the request.</p>
<p>Almost <a href="https://docs.djangoproject.com/en/1.10/topics/http/sessions/">all</a> web <a href="https://github.com/expressjs/session">frameworks</a> I have worked with have built-in support for sessions: they generate and assign Id’s under the hood. The only time I had to generate session Id’s manually was when I was building a REST application (game service) that needed a custom way to identify users and sessions. This blog post is the result of research I had to do to build that feature. I would highly recommend not rolling out your custom session handling code, unless you absolutely have to.</p>
<!--more-->
<h2 id="session-ids-are-unique-transient-and-non-guessable">Session Ids are unique, transient and non-guessable</h2>
<ol>
<li>
<p>Session Id’s must be unique across all users. Can you imagine two people getting assigned the same Social Security number? <a href="http://www.pcworld.com/article/3004654/government/a-tale-of-two-women-same-birthday-same-social-security-number-same-big-data-mess.html">That would be a disaster</a>.</p>
</li>
<li>
<p>Session Id’s have ‘best-by’ date and they <em>timeout</em> after a certain period. If they didn’t, a hacker could steal and use them indefinitely. Generally the expiry period ranges from minutes to weeks. High-risk applications expire session Ids more frequently than the low risk ones to minimize the attack window.</p>
</li>
<li>
<p>Session Id’s are not guessable. A bad example would be an algorithm that generates sequential session Id’s. Hackers can easily identify patterns and hijack user sessions.</p>
</li>
</ol>
<p>You can generate and assign session Id’s to users in many different ways. I’ll discuss three common methods below.</p>
<h2 id="1-random-session-ids">1. Random session Ids</h2>
<p>Random session Id’s have no meaning by virtue of being completely random. The server sends them to the client and stores them in a database along with the the user information.</p>
<table>
<thead>
<tr>
<th>Session Id</th>
<th>User Id</th>
<th>Expiry Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>fb2e77d.47a0479900504cb3ab4a1f626d174d2d</td>
<td>jimHalpert1</td>
<td>15 minutes</td>
</tr>
</tbody>
</table>
<p>If session Id’s are random numbers, how do we ensure that they cannot be <strong>guessed</strong> or predicted by hackers? In Cryptography theory, entropy is the measure of <strong>uncertainty</strong> associated with a random number. Session Id’s should have <em>very</em> high entropy to protect against attacks. <a href="https://www.owasp.org/index.php/Session_Management_Cheat_Sheet#Session_Expiration">OWASP</a> suggests at least 64 bits of entropy. Sounds complex? (it did to me.) The good news is that you and I should never have to worry about writing your own algorithms (Don’t even think about it - random number generation is very complex). Most languages have <a href="https://en.wikipedia.org/wiki/Pseudorandom_number_generator">pseudo random numbers generators (PRNGs)</a> that generate ‘cryptographically secure’ random numbers that have entropy. As an example, Tomcat uses SHA1<strong>PRNG</strong> to generate a random number and hash it with MD5 (see warning below) to create session Id’s. Here’s a link to the <a href="https://docs.jboss.org/jbossas/javadoc/4.0.2/org/jboss/web/tomcat/tc5/session/SessionIDGenerator.java.html">source code</a> (There’s a list of PRNGs for other languages near the end of this post.)</p>
<p><strong>Warning</strong>: <strong>Do NOT use MD5</strong> to generate session Id’s because it is <a href="https://security.stackexchange.com/questions/19906/is-md5-considered-insecure">considered insecure</a>. In the application I was building, I used SHA-2 (SHA-256).</p>
<h2 id="2-session-ids-using-shared-secret">2. Session Id’s using shared secret</h2>
<p>These types of session Id’s are created in such a way that the information needed to identify a user is embedded into the session Id itself. Since session Id’s are self-contained, the server <strong>doesn’t need to store them</strong>. Let’s look at a trivial algorithm that generates session Id’s by combining username, IP address and a client secret:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sessionId</span> <span class="o">=</span> <span class="no">SHA2</span><span class="o">(</span><span class="n">username</span> <span class="o">+</span> <span class="n">ipAddress</span> <span class="o">+</span> <span class="n">secretKey</span> <span class="cm">/* or salt */</span><span class="o">)</span>
</code></pre></div></div>
<p>When a request arrives, it contains the username and IP address is automatically recorded. The server then uses the username, the IP address and secret key to <em>re-generate</em> the session Id and see if it matches with the session Id passed by the client. If it does, the verification is successful.</p>
<p><strong>Note</strong>: If you use IP address to calculate session Id’s, keep in mind that the session Id will be invalidated when the IP address changes. This happens very frequently if your users are on a mobile network and are moving. If you are not sure, don’t use the IP address.</p>
<p>The <strong>advantage</strong> of this method is that the server doesn’t have to maintain state and store session Id’s in a database.</p>
<p><strong>Disclaimer:</strong> The Session Id generation formula above is simplistic. Real applications would combine many parameters such as the user’s access group, timestamp, etc. The timestamp is generated based on Session Id’s lifetime to allow it to expire.</p>
<p><strong>Note:</strong> There is a cool standard called <a href="https://jwt.io/">JSON Web Tokens</a> that allows the payload to carry the information. I haven’t used it but it looks promising.</p>
<h2 id="3-random-session-ids-with-a-predictable-part">3. Random session Id’s with a predictable part</h2>
<p>This is a slight modification of the Random session Id generation method. The session Id consists of both a random number and a hash combining some properties of the user such as the username and IP address.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sessionId</span> <span class="o">=</span> <span class="no">SHA2</span><span class="o">(</span><span class="n">userId</span> <span class="o">+</span> <span class="n">ipAddr</span><span class="o">)</span> <span class="o">+</span> <span class="n">prngRandomNumber</span>
</code></pre></div></div>
<p>The resulting session Id is stored in the session store and looked up for each request. I <em>feel</em> this is a little more secure than just using a (Cryptographically secure) random number. Over engineered? May be. But I’ll err on the side of caution.</p>
<h2 id="security">Security</h2>
<p>Before I end this article, let’s briefly discuss security. Because <strong>session Id’s are usually portable</strong>, as a developer, you need to ensure that they are not easily obtainable by eavesdroppers or can be shared by mistake.</p>
<ol>
<li>Don’t send session Id’s unencrypted. Use HTTPS to encrypt all traffic end-to-end.</li>
<li>Don’t send session Id’s a URL parameter: your users can inadvertently share URL’s thus revealing their session Id. Also, the session Id’s will appear in the web server or application logs and will be visible to anyone who has access to logs.</li>
</ol>
<h2 id="bonus">Bonus</h2>
<p>Here is a small list of cryptographically secure number generators in popular languages:</p>
<ol>
<li><a href="https://docs.oracle.com/javase/8/docs/api/java/security/SecureRandom.html">Java: SecureRandom</a></li>
<li><a href="https://msdn.microsoft.com/en-us/library/system.security.cryptography.rngcryptoserviceprovider(v=vs.110).aspx">.NET: RNGCryptoServiceProvider</a></li>
<li><a href="https://www.npmjs.com/package/uid-safe">Node.js</a></li>
<li><a href="http://linux.die.net/man/4/urandom">*nix: /dev/urandom</a></li>
</ol>
<p>Update: 3/11/2017 - Removed references to MD5 thanks to <a href="https://twitter.com/HollyGraceful">HollyGraceful</a> for calling it out.</p>
Developing Sense of Ownership in Employees - Let Your People 'Own' It!2016-04-12T00:00:00+00:00https://codeahoy.com/2016/04/12/let-them-own-it<p>To me, management is:</p>
<ul>
<li>about hiring the right people</li>
<li>telling them <em>what</em> needs to get done and <em>why</em>,</li>
<li>giving them the tools they need and</li>
<li><strong>getting out of their way</strong>.</li>
</ul>
<p><!--more-->
This might sound extreme. I don’t mean managers should leave their employees on their own without any supervision or accountability. Not at all.</p>
<p>Even the brightest people need a support from time to time to put them back on the right track.</p>
<p>Managers are responsible for making sure their teams understand the <em>vision</em> and stay on track. For inviting them to in the <strong>goal-setting</strong> process and listening to their feedback. For <strong>challenging</strong> their employees to perform at their best. For holding the team <strong>accountable</strong> for their actions.</p>
<p>The problem starts when managers get in the way of their employees and interfere with their ability to perform their duties and later wonder why it is so hard to retain employees. They don’t let a sense of ownership developer in their employees because of their inability to <strong>delegate</strong> and thinking they are responsible for all <strong>decision</strong> making.</p>
<p>Almost all organizational structures put managers at the top of the hierarchy implying that they are the ones responsible for all decision making. And most managers behave that way. <a href="http://www.amazon.com/Peopleware-Productive-Projects-Second-Edition/dp/0932633439">DeMarco and Lister</a> provide details of their encounter with one such “manager”:</p>
<blockquote>
<p>one senior manager we encountered at a professional society meeting in London. He summed up his entire view of the subject with this statement: “Management is kicking ass.” This equates to the view that managers provide all the thinking and the people underneath them just carry out their bidding. Again, that might be workable for cheeseburger production, but not for any effort for which people do the work with their heads rather than their hands. Everyone in such an environment has got to have the brain in gear. You may be able to kick people to make them active, but not to make them creative, inventive, and thoughtful.</p>
</blockquote>
<p><strong><a href="/2017/02/08/committing-teamicide-by-micromanagement/">Micromanagement</a></strong> doesn’t work and is not productive. Get the right people on the bus and let them take ownership. Check-in regularly to make sure everyone understand where the ship is headed. Remove barriers and move heaven and earth to give the team anything they need to succeed.</p>
<p>Bad managers who believe their job is to “<em>be an intimidating alpha male and to crack the whip</em>” will see the productivity and quality go down in any job that requires people to use their imagination and brain power. Then they become defensive and start putting “processes” and “measurements” in place. The slow decline begins. People abandon ownership or adopt the “<em>Meh. Why should I care?</em>” attitude. Good people leave (physically or mentally) and they end up with yes men. The project is doomed. If it doesn’t fail, it will be of mediocre quality.</p>
<p>In <a href="http://www.amazon.com/Peopleware-Productive-Projects-Second-Edition/dp/0932633439">Peopleware</a>, DeMarco and Lister make a great point:</p>
<blockquote>
<p>You take no steps to defend yourself from the people you’ve put into positions of trust. And all the people under you are in positions of trust. A person you can’t trust with any autonomy is of no use to you.</p>
</blockquote>
<p>Good managers understand that the path to success starts by hiring the right people and developing a sense of ownership in them.</p>
What Are Code Reviews and How to Do Them Effectively2016-04-03T00:00:00+00:00https://codeahoy.com/2016/04/03/effective-code-reviews<h2 id="what-is-code-review">What is Code Review?</h2>
<p>Code review is the process or rather an <em>activity</em> in which code written by a developer is inspected by <em>other</em> developers to look for defects and improvements. In other words, developers work on their code and ask for one of their peers to review their changes before they are merged into the main codebase.</p>
<p>In the last few years, code reviews have become part of normal workflow for large and small teams, ensuring that every change gets looked by at by least one other person. They are an integral part of almost every large company like Microsoft, Google, and Amazon to name a few, where every line of code is reviewed and approved by developers before it is merged into the main codebase.</p>
<h2 id="what-to-look-for-in-code-reviews">What To Look For in Code Reviews</h2>
<p>It varies from team to team, but generally, a code reviewer should consider the following:</p>
<ul>
<li>Does the code meet the <em>requirements</em> that it’s addressing? Does it do what the developer intended it to do?</li>
<li>Does the overall <em>design</em> makes sense and fit with the rest of the architecture?</li>
<li>Are there general <em>defects</em> like race conditions (for concurrent code), edge cases, and other bugs that users might encounter?</li>
<li>Is the code <em>readable</em> and <em>maintainable</em>? Are future developers likely to struggle to understand what’s going on? Is it more complex than it needs to be?</li>
<li>Does the code have appropriate <em>Test</em> coverage? (<a href="/2016/07/05/unit-integration-and-end-to-end-tests-finding-the-right-balance/">Unit, Integration, or End to End tests</a>)</li>
<li>Does the code adhere to your <a href="/2016/05/22/effective-coding-standards/">coding standards or style</a>? This includes things like naming, comments, etc. Note: Automate style-checking as much as possible.</li>
</ul>
<!--more-->
<h2 id="why-do-code-reviews">Why do Code Reviews?</h2>
<p>Code reviews take time: the change is held up until it is reviewed by usually 2 people. So why do it?</p>
<p>When done correctly, code review is a proven technique that helps <strong>improve code quality</strong> and spread core <strong>knowledge</strong> across the team. It forces developers to hold themselves to a <strong>higher standard</strong> because they know that their code will be reviewed by their peers. It’s also a great tool for <strong>mentoring</strong> new team members on nuances of the code base. Or as <a href="https://www.amazon.com/Refactoring-Improving-Existing-Addison-Wesley-Technology-ebook/dp/B007WTFWJ6/" rel="nofollow">Martin Fowler</a> puts it:</p>
<blockquote>
<p>Code reviews help <strong>spread knowledge</strong> through a development team. Reviews help more experienced developers <strong>pass knowledge</strong> to less experienced people. They help more people understand more aspects of a large software system. They are also very important in writing <strong>clear code</strong>. My code may look clear to me, but not to my team. That’s inevitable–it’s very hard for people to put themselves in the shoes of someone unfamiliar with the things they are working on.</p>
</blockquote>
<p>So even though code reviews introduce extra time, it’s an <strong>excellent trade-off</strong> considering all the benefits that we get out of them. (You could even argue that they save time that’d be spent in future on bug fixes, direct knowledge sharing or paying technical debt as a result of unmaintainable code.) According to <a href="http://www.amazon.com/Code-Complete-Developer-Best-Practices-ebook/dp/B00JDMPOSY" rel="nofollow">Code Complete 2</a>, code reviews are very effective at detecting bugs and cites several case studies:</p>
<blockquote>
<ul>
<li>IBM’s 500,000 line Orbit project used 11 levels of inspections. It was delivered early and had only about <em>1 percent</em> of the errors that would normally be expected.</li>
<li>A study of an organization at AT&T with more than 200 people reported a 14 percent increase in productivity and a <em>90 percent decrease</em> in defects after the organization introduced reviews.</li>
<li>Jet Propulsion Laboratories estimates that it saves about $25,000 per inspection by finding and fixing defects at an early stage.</li>
</ul>
</blockquote>
<p>However, many teams struggle with effective code reviews. In fact, when done incorrectly, code reviews can be quite painful. In <strong>dysfunctional</strong> teams and organizations, it can quickly turn into a rather <strong>nasty</strong> experience for everyone involved:</p>
<ul>
<li>Code reviewers show off their skills - or sometime even get back at the author - by demanding that their pointless opinions be implemented, that would make absolutely no difference on the outcome.</li>
<li>Code reviews take a long time to complete introducing delays to feature releases and become annoying for the author to keep resolving merge conflicts.</li>
<li>Authors ignore review comments and get their ally to approve.</li>
<li>It creates friction between developers.</li>
</ul>
<h2 id="how-to-do-code-reviews">How to do Code Reviews</h2>
<p>Let’s look at some techniques that the authors of the pull request, reviewers and company management can use to do code reviews <em>effectively</em>.</p>
<h3 id="advice-to-management">Advice to Management</h3>
<p>Managers should make sure that everyone understands the goals and importance of code reviews. Unless code reviews are part of your culture, developers are not going to ask their peers to review their code. Make sure developers get enough time in their Sprints for code reviews.</p>
<p>Managers can also help by setting up the right tools and adapting the release workflow to make code reviews a mandatory activity. Source control management systems like GitHub and Gitlab have built-in support for code reviewers that allows comments on specific lines of code, blocking merge until required number of approvals have been granted, etc. You can also use 3rd party tools like Crucible from Atlassian.</p>
<p>Make sure developers have enough bandwidth in their Sprints to review code. Otherwise, code reviews can end up taking a long time to complete.</p>
<h3 id="everyone-remember-the-human">Everyone: Remember the Human</h3>
<p>In his book, <a href="http://www.amazon.com/Peer-Reviews-Software-Practical-Guide/dp/0201734850" rel="nofollow">Peer Reviews in Software: A Practical Guide</a>, Wiegers writes:</p>
<blockquote>
<p>The dynamics between the work product’s author and its reviewers are critical. The author must trust and respect the reviewers enough to be receptive to their comments. Similarly, the reviewers must show respect for the author’s talent and hard work. Reviewers should thoughtfully select the words they use to raise an issue, focusing on what they observed about the product. Saying, “I didn’t see where these variables were initialized” is likely to elicit a constructive response, whereas “You didn’t initialize these variables” might get the author’s hackles up.</p>
</blockquote>
<p>It is easy to become fixated on the code, but remember, there’s a human at the other end of the table (or computer). A human who has opinions. A human who is entitled to have an <em>ego</em>. Remember that there are many ways to solve a problem.</p>
<ul>
<li>Be humble. I have seen both highly productive reviews and very unproductive ones because someone decided to be a prick – don’t be a prick:-)</li>
<li>Make sure you have <strong>coding standards</strong> in place. Coding standards are a shared set of guidelines in an organization with buy-in from everyone. If you don’t have coding standards, then don’t let the discussion turn into a pissing contest over coding styles (opening braces ‘{‘ on the same line or the next!) If you run into a situation like that, take the discussion offline to your coding standards forum.</li>
<li>Learn to communicate well. You must be able to clearly express your ideas and reasons.</li>
<li>When it comes to dealing with <em>opinions</em>, reviewers and authors should seek to understand each other’s perspectives but shouldn’t get into a philosophical debate.</li>
</ul>
<h3 id="advice-to-reviewers">Advice to Reviewers</h3>
<ul>
<li>The author isn’t there to be a sitting duck. Remember the purpose is to not demonstrate who the better programmer is: it is finding defects and ensuring that the code is simple and maintainable (or whatever your objectives are.)</li>
<li>Leave <strong>actionable</strong> feedback.</li>
<li>Ask questions when you are not sure about something. Don’t make demands or statements which could sound accusatory. For example, don’t say: “You didn’t use the XYZ library here”. A better way would be to genuinely seek to understand the developer’s perspective: “What do you think about library XYZ and if it applies here?”.</li>
<li>Avoid “<em>why did you</em>”, “<em>why did you not</em>” style questions when possible. It could put people on the defensive. “Why did you make this a global variable?” could be better expressed as “I don’t understand why this needs to be a global variable. Can you explain?”.</li>
<li>Use <strong>we</strong> instead of <em>you</em>.</li>
<li>If you are providing a suggestion, call it out as such. For example, instead of saying: “<em>make the color one shade more neutral</em>”, you should say: “<em>(suggestion/opinion) we should make the color one shade more neutral</em>”.</li>
<li>Review quickly and block time for code reviews.</li>
<li>If the author addressed your comments or did something <em>great</em> that you noticed, <strong>tell them</strong>! Most code reviews focus on finding mistakes but you should offer your <em>appreciation</em> and <strong>encouragement</strong> if the developer did something great. This type of feedback is super encouraging to developers and goes a long way!</li>
</ul>
<p>Some of these things won’t work if they come off as rehearsed or said in a sarcastic tone. Treat the code review as you would a normal conversation. You are listening to another person and should genuinely seek to understand their perspective. Offer suggestions and tips when they are necessary. If the code is great, don’t be compelled to find something negative to say about it.</p>
<h2 id="advice-to-authors">Advice to Authors</h2>
<ul>
<li>Don’t take things personally. Remember that the villains are the defects or inadequacies in the code, not you. Recognize that you may be attached to your code and that it is normal. If you take pride in your work, that’s a good sign that you are someone who cares about the craft. Have just the right amount of ego – enough to trust and defend your ideas, that is <em>the ability to negotiate</em>.</li>
<li>Don’t create mega-gigantic pull requests (10+ files changed.) Ask for reviews often and keep your pull requests small. If you can’t avoid this, give heads up to others (ahead of time such as during Sprint Planning) and set up some time to describe your changes and requirements.</li>
<li><em>Describe your change</em>: provide a complete description and link to the JIRA ticket so that reviewers can understand the requirements.</li>
<li>Add the <em>right reviewers</em>. You want to get the <strong>best feedback</strong> so it is important that you add the right people. For example, if you are making a change in the React code, make sure you add people who are not only familiar with not only React, but also with that part of the code. If you are making a change in a microservice that’s owned by a different team, add them as reviewers. On instances, you might even need to assign <em>different people</em> to different parts of the PR.</li>
<li>To err is human. The reviewer is acting as a second set of eyes and could point out things that you might have overlooked. Questions are as valuable as concrete advice.</li>
<li>Ask specific questions. “Does it make more sense to move all these classes into their own package?”</li>
<li>Respond to all feedback, whether you agree or disagree.</li>
</ul>
<h2 id="code-review-example">Code Review Example</h2>
<p>Modern tools like GitHub, GitLab, Crucible and others have made the code review process easier than ever. Let’s look at a <a href="https://github.com/neovim/neovim/pull/18832" rel="nofollow">good code review example</a> I found on GitHub. The developer made some changes and opened a pull request (PR). The first thing to notice is the detailed and to the point commit message by the author describing the change:</p>
<p><img src="/img/blogs/codereview-github-commit-message-1.png" class="rounded mx-auto d-block" alt="GitHub PR Commit Message for Code Review" /></p>
<p>Next, we see that the code reviewer suggests some changes, and explicitly calls out “good ideas” to <strong>encourage</strong> the contributor.</p>
<p><img src="/img/blogs/codereview-github-commit-message-2.png" class="rounded mx-auto d-block" alt="GitHub PR Comment for Code Review" />
<img src="/img/blogs/codereview-github-commit-message-3.png" class="rounded mx-auto d-block" alt="GitHub PR Comment for Code Review, Good Idea" /></p>
<p>The reviewer then <strong>approves</strong> the PR and leaves this final message:</p>
<blockquote>
<p>It’s great to see this fixed, thank you! Nice explanation + commit message too.</p>
<p>Another improvement we’ve considered is to debounce the scrollback management #8959 (or ideally put it in a ring buffer, but I’m not sure if that can work with libvterm).</p>
</blockquote>
<h2 id="summary">Summary</h2>
<p>Code review is an excellent technique for improving software quality when done right. Code reviews not only find defects but also helps developers grow by getting them feedback on their code from other developers, spreading code knowledge throughout and are a useful tool for mentoring new team members. Code reviews are part of a healthy engineering culture.</p>
<p><em>Updated</em>: June 2022 to add Code Review example.</p>
<p class="message">
I would love to hear your feedback, comments, thoughts on conducting effective code reviews. Please leave a
comment below sharing your experience or anything that would add value to this article and its future readers.
</p>
Checked vs Unchecked Exceptions in Java. Why it's so Confusing2016-04-02T00:00:00+00:00https://codeahoy.com/java/2016/04/02/checked-vs-unchecked-exceptions-in-java<p>This blog post is intended for new Java developers. It starts with a historical perspective and a look at what motivated the design and creation of Java’s exception handling mechanism. It also explores the hotly debated <strong>checked vs unchecked exceptions</strong> debate with some personal insights.</p>
<p>Let’s start.</p>
<!--more-->
<h2 id="historical-perspective">Historical Perspective</h2>
<p>Back in the time of the “C” programming language, it was customary to return values such as -1 or NULL from functions to indicate errors. This was practical for small applications but didn’t scale well for larger applications - developers had to check and track every possible return value: a return value of 2 might indicate ”host is down” error in library A whereas in library B, it could mean ”illegal filename”. Although, developers tried to fix this and attempts were made to standardize error codes by setting <strong>global variables</strong>, but it didn’t help much.</p>
<p><img src="https://codeahoy.com/img/blogs/errors_in_c.jpg" alt="C_Global_Errors" /></p>
<p>James Gosling and other designers felt that a similar approach would go against the design goals of Java programming language. They wanted:</p>
<ol>
<li>a cleaner, <strong>robust</strong> and portable approach</li>
<li><strong>built-in</strong> language support for error checking and handling.</li>
</ol>
<p>Essentially, one of their main design goals was to build a language that’s <strong>robust</strong> and able to cope with errors during execution or at least <strong>recognize</strong> when going go wrong. This principle is at the core of Java’s error handling design, as we’ll see later. James Gosling explains in one of his <a href="https://www.artima.com/intv/solid.html">interviews</a>:</p>
<blockquote>
<p>One of the traditional things to screw up in C code is opening a data file to read. It’s semi-traditional in the C world to <strong>not check</strong> the return code, because you just know the file is there, right? So you just open the file and you read it. But someday months from now when your program is in deployment, some system administrator reconfigures files, and the file ends up in the wrong place. Your program goes to open the file. It’s not there, and the open call returns you an error code that you never check. You take this file descriptor and slap it into your file descriptor variable. The value happens to be -1, which isn’t very useful as a file descriptor, but it’s still an integer, right? So you’re still happily calling reads. And as far as you can tell, the world is all rosy, except the data just isn’t there.</p>
</blockquote>
<p>They didn’t have to look too far. The inspiration for handling errors came from a very popular language of the 60’s: LISP. Java’s exception handling was born and (for better or worse) the rest is history.</p>
<h2 id="exception-handling-in-java">Exception Handling in Java</h2>
<p>So what is exception handling? It is unconventional but a simple concept: if an error is encountered during program execution, halt the normal execution and transfer control to a section specified by the programmer. Let’s look at an example:</p>
<pre class="prettyprint lang-java">
try {
f = new File("list.txt"); // throw error if file is not found...
f.readLine;
f.write("another item for the list");
f.close();
} catch (FileNotFoundException fnfe) { // transfer control to this block on error.
// do something with the error.
// notify user or try reading another location, etc
}
</pre>
<p>In other words, exceptions are exceptional conditions which disrupt the normal program flow. Instead of executing the next instruction in the sequence, the control is transferred to the Java Virtual Machine (JVM) which tries to find an appropriate exception handler in the program and transfer control to it (hence disrupting the normal program flow). In the last example, if the file <code class="language-plaintext highlighter-rouge">list.txt</code> is not found, the control will be transferred to the <code class="language-plaintext highlighter-rouge">catch(...)</code> block instead of continuing to the next line in the <code class="language-plaintext highlighter-rouge">try</code> block. Here are some more examples where an exception can be thrown:</p>
<ul>
<li>Accessing index outside the bounds of an array</li>
<li>Disk is full</li>
<li>IP address for a host couldn’t be determined</li>
<li>Using a <code class="language-plaintext highlighter-rouge">null</code> value when an object is required</li>
<li>Divide by 0</li>
<li>Violation of defined contract: e.g. invalid values passed to a method</li>
</ul>
<h2 id="two-types-of-exceptions-in-java-checked-and-unchecked">Two Types of Exceptions in Java: Checked and Unchecked</h2>
<p>In Java, there are two types of exceptions: checked and unchecked. Let’s take a look at them.</p>
<h3 id="checked-exceptions">Checked Exceptions</h3>
<p>Checked exceptions are used to represent recoverable error conditions e.g. file not found. Java requires that these exceptions are explicitly handled by developers or the code won’t compile. According to <a href="http://docs.oracle.com/javase/tutorial/essential/exceptions/catchOrDeclare.html">official documentation</a>:</p>
<blockquote>
<p>These are exceptional conditions that a well-written application <strong>should anticipate and recover from</strong>. For example, suppose an application prompts a user for an input file name, [..] But sometimes the user supplies the name of a nonexistent file, and the constructor throws java.io.FileNotFoundException. A well-written program will catch this exception and notify the user of the mistake, possibly prompting for a corrected file name.</p>
</blockquote>
<p>Omitting to catch a checked exception will result in a compile time error:</p>
<pre class="prettyprint lang-java">
Main.java:8: Warning: Exception java.io.FileNotFoundException must be caught,
or it must be declared in throws clause of this method.
f = new FileInputStream(filename);
^
</pre>
<p>Developers have a few options on how to handle checked exceptions. All of these require an explicit acknowledgement of the exception and taking an action on it.</p>
<ol>
<li>catch and do something with the exception, or,</li>
<li>re-throw it and give the methods higher up in a call trace a chance to handle it, or,</li>
<li>catch the exception, wrap it up in a different exception and throw it up. This is called exception wrapping.</li>
</ol>
<p>Since checked exceptions represents problems from which a program may wish to recover, the designers of Java wanted to <strong>force</strong> developers to at least pay attention so that these do not get accidentally ignored. (and are handled as close to the source as possibe.) As James Gosling <a href="https://www.artima.com/intv/solid2.html">explained</a>:</p>
<blockquote>
<p>… because the knowledge of the situation is always fairly localized. When you try to open a file and it’s not there, you’re coping strategy is really determined by what you were going for. Some guy miles away isn’t going to know what to do. The code that tried to open the file knows what to do, whether it be trying a backup file, looking in a different directory, or asking the user for another filename.</p>
</blockquote>
<p>You can’t accidentally ignore a checked exception. If you do choose to ignore, you must do so explicitly.</p>
<pre class="prettyprint lang-java">
try {
Set set = ...
// code which throws checked exceptions
} catch (Exception e) {
// do nothing or "I don't care". Extremely bad practice.
}
</pre>
<p><strong>Beware</strong>: Do <em>NOT</em> do this in real-life. It’s an extremely bad practice. At the very least, log the exception.</p>
<h3 id="unchecked-exceptions">Unchecked exceptions</h3>
<p>The <a href="http://docs.oracle.com/javase/tutorial/essential/exceptions/catchOrDeclare.html">documentation</a> says:</p>
<blockquote>
<p>(unchecked exceptions) are exceptional conditions that are internal to the application, and that the application usually cannot anticipate or recover from. These usually indicate programming bugs, such as logic errors or improper use of an API.</p>
</blockquote>
<p>Unchecked exceptions (aka RuntimeExceptions) represent problems which happen during program execution e.g. divide by 0, accessing object method on <code class="language-plaintext highlighter-rouge">null</code> object reference, etc. Unlike checked exception, Java <strong>doesn’t require that we catch</strong> unchecked exceptions and the compiler won’t complain. These can happen anywhere in your program and our code will be littered with <code class="language-plaintext highlighter-rouge">try-catch</code> if we were catching every RuntimeException. In real life this would be equivalent of putting diesel into a gasoline car. You broke the contract and shouldn’t have done it. A mistake was made. The car comes to a grinding halt. There’s nothing you can do to fix it and get to your destination. You must to take the car to a dealer.</p>
<p><strong>Should you catch RuntimeExceptions?</strong></p>
<p>What’s the point of catching RuntimeExceptions if the condition is irrecoverable? These errors are usually preventable by fixing your code in the first place. For example, dividing a number by 0 generates RuntimeException (<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ArithmeticException.html">ArithmeticException</a>.) But you could avoid it by checking that the argument is greater than zero i.e. <code class="language-plaintext highlighter-rouge">denominator > 0</code>. If this condition is not true, halt further execution. (and possibly throw <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/IllegalArgumentException.html">IllegalArgumentException</a>!)</p>
<p>But there are situations when it’s alright to catch RuntimeExceptions, log the error and move on. A while ago, I worked on a high-throughput client that was processing thousands of transactions a second. If a transaction was malformed, the code will complain and throw RuntimeException. To prevent that from happening, we were checking every single transaction to catch and ignore any malformed transactions.</p>
<pre class="prettyprint lang-java">
boolean isMalformed(Transaction t) {
// Check transaction and return false if it's malformed; true otherwise
}
</pre>
<p>When we looked at the metrics, it showed that malformed transactions were super rare. 99.99% of the transactions were good, yet we were wasting precious CPU cycles testing every single transaction (including String comparison on some of the fields). So we removed the <code class="language-plaintext highlighter-rouge">isMalformed(...)</code> method and let the code throw RuntimeException, log it and move on to the next transaction. The result was improved application performance.</p>
<h2 id="checked-vs-unchecked-exceptions">Checked vs Unchecked Exceptions</h2>
<p>Many people find the dichotomy between checked and unchecked exceptions confusing and counter-intuitive. The core argument is whether or not a language should <strong>force</strong> developers to catch exceptions. Other languages like C++ and the more modern C#, left out checked exceptions and only support unchecked exceptions. If you want to read up on the debate, you can visit this <a href="https://stackoverflow.com/questions/613954/the-case-against-checked-exceptions">Stack Overflow question</a>.</p>
<p>Both sides have equally compelling arguments. Personally, I use checked exception when coding in Java but only judiciously and where it makes sense. I use them so that callers of my code don’t <strong>accidentally ignore</strong> something which they shouldn’t. And this is how it’s supposed to be done. I have worked with code where developers who didn’t like checked exceptions were <strong>disguising</strong> them as unchecked (creating all exception subclasses from <code class="language-plaintext highlighter-rouge">java.lang.RuntimeException</code>.) This is dangerous because some users of their code or library (other Java developers) who are expecting to only handle checked exceptions may not pay attention to unchecked and ignore things which they shouldn’t. On the other extreme, there are developers who just don’t understand when to use which type and force others to catch ‘checked’ exceptions which they cannot recover from, causing pain and displeasure. A developer I knew would Google search for similar built-in exception types and blindly use or extend from what he found without stopping and thinking whether he should be using checked or unchecked. And that I feel is the root cause behind the confusion and the debate: <em>checked exceptions aren’t bad. People just don’t pay attention and choose unwittingly</em>.</p>
<p>If you are in the camp of developers who don’t like checked exceptions and wish they go away, you should know that it’s not going to happen any time soon. Oracle is still <a href="https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html">promoting</a> the use of checked exceptions as is evident from any Java documentation. Joshua Bloch also supports the use of checked exceptions in his seminal book <a href="https://www.goodreads.com/book/show/105099.Effective_Java_Programming_Language_Guide">Effective Java</a>:</p>
<blockquote>
<p>Item 40: Use checked exceptions for recoverable conditions and runtime exceptions for programming errors. E.g. host is down, throw checked exception so the caller can either pass in a different host address or move on to something else.</p>
<p>Item 41: Avoid unnecessary use of checked exceptions.</p>
</blockquote>
<p>Good or bad, checked exceptions are here to stay and and while they are not perfect or even good, we should use them as intended when programming in Java. Enough said. Let’s move on take a look at the exception class hierarchy in Java.</p>
<h2 id="the-java-exception-class-hierarchy">The Java Exception Class Hierarchy</h2>
<p>All exceptions in Java have a common ancestor: <code class="language-plaintext highlighter-rouge">java.lang.Throwable</code>. The following base exception classes are most commonly used by Java developers:</p>
<h3 id="1-javalangexception-and-the-javalangruntimeexception">1. <code class="language-plaintext highlighter-rouge">java.lang.Exception</code> and the <code class="language-plaintext highlighter-rouge">java.lang.RuntimeException</code></h3>
<p>Any class extending from <code class="language-plaintext highlighter-rouge">java.lang.Exception</code> is classified as checked exception and must be declared in a method’s <code class="language-plaintext highlighter-rouge">throw</code> clause. Likewise, any class extending from <code class="language-plaintext highlighter-rouge">java.lang.RuntimeException</code> is classified as unchecked exception.</p>
<p>Confusingly enough, <code class="language-plaintext highlighter-rouge">java.lang.RuntimeException</code> is a child of <code class="language-plaintext highlighter-rouge">java.lang.Exception</code> but it is classified as unchecked exception. (Special case.)</p>
<h3 id="2-javalangerror">2. <code class="language-plaintext highlighter-rouge">java.lang.Error</code></h3>
<p><code class="language-plaintext highlighter-rouge">Error</code> and its subclasses are the second category of unchecked exceptions (the first one being the <code class="language-plaintext highlighter-rouge">RuntimeException</code>). These are used to indicates serious or abnormal problems e.g. disk failing while your application is in the process of writing to it or <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/VirtualMachineError.html">VirtualMachineError</a> etc. Your application cannot recover from these errors and for practical purposes, you’d <strong>never have to catch or worry about this type</strong>.</p>
<p>Here’s a look at the Java Exception class hierarchy visually:</p>
<p><img src="https://codeahoy.com/img/blogs/throwable_hierarchy.png" alt="Throwable_exception_hierarchy" /></p>
<p>That’s all folks! Hope you found this useful.</p>
<p>Here’s a more recent talk on the subject by <a href="https://en.wikipedia.org/wiki/Elliotte_Rusty_Harold">Elliotte Rusty Harold</a>, author of several Java books. I think he completely nails the issue that checked exceptions are not bad, it is just that <strong>“Sun forgot to tell anybody how to use them!”</strong>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/rJ-Ihh7RNao?start=418" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>