Los Techies http://feed.informer.com/digests/ZWDBOR7GBI/feeder Los Techies Respective post owners and feed distributors Thu, 08 Feb 2018 14:40:57 +0000 Feed Informer http://feed.informer.com/ Surprise Go is ok for me now https://lostechies.com/ryansvihla/2018/12/13/surprise-go-is-ok/ Los Techies urn:uuid:53abf2a3-23f2-5855-0e2d-81148fb908bf Thu, 13 Dec 2018 20:23:00 +0000 I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve. <p>I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve.</p> <p>About 7 months ago I wrote about all the things I didn’t really care for in Go and now I either no longer am so bothered by it or things have improved.</p> <p>Go Modules so far is a huge improvement over Dep and Glide for dependency management. It’s easy to setup, performant and eliminates the GOPATH silliness. I haven’t tried it yet with some of the goofier libraries that gave me problems in the past (k8s api for example) so the jury is out on that, but again pretty impressed. I now longer have to check in vendor to speed up builds. Lesson use Go Modules.</p> <p>I pretty much stopped using channels for everything but shutdown signals and that fits my preferences pretty well, I use mutex and semaphores for my multithreaded code and feel no guilt about it. This cut out a lot of pain for me, and with the excellent race detector I feel really comfortable writing multi-threaded in Go now. Lesson, don’t use channels much.</p> <p>Lack of generics still sometimes sucks but I usually implement some crappy casting with dynamic types if I need that. I’ve sorta made my piece with just writing more code, and am no longer so hung up. Lesson relax.</p> <p>Error handling I’m still struggling with, I thought about using one of the error Wrap() libraries but an official one is in draft spec now, so I’ll wait on that. I now tend to have less nesting of functions as a result, this probably means longer functions than I like, but my code looks more “normal” now. This is a trade off I’m ok with. Lesson relax more.</p> <p>I see the main virtue of Go now that it is very popular in the infrastructure space where I am and so it’s becoming the common tongue (largely replacing Python for those sorts of tasks). For this, honestly it’s about right. It’s easy to rip out command line tools and deploy binaries for every platform with no runtime install.</p> <p>The community’s conservative attitude I sort of view as a feature now, in that there isn’t a bunch of different options that are popular and there is no arguing over what file format is used. This drove me up the wall initially, but I appreciate how much less time I spend on these things now.</p> <p>So now I suspect Go will be my “last” programming language. It’s not the one I would have chosen, but where I am at in my career, where most of my dev work is automation and tooling it fits the bill pretty well.</p> <p>Also equally important most of the people working with me didn’t have full time careers as developers or spend their time reading “Domain Driven Design” (amazing book) so adding in a bunched of nuanced stuff that maybe technically optimal but assumes the reader grasps all that nuance isn’t a good tradeoff for me.</p> <p>So I think I sorta get it now. I’ll never be a cheerleader for the language but it definitely solves my problems well enough.</p> AutoMapper 8.0.0 Released https://jimmybogard.com/automapper-8-0-0-released/ Jimmy Bogard urn:uuid:573e408e-7080-27eb-c313-b09b86aaf7c2 Sat, 17 Nov 2018 14:44:27 +0000 <p>Today we released AutoMapper 8.0.0:</p> <ul> <li><a href="http://docs.automapper.org/en/stable/8.0-Upgrade-Guide.html">Upgrade Guide</a></li> <li><a href="https://github.com/AutoMapper/AutoMapper/releases/tag/v8.0.0">Release Notes</a></li> </ul> <p>AutoMapper 8.0 brings some breaking API changes, meant to simplify our configuration options which have grown quite a bit over time and remove some confusion about what configuration options were effectively equivalent. The upgrade guide walks through the</p> <p>Today we released AutoMapper 8.0.0:</p> <ul> <li><a href="http://docs.automapper.org/en/stable/8.0-Upgrade-Guide.html">Upgrade Guide</a></li> <li><a href="https://github.com/AutoMapper/AutoMapper/releases/tag/v8.0.0">Release Notes</a></li> </ul> <p>AutoMapper 8.0 brings some breaking API changes, meant to simplify our configuration options which have grown quite a bit over time and remove some confusion about what configuration options were effectively equivalent. The upgrade guide walks through the breaking changes.</p> <p>The motivation for breaking the API also came from some confusion around mapping configuration used for in-memory mappings and LINQ projections. With the API consolidation, we've unified the APIs and made explicitly clear which configuration can be used with <code>ProjectTo</code>, and which cannot.</p> <p>We've also added a new feature, <a href="http://docs.automapper.org/en/stable/Value-converters.html">Value Converters</a>, which allow you to define reusable mappers scoped to individual members.</p> <p>As well as many other little fixes :)</p> <p>Enjoy!</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Le4mCktQWhM:5DeyS6xbKb4:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Le4mCktQWhM:5DeyS6xbKb4:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Le4mCktQWhM:5DeyS6xbKb4:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Le4mCktQWhM:5DeyS6xbKb4:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Le4mCktQWhM:5DeyS6xbKb4:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/Le4mCktQWhM" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - Sagas https://jimmybogard.com/life-beyond-distributed-transactions-sagas/ Jimmy Bogard urn:uuid:5327777c-2ed9-631a-dc1b-47d4f1f39994 Wed, 12 Sep 2018 15:25:04 +0000 <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/">Failures and Retries</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatcher-failure-recovery/">Failure Recovery</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-sagas/">Sagas</a></li> </ul> <p><a href="https://github.com/jbogard/adventureworkscosmos">Sample code from this series</a></p> <p>So far in this series, we've looked at the ins and outs of moving beyond distributed transactions using persisted messages as a means of coordination between different</p> <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/">Failures and Retries</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatcher-failure-recovery/">Failure Recovery</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-sagas/">Sagas</a></li> </ul> <p><a href="https://github.com/jbogard/adventureworkscosmos">Sample code from this series</a></p> <p>So far in this series, we've looked at the ins and outs of moving beyond distributed transactions using persisted messages as a means of coordination between different documents (or resources). One common question in the example I give is "how do I actually make sure either both operations happen or neither?" To answer this question, we need to recognize that this "all-or-nothing" approach is a kind of transaction. But we've already said we're trying to avoid distributed transactions!</p> <p>We won't be building a new kind of distributed transaction, but instead one that lives longer than any one single request, or a long-lived transaction. To implement a long-lived transaction, we need to look at the Saga pattern, first described in <a href="https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf">the original Sagas paper (Molina, Salem)</a>. The most common example of a saga I've seen described is booking a vacation. When you book a vacation, you need to:</p> <ul> <li>Book a flight</li> <li>Book a hotel</li> <li>Reserve a car</li> </ul> <p>You can't do all three at once - that's like getting a conference call together with all companies and getting consensus altogether. Not going to happen! Instead, we build this overall business transaction as a series of requests and compensating actions in case something goes wrong:</p> <ul> <li>Cancel flight</li> <li>Cancel hotel</li> <li>Cancel car reservation</li> </ul> <p>Our saga operations can be linear (<a href="https://lostechies.com/jimmybogard/2013/03/14/saga-implementation-patterns-controller/">controller pattern</a>) or parallel (<a href="https://lostechies.com/jimmybogard/2013/03/11/saga-implementation-patterns-observer/">observer pattern</a>) or in microservices terms, <a href="https://microservices.io/patterns/data/saga.html">orchestration/choreography</a>.</p> <p>In order to satisfy our saga constraints, our requests must:</p> <ul> <li>Be idempotent</li> <li>Can abort</li> </ul> <p>And the compensating requests must:</p> <ul> <li>Be idempotent</li> <li>Cannot abort</li> <li>Be commutative</li> </ul> <p>In our example, we have a model that basically assumes success. We:</p> <ul> <li>Approve an order</li> <li>Deduct stock</li> </ul> <p>Let's modify this a bit to create an order fulfillment saga. For this saga, we can fulfill an order if and only if:</p> <ul> <li>Our order is approved</li> <li>We have enough stock</li> </ul> <p>If our order is rejected, we need to release the stock. If we don't have enough stock, we need to un-approve (reject) our order. And keeping with our example, we need something to coordinate this activity - our saga. But rather than just call it some generic "saga", let's give it a meaningful name - <code>OrderFulfillmentSaga</code>:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/8/2018/Picture0055.png" alt=""></p> <p>This saga will coordinate the activities of the order request and stock. And because we need this saga to have the same communication properties of the other two documents, we can simply model this saga as just another document with our inbox/outbox!</p> <p>The overall flow will be:</p> <ul> <li>Once an order is created, kick off a new order fulfillment saga</li> <li>This saga will coordinate actions between the stock and order request</li> <li>If the order is rejected, the saga needs to initiate a stock return</li> <li>If there is not enough stock, the saga needs to cancel the order</li> </ul> <p>Let's start with kicking off the saga!</p> <h3 id="kickingthingsoff">Kicking things off</h3> <p>When should we kick the saga off? It's tempting to do this in the initial request that creates a new order request, but remember - we can't put more than one document in a transaction unless we're very sure fulfillment and order requests will live close together in our Cosmos DB instance. That means we need to use our document messages to communicate with a saga - even if it doesn't exist!</p> <p>We don't want to fulfill an order request twice, and for our simple scenario let's just assume an order request can't be "retried". Originally, our <code>OrderRequest</code> created an <code>ItemPurchased</code> document message for each item - we'll remove that in favor of a single <code>OrderCreated</code> document message:</p> <pre><code class="language-c#">public class OrderCreated : IDocumentMessage { public Guid Id { get; set; } public Guid OrderId { get; set; } public List&lt;LineItem&gt; LineItems { get; set; } public class LineItem { public int ProductId { get; set; } public int Quantity { get; set; } } } </code></pre> <p>We <em>could</em> just have the <code>OrderId</code> and have the receiver then load up the <code>OrderRequest</code>, but for simplicity sake (and assuming you can't change the order after created), we'll treat this information as immutable and keep it in the message. Now when we create an <code>OrderRequest</code>, we'll also send this message:</p> <pre><code class="language-c#">public class OrderRequest : DocumentBase { public OrderRequest(ShoppingCart cart) { Id = Guid.NewGuid(); Customer = new Customer { FirstName = "Jane", MiddleName = "Mary", LastName = "Doe" }; Items = cart.Items.Select(li =&gt; new LineItem { ProductId = li.Key, Quantity = li.Value.Quantity, ListPrice = li.Value.ListPrice, ProductName = li.Value.ProductName }).ToList(); Status = Status.New; Send(new OrderCreated { Id = Guid.NewGuid(), OrderId = Id, LineItems = Items .Select(item =&gt; new OrderCreated.LineItem { ProductId = item.ProductId, Quantity = item.Quantity }) .ToList() }); } </code></pre> <p>It's not much different than our original order creation - we're just now including the document message to initiate the order fulfillment saga.</p> <p>Our handler for this document message needs to find the right <code>OrderFulfillment</code> saga document and let the saga handle the message:</p> <pre><code class="language-c#">public class OrderCreatedHandler : IDocumentMessageHandler&lt;OrderCreated&gt; { private readonly IDocumentDBRepository&lt;OrderFulfillment&gt; _repository; public OrderCreatedHandler(IDocumentDBRepository&lt;OrderFulfillment&gt; repository) =&gt; _repository = repository; public async Task Handle(OrderCreated message) { var orderFulfillment = (await _repository .GetItemsAsync(s =&gt; s.OrderId == message.OrderId)) .FirstOrDefault(); if (orderFulfillment == null) { orderFulfillment = new OrderFulfillment { Id = Guid.NewGuid(), OrderId = message.OrderId }; await _repository.CreateItemAsync(orderFulfillment); } orderFulfillment.Handle(message); await _repository.UpdateItemAsync(orderFulfillment); } } </code></pre> <p>Not shown here - but we do need to make sure we only have a single fulfillment saga per order, so we can configure inside Cosmos DB <code>OrderId</code> as a unique index.</p> <p>The <code>orderFulfillment.Handle</code> method needs to start, and request stock:</p> <pre><code class="language-c#">public void Handle(OrderCreated message) { Process(message, m =&gt; { if (IsCancelled) return; LineItems = m.LineItems .Select(li =&gt; new LineItem { ProductId = li.ProductId, AmountRequested = li.Quantity }) .ToList(); foreach (var lineItem in LineItems) { Send(new StockRequest { Id = Guid.NewGuid(), ProductId = lineItem.ProductId, AmountRequested = lineItem.AmountRequested, OrderFulfillmentId = Id }); } }); } </code></pre> <p>In my example, I've made the <code>OrderFulfillment</code> saga coordinate with <code>Stock</code> with our <code>StockRequest</code>. This is instead of <code>Stock</code> listening for <code>OrderCreated</code> itself. My general thought here is that fulfillment manages the requests/returns for stock, and any business logic around that.</p> <p>I also have a little check at the beginning - if an order is cancelled, we don't want to send out stock requests. This is the piece that's enforcing commutative requests - we might receive an order rejected notice <em>before</em> receiving the order created notice! When it comes to messaging, I always assume messages are received out of order, which means our business logic needs to be able to handle these situations.</p> <h3 id="handlingstockrequests">Handling stock requests</h3> <p>Our original <code>Stock</code> implementation was quite naive, but this time we want to more intelligently handle orders. In our stock handler, we'll still have a document per product, but now it can make a decision based on the quantity available:</p> <pre><code class="language-c#">public void Handle(StockRequest message) { Process(message, e =&gt; { if (QuantityAvailable &gt;= message.AmountRequested) { QuantityAvailable -= e.AmountRequested; Send(new StockRequestConfirmed { Id = Guid.NewGuid(), OrderFulfillmentId = e.OrderFulfillmentId, ProductId = ProductId }); } else { Send(new StockRequestDenied { Id = Guid.NewGuid(), OrderFulfillmentId = e.OrderFulfillmentId, ProductId = ProductId }); } }); } </code></pre> <p>Because we're using document messages with our inbox de-duping messages, we don't need to worry about processing the stock request twice. Our simple logic just checks the stock, and if it's successful we can deduct the stock and return a <code>StockRequestConfirmed</code> message. If not, we can return a <code>StockRequestDenied</code> message.</p> <h3 id="asuccessfulorderfulfillment">A successful order fulfillment</h3> <p>Our original logic said that "an order can be fulfilled if the order is approved and we have enough stock". Approving an order is a human decision, so we have a basic form for doing so:</p> <pre><code class="language-c#">@if (Model.Order.Status == Status.New) { &lt;form asp-controller="Order" asp-action="Reject" asp-route-id="@Model.Order.Id" method="post"&gt; &lt;input type="submit" value="Reject"/&gt; &lt;/form&gt; &lt;form asp-controller="Order" asp-action="Approve" asp-route-id="@Model.Order.Id" method="post"&gt; &lt;input type="submit" value="Approve"/&gt; &lt;/form&gt; } </code></pre> <p>And when the order is approved, we just delegate to MediatR to handle this request:</p> <pre><code class="language-c#">public class ApproveOrder { public class Request : IRequest { public Guid Id { get; set; } } public class Handler : IRequestHandler&lt;Request&gt; { private readonly IDocumentDBRepository&lt;OrderRequest&gt; _orderRepository; public Handler(IDocumentDBRepository&lt;OrderRequest&gt; orderRepository) { _orderRepository = orderRepository; } public async Task&lt;Unit&gt; Handle(Request request, CancellationToken cancellationToken) { var orderRequest = await _orderRepository.GetItemAsync(request.Id); orderRequest.Approve(); await _orderRepository.UpdateItemAsync(orderRequest); return Unit.Value; } } } </code></pre> <p>Which then delegates to our document to approve the order request:</p> <pre><code class="language-c#">public void Approve() { if (Status == Status.Approved) return; if (Status == Status.Rejected) throw new InvalidOperationException("Cannot approve a rejected order."); Status = Status.Approved; Send(new OrderApproved { Id = Guid.NewGuid(), OrderId = Id }); } </code></pre> <p>We only want to send out the <code>OrderApproved</code> message once, so just some basic status checking handles that.</p> <p>On the order fulfillment side:</p> <pre><code class="language-c#">public void Handle(OrderApproved message) { Process(message, m =&gt; { OrderApproved = true; if (IsCancelled) { ProcessCancellation(); } else { CheckForSuccess(); } }); } </code></pre> <p>Each time we receive some external notification, we need to process the success/failure path, which I'll come back to in a bit. Our handler for <code>StockRequestConfirmed</code> will be similar, except we're tracking stock on a line item by line item basis:</p> <pre><code class="language-c#">public void Handle(StockRequestConfirmed message) { Process(message, m =&gt; { var lineItem = LineItems.Single(li =&gt; li.ProductId == m.ProductId); lineItem.StockConfirmed = true; if (IsCancelled) { ReturnStock(lineItem); } else { CheckForSuccess(); } }); } </code></pre> <p>The <code>CheckForSuccess</code> method will look to see if all the order fulfillment requirements are met:</p> <pre><code class="language-c#">private void CheckForSuccess() { if (IsCancelled) return; if (LineItems.All(li =&gt; li.StockConfirmed) &amp;&amp; OrderApproved) { Send(new OrderFulfillmentSuccessful { Id = Guid.NewGuid(), OrderId = OrderId }); } } </code></pre> <p>Only if all of our stock has been confirmed and our order has been approved will we send a message back to the <code>Order</code> document to then finally complete the order:</p> <pre><code class="language-c#">public void Handle(OrderFulfillmentSuccessful message) { Process(message, m =&gt; { if (Status == Status.Rejected || Status == Status.Cancelled) return; Status = Status.Completed; }); } </code></pre> <p>The overall message flow looks something like this:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/8/2018/Picture0056.png" alt=""></p> <p>For each step along the way, we've got idempotency handled for us by the inbox/outbox structures. However, we still need to handle out-of-order messages, which is why you'll see success/fail checks every time we receive a notification.</p> <p>Now that we've got the success path taken care of, let's look at the failure paths.</p> <h3 id="cancellingtheorderfulfillment">Cancelling the order fulfillment</h3> <p>The first way our order fulfillment can be cancelled is if an order is rejected. From the web app, our <code>Order</code> document handles a rejection:</p> <pre><code class="language-c#">public void Reject() { if (Status == Status.Rejected) return; if (Status == Status.Approved) throw new InvalidOperationException("Cannot reject an approved order."); if (Status == Status.Approved) throw new InvalidOperationException("Cannot reject a completed order."); Status = Status.Rejected; Send(new OrderRejected { Id = Guid.NewGuid(), OrderId = Id }); } </code></pre> <p>Our order sends an <code>OrderRejected</code> document message that our order fulfillment document receives:</p> <pre><code class="language-c#">public void Handle(OrderRejected message) { Process(message, m =&gt; { OrderRejected = true; Cancel(); }); } </code></pre> <p>The <code>Cancel</code> method marks the order fulfillment as cancelled and then processes the cancellation:</p> <pre><code class="language-c#">private void Cancel() { IsCancelled = true; ProcessCancellation(); } </code></pre> <p>Similarly, a notification of <code>StockRequestDenied</code> will cancel the order fulfillment:</p> <pre><code class="language-c#">public void Handle(StockRequestDenied message) { Process(message, m =&gt; { Cancel(); }); } </code></pre> <p>In order to process our order fulfillment cancellation, we need to do a couple of things. First, we need to notify our <code>Order</code> document that it needs to be cancelled. And for any <code>Stock</code> items that were fulfilled, we need to return that stock:</p> <pre><code class="language-c#">private void ProcessCancellation() { if (!CancelOrderRequested &amp;&amp; !OrderRejected) { CancelOrderRequested = true; Send(new CancelOrderRequest { Id = Guid.NewGuid(), OrderId = OrderId }); } foreach (var lineItem in LineItems.Where(li =&gt; li.StockConfirmed)) { ReturnStock(lineItem); } } </code></pre> <p>Each step along the way, we keep track of what messages we've sent out so that we don't send notifications twice. To return stock:</p> <pre><code class="language-c#">private void ReturnStock(LineItem lineItem) { if (lineItem.StockReturnRequested) return; lineItem.StockReturnRequested = true; Send(new StockReturnRequested { Id = Guid.NewGuid(), ProductId = lineItem.ProductId, AmountToReturn = lineItem.AmountRequested }); } </code></pre> <p>If stock item has already had a return requested, we just skip it. Finally, the order can receive the cancel order request:</p> <pre><code class="language-c#">public void Handle(CancelOrderRequest message) { Process(message, m =&gt; { if (Status == Status.Rejected) return; Status = Status.Cancelled; }); } </code></pre> <p>With our failure flow in place, the message flows looks something like:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/8/2018/Picture0057.png" alt=""></p> <p>Our order fulfillment saga can now handle the complex process of managing stock and order approvals, keeping track of each step along the way and dealing with success/failure when it receives the notifications. It handles idempotency, retries, and commutative/out-of-order messages.</p> <p>In the next post, we'll look at how we can implement the inbox/outbox pattern for other resources, allowing us to bridge to other kinds of databases where a distributed transaction is just plain impossible.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=PZ0sOaalWiM:8ktwFvqIFtg:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=PZ0sOaalWiM:8ktwFvqIFtg:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=PZ0sOaalWiM:8ktwFvqIFtg:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=PZ0sOaalWiM:8ktwFvqIFtg:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=PZ0sOaalWiM:8ktwFvqIFtg:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/PZ0sOaalWiM" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - Dispatcher Failure Recovery https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatcher-failure-recovery/ Jimmy Bogard urn:uuid:1e3616c3-afbb-7454-4196-f04d76ad6fa6 Thu, 30 Aug 2018 13:15:14 +0000 <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/">Failures and Retries</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatcher-failure-recovery/">Failure Recovery</a></li> </ul> <p><a href="https://github.com/jbogard/adventureworkscosmos">Sample code from this series</a></p> <p>In the last post, we looked at how we can recover from exceptions from <em>inside</em> our code handling messages. We perform some action in our document, and something</p> <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/">Failures and Retries</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatcher-failure-recovery/">Failure Recovery</a></li> </ul> <p><a href="https://github.com/jbogard/adventureworkscosmos">Sample code from this series</a></p> <p>In the last post, we looked at how we can recover from exceptions from <em>inside</em> our code handling messages. We perform some action in our document, and something goes wrong. But what happens when something goes wrong <em>during</em> the dispatch process:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0054.png" alt=""></p> <p>If our dispatcher itself fails, either:</p> <ul> <li>Pulling a message from the outbox</li> <li>Sending a message to the receiver</li> <li>Saving documents</li> </ul> <p>Then our documents are still consistent, but we've lost the execution flow to dispatch. We've mitigated our failures somewhat, but we still can have the possibility of some unrecoverable failure in our dispatcher, and no amount of exception handling can prevent a document with a message sitting in its outbox, waiting for processing.</p> <p>If we were able to wrap everything, documents and queues, in a transaction, then we could gracefully recover. However, the point of this series is that we <em>don't</em> have access to distributed transactions, so that option is out.</p> <p>What we need is some kind of background process looking for documents with pending messages in their outbox, ready to process.</p> <h3 id="designingthedispatchrescuer">Designing the dispatch rescuer</h3> <p>We already have the dispatch recovery process using async messaging to retry an individual dispatch, which works great when the failure is in application code. When the failure is environmental, our only clue something is wrong is a document with messages in their outbox.</p> <p>The general process to recover from these failures would be:</p> <ul> <li>Find any documents with unprocessed messages (ideally oldest first)</li> <li>Retry them one at a time</li> </ul> <p>We have the possibility though that we have:</p> <ul> <li>In flight dispatching</li> <li>In flight retries</li> </ul> <p>Ideally, we have some sort of lagging processor, so that when we have issues, we don't interfere with normal processing. Luckily for us, Cosmos DB already comes with the ability to be notified that documents have been changed, the <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed">Change Feed</a>, and this change feed even lets us work with a built-in delay. After each document changes, we can wait some amount of time where we assume that dispatching happened, and re-check the document to make sure dispatching occurred.</p> <p>Our rescuer will:</p> <ul> <li>Get notified when a document changes</li> <li>Check to see if there are still outbox messages to process</li> <li>Send a message to reprocess that document</li> </ul> <p>It's somewhat naive, as we'll get notified for all document changes. To make our lives a little bit easier, we can turn off immediate processing for document message dispatching and just dispatch through asynchronous processes, but it's not necessary.</p> <h3 id="creatingthechangefeedprocessor">Creating the change feed processor</h3> <p>Using the <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed#using-the-change-feed-processor-library">documentation as our guide</a>, we need to create two components:</p> <ul> <li>A document feed observer to receive document change notifications</li> <li>A change feed processor to host and invoke our observer</li> </ul> <p>Since we already have a background processor in our dispatcher, we can simply host the observer in the same endpoint. The observer won't actually be doing the work, however - we'll still send a message out to process the document. This is because NServiceBus still provides all the logic around retries and poison messages that I don't want to code again. Like most of my integrations, I kick out the work into a durable message and NServiceBus as quickly as possible.</p> <p>That makes my observer pretty small:</p> <pre><code class="language-c#">public class DocumentFeedObserver&lt;T&gt; : IChangeFeedObserver where T : DocumentBase { static ILog log = LogManager.GetLogger&lt;DocumentFeedObserver&lt;T&gt;&gt;(); public Task OpenAsync(IChangeFeedObserverContext context) =&gt; Task.CompletedTask; public Task CloseAsync( IChangeFeedObserverContext context, ChangeFeedObserverCloseReason reason) =&gt; Task.CompletedTask; public async Task ProcessChangesAsync( IChangeFeedObserverContext context, IReadOnlyList&lt;Document&gt; docs, CancellationToken cancellationToken) { foreach (var doc in docs) { log.Info($"Processing changes for document {doc.Id}"); var item = (dynamic)doc; if (item.Outbox.Count &gt; 0) { ProcessDocumentMessages message = ProcessDocumentMessages.New&lt;T&gt;(item); await Program.Endpoint.SendLocal(message); } } } } </code></pre> <p>The <code>OpenAsync</code> and <code>CloseAsync</code> methods won't do anything, all my logic is in the <code>ProcessChangesAsync</code> method. In that method, I get a collection of changed documents. I made my <code>DocumentChangeObserver</code> generic because each observer observes only one collection, so I have to create distinct observer instances per concrete <code>DocumentBase</code> type.</p> <p>In the method, I loop over all the documents passed in and look to see if the document has any messages in the outbox. If so, I'll create a new <code>ProcessDocumentMessages</code> to send to myself (as I'm also hosting NServiceBus in this application), which will then process the document messages.</p> <p>With our simple observer in place, we need to incorporate the observer in our application startup.</p> <h3 id="configuringtheobserver">Configuring the observer</h3> <p>For our observer, we have a couple of choices on how we want to process document changes. Because our observer will get called for <em>every</em> document change, we want to be careful about the work it does.</p> <p>Our original design had document messages dispatched in the same request as the original work. If we keep this, we want to make sure that we minimize the amount of rework for a document with messages. Ideally, our observer only kicks out messages when there is truly something wrong with dispatching. This will also minimize the amount of queue messages, reserving them for the error case.</p> <p>So a simple solution would be to just introduce some delay in our processing:</p> <pre><code class="language-c#">private static ChangeFeedProcessorBuilder CreateBuilder&lt;T&gt;(DocumentClient client) where T : DocumentBase { var builder = new ChangeFeedProcessorBuilder(); var uri = new Uri(CosmosUrl); var dbClient = new ChangeFeedDocumentClient(client); builder .WithHostName(HostName) .WithFeedCollection(new DocumentCollectionInfo { DatabaseName = typeof(T).Name, CollectionName = "Items", Uri = uri, MasterKey = CosmosKey }) .WithLeaseCollection(new DocumentCollectionInfo { DatabaseName = typeof(T).Name, CollectionName = "Leases", Uri = uri, MasterKey = CosmosKey }) .WithProcessorOptions(new ChangeFeedProcessorOptions { FeedPollDelay = TimeSpan.FromSeconds(15), }) .WithFeedDocumentClient(dbClient) .WithLeaseDocumentClient(dbClient) .WithObserver&lt;DocumentFeedObserver&lt;T&gt;&gt;(); return builder; } </code></pre> <p>The <code>ChangeFeedProcessorBuilder</code> is configured for every document type we want to observe, with a timespan in this example of 15 seconds. I could bump this up a bit - say to an hour or so. It will really depend on the business, the SLAs they expect for work to complete.</p> <p>Finally, in our application startup, we need to create the builder, processor, and start it all up:</p> <pre><code class="language-c#">Endpoint = await NServiceBus.Endpoint.Start(endpointConfiguration) .ConfigureAwait(false); var builder = CreateBuilder&lt;OrderRequest&gt;(client); var processor = await builder.BuildAsync(); await processor.StartAsync(); Console.WriteLine("Press any key to exit"); Console.ReadKey(); await Endpoint.Stop() .ConfigureAwait(false); await processor.StopAsync(); </code></pre> <p>With this in place, we can have a final guard against failures, assuming that someone completely pulled the plug on our application and all we have left is a document with messages sitting in its outbox.</p> <p>In our next post, we'll look at using sagas to coordinate changes between documents - what happens if we want either all, or none of our changes to be processed in our documents?</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=1XV3c9D3P0s:t99uu28uzuY:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=1XV3c9D3P0s:t99uu28uzuY:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=1XV3c9D3P0s:t99uu28uzuY:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=1XV3c9D3P0s:t99uu28uzuY:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=1XV3c9D3P0s:t99uu28uzuY:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/1XV3c9D3P0s" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - Failures and Retries https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/ Jimmy Bogard urn:uuid:33b3ae6e-4542-449e-b9d9-465a738e17cf Thu, 16 Aug 2018 15:05:49 +0000 <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/">Failures and Retries</a></li> </ul> <p>In the last post, we looked at an example of dispatching document messages to other documents using a central dispatcher. Our example worked well in the happy path scenario, but what happens when something goes</p> <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-failures-and-retries/">Failures and Retries</a></li> </ul> <p>In the last post, we looked at an example of dispatching document messages to other documents using a central dispatcher. Our example worked well in the happy path scenario, but what happens when something goes wrong? We of course do not want a failure in dispatching messages to make the entire request fail, but what would that mean for us?</p> <p>We described a general solution to put the dispatching work aside using queues and messaging, effectively saying "yes, dispatching failed, so let's put it aside to look at in the future". This would allow the overall main request to complete:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0051.png" alt=""></p> <p>Our original example also assumed that we would dispatch our messages <em>immediately</em> in the context of the same request, which isn't a bad default but maybe isn't always desirable. Let's first look at the scenario of dispatching immediately, and what failures could mean.</p> <h3 id="characterizingourfailures">Characterizing our failures</h3> <p>Dispatching failures could happen for a number of reasons, but I generally see a continuum:</p> <ul> <li>Transient</li> <li>Delayed</li> <li>Permanent</li> </ul> <p>My failures usually have some sort of time component associated with them. A transient failure may mean that if I simply try the action again immediately, it may work. This most often comes up with some sort of concurrency violation against the database.</p> <p>Delayed failures are a bit different, where I won't succeed if I try immediately, but I will if I just wait some amount of time.</p> <p>Permanent failures mean there's an unrecoverable failure, and no amount of retries will allow the operation to succeed.</p> <p>Of course, we could simply ignore failures, but our business and customers might not agree with that approach. How might we handle each of these kinds of failures?</p> <h3 id="transientfailures">Transient failures</h3> <p>If something goes wrong, can we simply retry the operation? That seems fairly straightforward - but we don't want to retry <em>too</em> many times. We can implement some simple policies, either with a hardcoded number of retries or using something like the <a href="http://www.thepollyproject.org/">Polly Project</a> to retry an action.</p> <p>To keep things simple, we can have a policy to address the most common transient failure - optimistic concurrency problems. We first want to <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/faq#how-does-the-sql-api-provide-concurrency">enable OCC checks</a>, of course:</p> <pre><code class="language-c#">public async Task&lt;Document&gt; UpdateItemAsync(T item) { var ac = new AccessCondition { Condition = item.ETag, Type = AccessConditionType.IfMatch }; return await _client.ReplaceDocumentAsync( UriFactory.CreateDocumentUri(DatabaseId, CollectionId, item.Id.ToString()), item, new RequestOptions { AccessCondition = ac }); } </code></pre> <p>When we get a concurrency violation, this results in a <code>DocumentClientException</code> with a special status code (hooray HTTP!). We need some way to wrap our request and retry if necessary - time for another MediatR behavior! This one will retry our action some number of times:</p> <pre><code class="language-c#">public class RetryUnitOfWorkBehavior&lt;TRequest, TResponse&gt; : IPipelineBehavior&lt;TRequest, TResponse&gt; { private readonly IUnitOfWork _unitOfWork; public RetryUnitOfWorkBehavior(IUnitOfWork unitOfWork) =&gt; _unitOfWork = unitOfWork; public Task&lt;TResponse&gt; Handle( TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate&lt;TResponse&gt; next) { var retryCount = 0; while (true) { try { return next(); } catch (DocumentClientException e) { if (e.StatusCode != HttpStatusCode.PreconditionFailed) throw; if (retryCount &gt;= 5) throw; _unitOfWork.Reset(); retryCount++; } } } } </code></pre> <p>If our action files due to a concurrency problem, we need to clear out our unit of work's identity map and try again:</p> <pre><code class="language-c#">public void Reset() { _identityMap.Clear(); } </code></pre> <p>Then we just need to register our behavior with the container like our original unit of work behavior, and we're set. We could have of course modified our original behavior to add retries - but I want to keep them separate because they truly are different concerns.</p> <p>That works for immediate failures, but we still haven't looked at failures in our message dispatching. For that, we'll need to involve some messaging.</p> <h3 id="deferreddispatchingwithmessagingandnservicebus">Deferred dispatching with messaging and NServiceBus</h3> <p>The immediate retries can take care of transient failures during a request, but if the there's some deeper issue, we want to defer the document message dispatching to some time in the future. To make my life easier, and not have to implement half the <a href="https://www.enterpriseintegrationpatterns.com/">Enterprise Integration Patterns book</a> myself, I'll leverage <a href="https://particular.net/nservicebus">NServiceBus</a> to manage my messaging.</p> <p>Our original dispatcher looped through our unit of work's identity map to find documents with messages that need dispatching. We'll want to augment that behavior to catch any failures, and dispatch those messages offline:</p> <pre><code class="language-c#">public interface IOfflineDispatcher { Task DispatchOffline(DocumentBase document); } </code></pre> <p>Our <code>Complete</code> method of the unit of work will now take these failed dispatches and instruct our offline dispatcher to dispatch these offline:</p> <pre><code class="language-c#">public async Task Complete() { var toSkip = new HashSet&lt;DocumentBase&gt;(DocumentBaseEqualityComparer.Instance); while (_identityMap .Except(toSkip, DocumentBaseEqualityComparer.Instance) .Any(a =&gt; a.Outbox.Any())) { var document = _identityMap .Except(toSkip, DocumentBaseEqualityComparer.Instance) .FirstOrDefault(a =&gt; a.Outbox.Any()); if (document == null) continue; var ex = await _dispatcher.Dispatch(document); if (ex != null) { toSkip.Add(document); await _offlineDispatcher.DispatchOffline(document); } } } </code></pre> <p>This is a somewhat naive implementation - it doesn't allow for partial document message processing. If a document has 3 messages, we mark the retry the entire document instead of an individual message at at time. We could manage this more granularly, by including a "retry" collection on our document. But this introduces more issues - we could still have some system failure after dispatch and our document message never make it to retry.</p> <p>When our transaction scope is individual operations instead of the entire request, we have to assume failure at <em>every</em> instance and examine what might go wrong.</p> <p>The offline dispatcher uses NServiceBus to send a durable message out:</p> <pre><code class="language-c#">public class UniformSessionOfflineDispatcher : IOfflineDispatcher { private readonly IUniformSession _uniformSession; public UniformSessionOfflineDispatcher(IUniformSession uniformSession) =&gt; _uniformSession = uniformSession; public Task DispatchOffline(DocumentBase document) =&gt; _uniformSession.Send(ProcessDocumentMessages.New(document)); } </code></pre> <p>The <a href="https://docs.particular.net/nservicebus/messaging/uniformsession"><code>IUniformSession</code> piece</a> from NServiceBus to send a message from any context (in a web application, backend service, etc.). Our message just includes the document ID and type:</p> <pre><code class="language-c#">public class ProcessDocumentMessages : ICommand { public Guid DocumentId { get; set; } public string DocumentType { get; set; } // For NSB public ProcessDocumentMessages() { } private ProcessDocumentMessages(Guid documentId, string documentType) { DocumentId = documentId; DocumentType = documentType; } public static ProcessDocumentMessages New&lt;TDocument&gt;( TDocument document) where TDocument : DocumentBase { return new ProcessDocumentMessages( document.Id, document.GetType().AssemblyQualifiedName); } } </code></pre> <p>We can use this information to load our document from the repository. With this message in place, we now need the component that will <em>receive</em> our message. For this, it will really depend on our deployment, but for now I'll just make a .NET Core console application that includes our NServiceBus hosting piece and a handler for that message:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0052.png" alt=""></p> <p>I won't dig too much into the NServiceBus configuration as it's really not that germane, but let's look at the handler for that message:</p> <pre><code class="language-c#">public class ProcessDocumentMessagesHandler : IHandleMessages&lt;ProcessDocumentMessages&gt; { private readonly IDocumentMessageDispatcher _dispatcher; public ProcessDocumentMessagesHandler(IDocumentMessageDispatcher dispatcher) =&gt; _dispatcher = dispatcher; public Task Handle(ProcessDocumentMessages message, IMessageHandlerContext context) =&gt; _dispatcher.Dispatch(message); } </code></pre> <p>Also not very exciting! This class is what NServiceBus dispatches the durable message to. For our simple example, I'm using RabbitMQ, so if something goes wrong our message goes into a queue:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0053.png" alt=""></p> <p>Our handler receives this message to process it. The dispatcher is slightly different, as it needs to work with a message instead of an actual document, so it needs to load it up first:</p> <pre><code class="language-c#">public async Task Dispatch(ProcessDocumentMessages command) { var documentType = Type.GetType(command.DocumentType); var repository = GetRepository(documentType); var document = await repository.FindById(command.DocumentId); if (document == null) { return; } foreach (var message in document.Outbox.ToArray()) { var handler = GetHandler(message); await handler.Handle(message, _serviceFactory); document.ProcessDocumentMessage(message); await repository.Update(document); } } </code></pre> <p>One other key difference in this dispatch method is that we don't wrap anything in any kind of <code>try-catch</code> to report back errors. In an in-process dispatch mode, we still want the main request to succeed. In the offline processing mode, we're only dealing with document dispatching. And since we're using NServiceBus, we can rely on its <a href="https://docs.particular.net/nservicebus/recoverability/">built-in recoverability behavior</a> with immediate and delayed retries, eventually moving messages to an error queue.</p> <p>With this in place, we can put forth an optimistic, try-immediately policy, but fall back on durable messaging if something goes wrong with our immediate dispatch. It's not bulletproof, and in the next post, I'll look at how we can implement some sort of doomsday scenario, where we have a failure between dispatch failure and queuing the retry message.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=5IR7l0HDz8s:g0VmD5uNifc:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=5IR7l0HDz8s:g0VmD5uNifc:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=5IR7l0HDz8s:g0VmD5uNifc:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=5IR7l0HDz8s:g0VmD5uNifc:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=5IR7l0HDz8s:g0VmD5uNifc:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/5IR7l0HDz8s" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - Dispatching Example https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/ Jimmy Bogard urn:uuid:e4c35daa-5c50-d5b4-c719-e7a58e27ac7e Mon, 13 Aug 2018 19:24:06 +0000 <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> </ul> <p>In the last post, we looked at refactoring our documents to use messaging to communicate changes. We're still missing something, however - the dispatcher:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0050.png" alt=""></p> <p>Our dispatcher is the main component that facilitates document communication. For a given document,</p> <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-dispatching-example/">Dispatching Example</a></li> </ul> <p>In the last post, we looked at refactoring our documents to use messaging to communicate changes. We're still missing something, however - the dispatcher:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0050.png" alt=""></p> <p>Our dispatcher is the main component that facilitates document communication. For a given document, it needs to:</p> <ul> <li>Read messages out of a document's outbox</li> <li>Find the the document message handler for each, and invoke</li> <li>Manage failures for a document message handler</li> </ul> <p>We'll tackle that last piece in a future post. There's one piece we do need to think about first - where does the dispatcher get its list of documents to dispatch messages to?</p> <p>Before we get to the dispatcher, we need to solve for this problem - knowing which documents need dispatching!</p> <h3 id="introducingaunitofwork">Introducing a unit of work</h3> <p>For a given request, we'll load up a document and affect some change with it. We already have a pinch point in which our documents are loaded - the repository. If we want to dispatch document messages in the same request, we'll need to keep track of our documents that we've loaded in a request. For this, we can use a <a href="https://martinfowler.com/eaaCatalog/unitOfWork.html">Unit of Work</a>.</p> <p>Any ORM that you use will implement this pattern - for Entity Framework, for example, the DbContext is your Unit of Work. For Cosmos DB's SDK, there really isn't a concept of these ORM patterns. We have to introduce them ourselves.</p> <p>Our unit of work will keep track of documents for a given session/request, letting us interact with the loaded documents during the dispatch phase of a request. Our Unit of Work will also serve as an <a href="https://www.martinfowler.com/eaaCatalog/identityMap.html">identity map</a> - the thing that makes sure that when we load a document in a request, it's only loaded once. Here's our basic <code>IUnitOfWork</code> interface:</p> <pre><code class="language-c#">public interface IUnitOfWork { T Find&lt;T&gt;(Guid id) where T : DocumentBase; void Register(DocumentBase document); void Register(IEnumerable&lt;DocumentBase&gt; documents); Task Complete(); } </code></pre> <p>The implementation contains the "identity map" as a simple <code>HashSet</code></p> <pre><code class="language-c#">public class UnitOfWork : IUnitOfWork { private readonly ISet&lt;DocumentBase&gt; _identityMap = new HashSet&lt;DocumentBase&gt;(DocumentBaseEqualityComparer.Instance); </code></pre> <p>Then we can register an instance with our <code>UnitOfWork</code>:</p> <pre><code class="language-c#">public void Register(DocumentBase document) { _identityMap.Add(document); } public void Register(IEnumerable&lt;DocumentBase&gt; documents) { foreach (var document in documents) { Register(document); } } </code></pre> <p>Finding an existing <code>DocumentBase</code> just searches our identity map:</p> <pre><code class="language-c#">public T Find&lt;T&gt;(Guid id) where T : DocumentBase =&gt; _identityMap.OfType&lt;T&gt;().FirstOrDefault(ab =&gt; ab.Id == id); </code></pre> <p>We'll come back to the <code>Complete</code> method, because this will be the part where we dispatch. We still need the part where we register our documents in the unit of work, and this will be in our repository implementation:</p> <pre><code class="language-c#">public async Task&lt;T&gt; GetItemAsync(Guid id) { try { var root = _unitOfWork.Find&lt;T&gt;(id); if (root != null) return root; Document document = await _client.ReadDocumentAsync(UriFactory.CreateDocumentUri(DatabaseId, CollectionId, id.ToString())); var item = (T)(dynamic)document; _unitOfWork.Register(item); return item; } catch (DocumentClientException e) { if (e.StatusCode == System.Net.HttpStatusCode.NotFound) { return null; } throw; } } </code></pre> <p>We'll repeat this for any method in our repository that loads a document, registering and looking up in our unit of work.</p> <p>With a means to track our documents, let's see how we'll dispatch.</p> <h3 id="dispatchingdocumentmessages">Dispatching document messages</h3> <p>Our dispatcher's fairly straightforward - the only wrinkle is we'll need to surface any potential exception out. Instead of just crashing in case something goes awry, we'll want to just surface the exception and let the caller decide how to handle failures:</p> <pre><code class="language-c#">public interface IDocumentMessageDispatcher { Task&lt;Exception&gt; Dispatch(DocumentBase document); } </code></pre> <p>If I'm dispatching a document message to three handlers, I don't want one handler prevent dispatching to others.</p> <p>We have another challenge - our interface is not generic for dispatching, but the handlers and repositories are! We'll have to do some generics tricks to unwrap our base type to the correct generic types. The basic flow will be:</p> <ul> <li>For each document message:</li> <li>Find document message handlers</li> <li>Call the handler</li> <li>Remove the document message from the outbox</li> <li>Save the document</li> </ul> <p>Here's our basic implementation:</p> <pre><code class="language-c#">public async Task&lt;Exception&gt; Dispatch(DocumentBase document) { var repository = GetRepository(document.GetType()); foreach (var documentMessage in document.Outbox.ToArray()) { try { var handler = GetHandler(documentMessage); await handler.Handle(documentMessage, _serviceFactory); document.ProcessDocumentMessage(documentMessage); await repository.Update(document); } catch (Exception ex) { return ex; } } return null; } </code></pre> <p>We first build a repository based on the document type. Next, we loop through each document message in the outbox. For each document message, we'll find the handler(s) and call them. Once those succeed, we'll process our document message (removing it from the outbox) and update our document. We want to update for each document message in the outbox - if there's 3 document messages in the outbox, we save 3 times to make sure once message completes we don't have to go back to it if something goes wrong.</p> <p>The <code>GetHandler</code> method is a bit wonky, because we're bridging generics. Basically, we create a non-generic version of the document message handlers:</p> <pre><code class="language-c#">private abstract class DomainEventDispatcherHandler { public abstract Task Handle( IDocumentMessage documentMessage, ServiceFactory factory); } </code></pre> <p>Then create a generic version that inherits from this:</p> <pre><code class="language-c#">private class DomainEventDispatcherHandler&lt;T&gt; : DomainEventDispatcherHandler where T : IDocumentMessage { public override Task Handle(IDocumentMessage documentMessage, ServiceFactory factory) { return HandleCore((T)documentMessage, factory); } private static async Task HandleCore(T domainEvent, ServiceFactory factory) { var handlers = factory.GetInstances&lt;IDocumentMessageHandler&lt;T&gt;&gt;(); foreach (var handler in handlers) { await handler.Handle(domainEvent); } } } </code></pre> <p>I've used this pattern countless times, basically to satisfy the compiler. I've tried <code>dynamic</code> too but it introduces other problems. Then to call this, our <code>GetHandler</code> instantiates the generic version, but returns the non-generic base class:</p> <pre><code class="language-c#">private static DomainEventDispatcherHandler GetHandler( IDocumentMessage documentMessage) { var genericDispatcherType = typeof(DomainEventDispatcherHandler&lt;&gt;) .MakeGenericType(documentMessage.GetType()); return (DomainEventDispatcherHandler) Activator.CreateInstance(genericDispatcherType); } </code></pre> <p>With this, I can have non-generic code still call into generics. I'll do something similar with the repository:</p> <pre><code class="language-c#">private abstract class DocumentDbRepo { public abstract Task&lt;DocumentBase&gt; FindById(Guid id); public abstract Task Update(DocumentBase document); } </code></pre> <p>With these bridges in place, my dispatcher can interact with the concrete generic repositories and handlers. The final piece is the document cleaning up its outbox:</p> <pre><code class="language-c#">public void ProcessDocumentMessage( IDocumentMessage documentMessage) { _outbox?.Remove(documentMessage); } </code></pre> <p>With our dispatcher done, and our unit of work in place, we can now focus on the piece that will <em>invoke</em> our unit of work.</p> <h3 id="buildingamediatrbehavior">Building a MediatR behavior</h3> <p>We want our unit of work to complete with each request once everything is "done". For ASP.NET Core applications, this might mean some kind of filter. For us, I want the dispatching to work really with any context, so one possibility is to use a <a href="https://github.com/jbogard/MediatR/wiki/Behaviors">MediatR behavior</a> to wrap our MediatR handler. A filter would work too of course, but we'd need to mimic our filters in tests if we want everything to still get dispatched appropriately.</p> <p>The behavior is pretty straightforward:</p> <pre><code class="language-c#">public class UnitOfWorkBehavior&lt;TRequest, TResponse&gt; : IPipelineBehavior&lt;TRequest, TResponse&gt; { private readonly IUnitOfWork _unitOfWork; public UnitOfWorkBehavior(IUnitOfWork unitOfWork) { _unitOfWork = unitOfWork; } public async Task&lt;TResponse&gt; Handle( TRequest request, CancellationToken token, RequestHandlerDelegate&lt;TResponse&gt; next) { var response = await next(); await _unitOfWork.Complete(); return response; } } </code></pre> <p>We do the main work, then once that's finished, complete our unit of work.</p> <p>That's all of our infrastructure pieces, and the last part is registering these components with the DI container at startup:</p> <pre><code class="language-c#">services.AddMediatR(typeof(Startup)); services.AddScoped(typeof(IDocumentDBRepository&lt;&gt;), typeof(DocumentDBRepository&lt;&gt;)); services.AddScoped&lt;IUnitOfWork, UnitOfWork&gt;(); services.AddScoped&lt;IDocumentMessageDispatcher, DocumentMessageDispatcher&gt;(); services.AddScoped(typeof(IPipelineBehavior&lt;,&gt;), typeof(UnitOfWorkBehavior&lt;,&gt;)); services.Scan(c =&gt; { c.FromAssembliesOf(typeof(Startup)) .AddClasses(t =&gt; t.AssignableTo(typeof(IDocumentMessageHandler&lt;&gt;))) .AsImplementedInterfaces() .WithTransientLifetime(); }); </code></pre> <p>We add our MediatR handlers using the <a href="https://www.nuget.org/packages/MediatR.Extensions.Microsoft.DependencyInjection">MediatR.Extensions.Microsoft.DependencyInjection</a> package, our generic repository, unit of work, dispatcher, and unit of work behavior. Finally, we add all of the <code>IDocumentMessageHandler</code> implementations using <a href="https://github.com/khellang/Scrutor">Scrutor</a>, making our lives much easier to add all the handlers in one go.</p> <p>With all this in place, we can run and verify that our handlers fire and we can see the message in the inbox of the Stock item:</p> <pre><code class="language-json">{ "QuantityAvailable": 99, "ProductId": 771, "id": "cfbb6333-ed9f-49e7-8640-bb920d5c9106", "Outbox": { "$type": "System.Collections.Generic.HashSet`1[[AdventureWorksCosmos.UI.Infrastructure.IDocumentMessage, AdventureWorksCosmos.UI]], System.Core", "$values": [] }, "Inbox": { "$type": "System.Collections.Generic.HashSet`1[[AdventureWorksCosmos.UI.Infrastructure.IDocumentMessage, AdventureWorksCosmos.UI]], System.Core", "$values": [ { "$type": "AdventureWorksCosmos.UI.Models.Orders.ItemPurchased, AdventureWorksCosmos.UI", "ProductId": 771, "Quantity": 1, "Id": "2ab2108c-9698-49e8-93de-a3ced453836a" } ] }, "_rid": "WQk4AKSQMwACAAAAAAAAAA==", "_self": "dbs/WQk4AA==/colls/WQk4AKSQMwA=/docs/WQk4AKSQMwACAAAAAAAAAA==/", "_etag": "\"060077c2-0000-0000-0000-5b71d8a10000\"", "_attachments": "attachments/", "_ts": 1534187681 } </code></pre> <p>We now have effective document messaging between our documents!</p> <p>Well, almost.</p> <p>In the next post, we'll walk through what to do when things go wrong: failures and retries.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=h_qmyNvXZ8M:w2kXVjou_i4:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=h_qmyNvXZ8M:w2kXVjou_i4:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=h_qmyNvXZ8M:w2kXVjou_i4:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=h_qmyNvXZ8M:w2kXVjou_i4:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=h_qmyNvXZ8M:w2kXVjou_i4:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/h_qmyNvXZ8M" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - Document Example https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/ Jimmy Bogard urn:uuid:d1ccd01b-f000-9339-f675-b0d934468ef8 Thu, 09 Aug 2018 15:59:41 +0000 <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> </ul> <p>In the last post, I walked through the "happy path" scenario of coordinated communication/activities between multiple resources that otherwise can't participate in a transaction. In this post, I'll walk through a code example of building out document coordination in</p> <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-document-example/">Document Example</a></li> </ul> <p>In the last post, I walked through the "happy path" scenario of coordinated communication/activities between multiple resources that otherwise can't participate in a transaction. In this post, I'll walk through a code example of building out document coordination in <a href="https://azure.microsoft.com/en-us/services/cosmos-db/">Azure Cosmos DB</a>. My starting point is this set of code for approving an invoice and updating stock:</p> <pre><code class="language-c#">[HttpPost] public async Task&lt;IActionResult&gt; Approve(Guid id) { var orderRequest = await _orderRepository.GetItemAsync(id); orderRequest.Approve(); await _orderRepository.UpdateItemAsync(orderRequest); foreach (var lineItem in orderRequest.Items) { var stock = (await _stockRepository .GetItemsAsync(s =&gt; s.ProductId == lineItem.ProductId)) .FirstOrDefault(); stock.QuantityAvailable -= lineItem.Quantity; await _stockRepository.UpdateItemAsync(stock); } return RedirectToPage("/Orders/Show", new { id }); } </code></pre> <p>The repositories in my example are straight from the example code when you download a sample application in the Azure Portal, and just wrap the underlying <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.documentclient?view=azure-dotnet">DocumentClient</a>.</p> <h3 id="modelingourdocument">Modeling our document</h3> <p>First, we need to baseline our document messages. These objects can be POCOs, but we still need some base information. Since we want to enforce idempotent actions, we need to be able to distinguish between different messages. The easiest way to do so is with a unique identifier per message:</p> <pre><code class="language-c#">public interface IDocumentMessage { Guid Id { get; } } </code></pre> <p>Since our documents need to store and process messages in an inbox/outbox, we need to build out our base Document class to include these items. We can also build a completely separate object for our inbox/outbox, but for simplicity sake, we'll just use a base class:</p> <pre><code class="language-c#">public abstract class DocumentBase { [JsonProperty(PropertyName = "id")] public Guid Id { get; set; } private HashSet&lt;IDocumentMessage&gt; _outbox = new HashSet&lt;IDocumentMessage&gt;(DocumentMessageEqualityComparer.Instance); private HashSet&lt;IDocumentMessage&gt; _inbox = new HashSet&lt;IDocumentMessage&gt;(DocumentMessageEqualityComparer.Instance); public IEnumerable&lt;IDocumentMessage&gt; Outbox { get =&gt; _outbox; protected set =&gt; _outbox = value == null ? new HashSet&lt;IDocumentMessage&gt;(DocumentMessageEqualityComparer.Instance) : new HashSet&lt;IDocumentMessage&gt;(value, DocumentMessageEqualityComparer.Instance); } public IEnumerable&lt;IDocumentMessage&gt; Inbox { get =&gt; _inbox; protected set =&gt; _inbox = value == null ? new HashSet&lt;IDocumentMessage&gt;(DocumentMessageEqualityComparer.Instance) : new HashSet&lt;IDocumentMessage&gt;(value, DocumentMessageEqualityComparer.Instance); } </code></pre> <p>Each of our mailboxes are a <a href="https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.hashset-1?view=netframework-4.7.2">HashSet</a>, to ensure we enforce uniqueness of document messages inside our document. We wrap our mailboxes in a couple of convenience properties for storage purposes (since our documents are serialized using JSON.NET, we have to model appropriately for its serialization needs).</p> <p>We're using a custom equality comparer for document messages based on that interface and ID we added earlier:</p> <pre><code class="language-c#">public class DocumentMessageEqualityComparer : IEqualityComparer&lt;IDocumentMessage&gt; { public static readonly DocumentMessageEqualityComparer Instance = new DocumentMessageEqualityComparer(); public bool Equals(IDocumentMessage x, IDocumentMessage y) { return x.Id == y.Id; } public int GetHashCode(IDocumentMessage obj) { return obj.Id.GetHashCode(); } } </code></pre> <p>With this, we can make sure that our document messages only exist in our inbox/outboxes once (assuming we can pick unique GUIDs).</p> <p>Next, we need to be able to send a message in our <code>DocumentBase</code> class:</p> <pre><code class="language-c#">protected void Send(IDocumentMessage documentMessage) { if (_outbox == null) _outbox = new HashSet&lt;IDocumentMessage&gt;(DocumentMessageEqualityComparer.Instance); _outbox.Add(documentMessage); } </code></pre> <p>We have to check that the outbox exists and create it if it's not (due to serialization, it might not exist), then simply add the document message to the outbox.</p> <p>To process a document message, we need to make sure this action is idempotent. To check for idempotency, we'll examine our <code>Inbox</code> before executing the action. We can wrap this all up in a single method that our derived documents will use:</p> <pre><code class="language-c#">protected void Process&lt;TDocumentMessage&gt;( TDocumentMessage documentMessage, Action&lt;TDocumentMessage&gt; action) where TDocumentMessage : IDocumentMessage { if (_inbox == null) _inbox = new HashSet&lt;IDocumentMessage&gt;(DocumentMessageEqualityComparer.Instance); if (_inbox.Contains(documentMessage)) return; action(documentMessage); _inbox.Add(documentMessage); } </code></pre> <p>Our derived documents will need to call this method to process their messages with the idempotency check. Once a message is processed successfully, we'll add it to the inbox. And since our transaction boundary is the document, if something fails, the action never happened and the message never gets stored to the inbox. Only by keeping our inbox, outbox, and business data inside a transaction boundary can we guarantee all either succeeds or fails.</p> <h3 id="refactoringouraction">Refactoring our action</h3> <p>Now that we have our basic mechanism of storing and processing messages, we can refactor our original action. It was split basically into two actions - one of approving the invoice, and another of updating stock.</p> <p>We need to "send" a message from our Order to Stock. But what should that message look like? A few options:</p> <ul> <li>Command, "update stock"</li> <li>Event, "order approved"</li> <li>Event, "item purchased"</li> </ul> <p>If I go with a command, I'm coupling the primary action with the intended side effect. But what if this side effect needs to change? Be removed? I don't want burden the main Order logic with that.</p> <p>What about the first event, "order approved"? I could go with this - but looking at the work done and communication, Stock doesn't care that an order was approved, it only really cares if an item was purchased. Approvals are really the internal business rules of an order, but the ultimate side effect is that items finally become "purchased" at this point in time. So if I used "order approved", I'd be coupling Stock to the internal business rules of Order. Even though it's an event, "order approved" concerns internal business processes that other documents/services shouldn't care about.</p> <p>Finally, we have "item purchased". This most closely matches what Stock cares about, and removes any kind of process coupling between these two aggregates. If I went with the macro event, "order approved", I'd still have to translate that to what it means for Stock.</p> <p>With this in mind, I'll create a document message representing this event:</p> <pre><code class="language-c#">public class ItemPurchased : IDocumentMessage { public int ProductId { get; set; } public int Quantity { get; set; } public Guid Id { get; set; } } </code></pre> <p>I know how much of which product was purchased, and that's enough for Stock to deal with the consequences of that event.</p> <p>My <code>Order</code> class then models its <code>Approve</code> method to include sending these new messages:</p> <pre><code class="language-c#">public void Approve() { Status = Status.Approved; foreach (var lineItem in Items) { Send(new ItemPurchased { ProductId = lineItem.ProductId, Quantity = lineItem.Quantity, Id = Guid.NewGuid() }); } } </code></pre> <p>I don't have an idempotency check here (if the order is already approved, do nothing), but you get the idea.</p> <p>On the Stock side, I need to add a method to process the <code>ItemApproved</code> message:</p> <pre><code class="language-c#">public void Handle(ItemPurchased message) { Process(message, e =&gt; { QuantityAvailable -= e.Quantity; }); } </code></pre> <p>Finally, we need some way of linking our <code>ItemPurchased</code> message with the <code>Stock</code>, and that's the intent of an <code>IDocumentMessageHandler</code>:</p> <pre><code class="language-c#">public interface IDocumentMessageHandler&lt;in T&gt; where T : IDocumentMessage { Task Handle(T message); } </code></pre> <p>The part of our action that loaded up each <code>Stock</code> is the code we'll put into our handler:</p> <pre><code class="language-c#">public class UpdateStockFromItemPurchasedHandler : IDocumentMessageHandler&lt;ItemPurchased&gt; { private readonly IDocumentDBRepository&lt;Stock&gt; _repository; public UpdateStockFromItemPurchasedHandler( IDocumentDBRepository&lt;Stock&gt; repository) =&gt; _repository = repository; public async Task Handle(ItemPurchased message) { var stock = (await _repository .GetItemsAsync(s =&gt; s.ProductId == message.ProductId)) .Single(); stock.Handle(message); await _repository.UpdateItemAsync(stock); } } </code></pre> <p>Not that exciting, as our document will handle the real business logic of handling the request. This class just connects the dots between an <code>IDocumentMessageHandler</code> and some <code>DocumentBase</code> instance.</p> <p>With these basic building blocks, we'll modify our action to only update the <code>Order</code> instance:</p> <pre><code class="language-c#">[HttpPost] public async Task&lt;IActionResult&gt; Approve(Guid id) { var orderRequest = await _orderRepository.GetItemAsync(id); orderRequest.Approve(); await _orderRepository.UpdateItemAsync(orderRequest); return RedirectToPage("/Orders/Show", new { id }); } </code></pre> <p>Now when we approve our order, we only create messages in the outbox, which get persisted along with the order. If I look at the saved order in Cosmos DB, I can verify the items are persisted:</p> <pre><code class="language-json">{ "Items": [ { "Quantity": 1, "ListPrice": 3399.99, "ProductId": 771, "ProductName": "Mountain-100 Silver, 38", "Subtotal": 3399.99 } ], "Status": 2, "Total": 3399.99, "Customer": { "FirstName": "Jane", "LastName": "Doe", "MiddleName": "Mary" }, "id": "8bf4bda2-3796-431e-9936-8511243352d2", "Outbox": { "$type": "System.Collections.Generic.HashSet`1[[AdventureWorksCosmos.UI.Infrastructure.IDocumentMessage, AdventureWorksCosmos.UI]], System.Core", "$values": [ { "$type": "AdventureWorksCosmos.UI.Models.Orders.ItemPurchased, AdventureWorksCosmos.UI", "ProductId": 771, "Quantity": 1, "Id": "987ce801-e7cf-4abf-aba7-83d7eed00610" } ] }, "Inbox": { "$type": "System.Collections.Generic.HashSet`1[[AdventureWorksCosmos.UI.Infrastructure.IDocumentMessage, AdventureWorksCosmos.UI]], System.Core", "$values": [] }, "_rid": "lJFnANVMlwADAAAAAAAAAA==", "_self": "dbs/lJFnAA==/colls/lJFnANVMlwA=/docs/lJFnANVMlwADAAAAAAAAAA==/", "_etag": "\"02002652-0000-0000-0000-5b48f2140000\"", "_attachments": "attachments/", "_ts": 1531507220 } </code></pre> <p>In order to get that polymorphic behavior for my <code>IDocumentMessage</code> collections, I needed to configure the JSON serializer settings in my repository:</p> <pre><code class="language-c#">_client = new DocumentClient(new Uri(Endpoint), Key, new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.Auto }); </code></pre> <p>With these pieces in place, I've removed the process coupling between updating an order's status and updating stock items using document messaging. Of course, we don't actually have anything <em>dispatching</em> our messages. We'll cover the infrastructure for dispatching messages in the next post.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=gm7Aru-c8xk:ypdwK1jbSVk:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=gm7Aru-c8xk:ypdwK1jbSVk:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=gm7Aru-c8xk:ypdwK1jbSVk:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=gm7Aru-c8xk:ypdwK1jbSVk:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=gm7Aru-c8xk:ypdwK1jbSVk:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/gm7Aru-c8xk" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - Document Coordination https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/ Jimmy Bogard urn:uuid:3858773b-7e88-f2b4-c449-1928e0c80973 Tue, 07 Aug 2018 18:24:19 +0000 <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> </ul> <p><mark>Quick note - I've updated this post to use the more accurate term "Document" than the less-accurate, DDD-specific term "Aggregate". In the images, it still has the older term "Aggregate", let's pretend I fixed all those images.</mark></p> <p>In the last post, I</p> <p>Posts in this series:</p> <ul> <li><a href="https://jimmybogard.com/life-beyond-transactions-implementation-primer/">A Primer</a></li> <li><a href="https://jimmybogard.com/life-beyond-distributed-transactions-an-apostates-implementation-aggregate-coordination/">Document Coordination</a></li> </ul> <p><mark>Quick note - I've updated this post to use the more accurate term "Document" than the less-accurate, DDD-specific term "Aggregate". In the images, it still has the older term "Aggregate", let's pretend I fixed all those images.</mark></p> <p>In the last post, I walked through the general problem of distributed transactions, and some potential ideas around coordinating activities between similar or disparate resources. In a lot of cases, I would prefer to simply wrap our actions in a transaction, but as the resources I'm trying to transact together move further apart (process-wise or network-wise), transactions between these multiple resources becomes impossible.</p> <p>If I can only perform reliable transactions with a single resource at a time, whether that resource is a database or single record in Cosmos DB, I need to design my interactions so that both the <em>business data</em> and <em>communications</em> transact together. That means that my communication needs to be in the same transactional store as my business data!</p> <p>In the case of Azure Cosmos DB, I need to place my communication that I both send and receive inside my document. I store <em>incoming</em> communication because any request to change data needs to be idempotent. I could just have some business rule "all actions must be idempotent", I can also achieve this by simply storing the incoming requests and checking to see if I've already processed this request before performing the action.</p> <p>For outgoing communication, I can't interact with any other resource transactionally, so I store outgoing communication on what I <em>can</em> control - the document. So our final document looks like:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0034.png" alt=""></p> <p>When I perform an operation, communication and business data either save successfully or the entire operation rolls back.</p> <h3 id="dispatchingchanges">Dispatching Changes</h3> <p>When I find I have an operation that needs to affect more than one resource, I then use my outbox to communicate with those external resources:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0035.png" alt=""></p> <p>I can then safely transact that single resource:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0036.png" alt=""></p> <p>Now that I've completed the first action, I can now begin my interaction with other resources. I use a dispatcher to do so - something that's responsible for reading from a resource's outbox:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0037.png" alt=""></p> <p>And pass them to the correct resource:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0039.png" alt=""></p> <p>Each resource receives the incoming message, and performs 2 actions:</p> <ul> <li>Affect the business data</li> <li>Store the incoming message in the inbox</li> </ul> <p>This entire operation is transactional:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0040.png" alt=""></p> <p>With that resource successful, the dispatcher moves on to the next resource. However, this operation can fail:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0041.png" alt=""></p> <p>When this happens, we move the message to an external retry queue:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0042.png" alt=""></p> <p>At this point the dispatcher is complete - but because we still have a failed resource, we can't remove our event from the original document's outbox.</p> <h3 id="retries">Retries</h3> <p>In our half-completed state, we need to re-process the outbox for our document. So our dispatcher wakes up from the Retry message and reads the messages in the original document's outbox:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0043.png" alt=""></p> <p>The dispatcher dispatches our message again to each of the receiving resources, processing each one in turn. Our first resource has already processed this message, which it knows because it's checked its Inbox before changing the data (making it idempotent):</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0044.png" alt=""></p> <p>With that complete, the dispatcher can move on to the (previously) failing document:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0045.png" alt=""></p> <p>And that document can now succeed. Of course, there might be a bug or something actively preventing us from succeeding - so in that case we'll need to design explicit failure strategies. For now, let's assume that our operations will <em>eventually</em> succeed. Once we've confirmed that we've successfully dispatched our message to all receivers, we can go back to our original document and remove the message:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0046.png" alt=""></p> <p>This is then saved in its own individual transaction:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/Picture0047.png" alt=""></p> <p>In each step of the way, we have to work in smaller transactional steps that can be individually retried at least once, since each operation is idempotent. By storing our communication in the same transactional boundary of our business data, we can introduce reliable resource coordination when each of our resources isn't able to participate in that transaction.</p> <p>In the next post, I'll walk through some code examples of this pattern in Cosmos DB.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=TNP7zMuYPS4:L5TjgzzWPqs:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=TNP7zMuYPS4:L5TjgzzWPqs:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=TNP7zMuYPS4:L5TjgzzWPqs:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=TNP7zMuYPS4:L5TjgzzWPqs:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=TNP7zMuYPS4:L5TjgzzWPqs:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/TNP7zMuYPS4" height="1" width="1" alt=""/> Life Beyond Distributed Transactions: An Apostate's Implementation - A Primer https://jimmybogard.com/life-beyond-transactions-implementation-primer/ Jimmy Bogard urn:uuid:df26ea12-e99c-15d6-d131-6aaa57bcc48c Wed, 01 Aug 2018 13:56:53 +0000 <p>Posts in this series:</p> <ul> <li>A Primer</li> </ul> <p>For those working with SQL databases, working with transactions is more or less a given. The most we may need to worry about is:</p> <ul> <li>Using the appropriate isolation level</li> <li>Not doing too much in a single transaction to prevent excessive locks</li> </ul> <p>The vast majority</p> <p>Posts in this series:</p> <ul> <li>A Primer</li> </ul> <p>For those working with SQL databases, working with transactions is more or less a given. The most we may need to worry about is:</p> <ul> <li>Using the appropriate isolation level</li> <li>Not doing too much in a single transaction to prevent excessive locks</li> </ul> <p>The vast majority of applications I see can blissfully ignore the inner workings of transactions in database. We take for granted that our operations on one or more rows are either committed or rolled back.</p> <p>Things get more complicated when we start dealing with transactions no longer confined to a single resource, or in the case of many NoSQL databases, multiple entities. In many NoSQL databases, transactions are limited to a single entity/record. And if multi-entity transaction <em>are</em> supported, there are a number of limitations that might make that choice undesirable.</p> <p>When we have two non-transactional resources, we know we have a number of overall patterns to try to coordinate these actions (covered in my <a href="https://jimmybogard.com/refactoring-towards-resilience-a-primer/">Refactoring Towards Resilience series</a>:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/0/2017/Picture2.png" alt=""></p> <p>All of these options assume "I must have these two actions temporally coupled" and have them happen at the same time.</p> <p>But what if that wasn't the case? What if we moved away from trying to coordinate two actions, and had more loose coupling between our resources? What might that look like?</p> <p>And that's the main scope of Pat Helland's paper, <a href="https://queue.acm.org/detail.cfm?id=3025012">Life Beyond Transactions: An Apostate's Opinion</a>. In this paper, Pat describes a mechanism to overcome the fundamental issue of coordinating actions between resources when our transactions only cover a single entity - messaging!</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/helland7.png" alt=""></p> <p>We use some sort of messaging where our messages are saved <em>inside the entities</em> to direct to other entities:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/7/2018/helland5.png" alt=""></p> <p>Since the scope of a transaction is a single entity, if we need to affect other entities, we can't do that directly. Instead, we store the <em>intent</em> to affect change as a message inside our entity. The transaction covers our business data, <em>and</em> communication to the outside world.</p> <p>This certainly isn't the only way to tackle this problem, as I could use the <a href="http://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf">Saga pattern</a> as a means to manage failures between multi-entity activities. <a href="https://caitiem.com/">Caitie McCaffrey</a> has a <a href="https://www.youtube.com/watch?v=0UTOLRTwOX0">great talk about this</a>, but for my situation, I couldn't directly use the Saga pattern, since that implied there was some sort of logical "undo". Instead, I wanted to be able to any of the many coordination patterns available to me, <em>including</em> the Saga pattern.</p> <p>Before we get into implementation details, let's look at some real code in a real database that doesn't completely allow multi-entity transactions.</p> <h3 id="realworldexample">Real World Example</h3> <p>Let's suppose I have an ecommerce application, where I can view products, add items to a cart, check out, then finally, approve orders. As part of approving an order, I need to decrement stock. We'll skip any complex business rules, like negative stock, reservations, and the like. Just to keep things simple, when we order, we just subtract the quantity order from our stock reserve:</p> <pre><code class="language-c#">var orderRequest = await _orderRepository.GetItemAsync(id); orderRequest.Approve(); await _orderRepository.UpdateItemAsync(orderRequest); foreach (var lineItem in orderRequest.Items) { var stock = (await _stockRepository .GetItemsAsync(s =&gt; s.ProductId == lineItem.ProductId)) .Single(); stock.QuantityAvailable -= lineItem.Quantity; await _stockRepository.UpdateItemAsync(stock); } </code></pre> <p>In my case, I'm using Azure Cosmos DB, which supports a variety of <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels">consistency levels</a>. Azure Cosmos DB also supports multi-document transactions, but only in the form of <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/programming#database-program-transactions">stored procedures and functions</a>, and even then, only with some limitations. Once you introduce other resources into the mix, like Azure Service Bus, Azure SQL Database, or really anything outside a single partition key, we can no longer use transactions.</p> <p>With this in mind, we look back at our original code, and ask ourselves, "Do this action need to couple these resources, or can we decouple them?" There are certainly cases where we need to coordinate two actions (using a coordinator as we saw before), but there are many cases we don't, and don't want to incur the cost of a larger-scoped transaction.</p> <p>For our above case, do we need to deduct stock right at the time we approve an order? Or can it happen later? According to the business, deducting stock doesn't need to happen immediately, but it does need to happen, eventually.</p> <p>In the next few posts, I'll walk through building out a mechanism for communication with other entities (and even other resources), to see how we might build out an atomic communication system with Azure Cosmos DB as our example.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=hSI-fR-uQQY:K44F-QrzL7o:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=hSI-fR-uQQY:K44F-QrzL7o:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=hSI-fR-uQQY:K44F-QrzL7o:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=hSI-fR-uQQY:K44F-QrzL7o:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=hSI-fR-uQQY:K44F-QrzL7o:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/hSI-fR-uQQY" height="1" width="1" alt=""/> MediatR 5.1 Released https://jimmybogard.com/mediatr-5-1-released/ Jimmy Bogard urn:uuid:93b4541f-e927-c6c9-b551-24b9ae625db3 Mon, 30 Jul 2018 15:24:08 +0000 <p>This release changes the default behavior of awaiting an enumerable of <code>Task</code>. Previous to this release, tasks where awaited using <code>Task.WhenAll</code>. This causes problems in a variety of environments and situations that expect sequential ordering.</p> <p>In this release, the default behavior for enumerables is to await in a <code>foreach</code></p> <p>This release changes the default behavior of awaiting an enumerable of <code>Task</code>. Previous to this release, tasks where awaited using <code>Task.WhenAll</code>. This causes problems in a variety of environments and situations that expect sequential ordering.</p> <p>In this release, the default behavior for enumerables is to await in a <code>foreach</code>, for:</p> <ul> <li><code>Mediator.Publish</code></li> <li><code>RequestPreProcessorBehavior</code></li> <li><code>RequestPostProcessorBehavior</code></li> </ul> <p>You can override the <code>Mediator.Publish</code> behavior by overriding the virtual <code>PublishCore</code> method, while the other two you can simply replace with your own implementation.</p> <p>Install via NuGet:</p> <ul> <li><a href="https://www.nuget.org/packages/MediatR/">MediatR</a></li> <li><a href="https://www.nuget.org/packages/MediatR.Extensions.Microsoft.DependencyInjection/">MediatR.Extensions.Microsoft.DependencyInjection</a></li> </ul> <p>Enjoy!</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=EZcYU-AyvAc:u8kWNxWZqTo:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=EZcYU-AyvAc:u8kWNxWZqTo:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=EZcYU-AyvAc:u8kWNxWZqTo:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=EZcYU-AyvAc:u8kWNxWZqTo:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=EZcYU-AyvAc:u8kWNxWZqTo:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/EZcYU-AyvAc" height="1" width="1" alt=""/> Contoso University Examples with CQRS, MediatR, AutoMapper and more https://jimmybogard.com/contoso-university-examples-with-cqrs-mediatr-automapper-and-more/ Jimmy Bogard urn:uuid:e40416a1-6d11-ea6f-0a0c-7956c915baa4 Thu, 26 Jul 2018 15:07:02 +0000 <p>I was a bit surprised yesterday to see <a href="https://www.hanselman.com/blog/ExampleCodeOpinionatedContosoUniversityOnASPNETCore20sRazorPages.aspx">Scott Hanselman's post on my Contoso University sample app</a>. That application basically takes the traditional Contoso University sample applications (of which there have been many), and try to adapt it to how our "normal" ASP.NET applications are built. Mainly these applications</p> <p>I was a bit surprised yesterday to see <a href="https://www.hanselman.com/blog/ExampleCodeOpinionatedContosoUniversityOnASPNETCore20sRazorPages.aspx">Scott Hanselman's post on my Contoso University sample app</a>. That application basically takes the traditional Contoso University sample applications (of which there have been many), and try to adapt it to how our "normal" ASP.NET applications are built. Mainly these applications serve as working examples for our own internal teams, so that when we spin up a new application, we have something to work from.</p> <p>These applications highlight our typical usage of:</p> <ul> <li>AutoMapper</li> <li>MediatR</li> <li>Feature Folders</li> <li>FluentValidation</li> <li>HtmlTags</li> <li>Filters</li> <li>Fixie</li> <li>Integration testing</li> <li>Respawn</li> <li>AppVeyor</li> <li>Octopus Deploy</li> </ul> <p>To achieve a cleaner architecture. Since I'm just going off of the Contoso University samples from Microsoft, they're not the most complicated applications, but good enough for our purposes. If I was building an actual Contoso University like the sample, I wouldn't use many of these tools, but this application is "big enough" to show some different usages and give our teams a good starting point.</p> <p>The different Contoso University samples I've created include:</p> <ul> <li><a href="https://github.com/jbogard/ContosoUniversity">ASP.NET MVC 5</a></li> <li><a href="https://github.com/jbogard/ContosoUniversityCore">ASP.NET Core MVC 1.0 on full .NET</a></li> <li><a href="https://github.com/jbogard/ContosoUniversityDotNetCore">ASP.NET Core MVC 2.0 on .NET Core</a></li> <li><a href="https://github.com/jbogard/ContosoUniversityDotNetCore-Pages">ASP.NET Core Razor Pages 2.1 on .NET Core</a> &lt;- most recent</li> </ul> <p>There are a couple of other Docker-based versions, too, but those are more experiments than examples for our team.</p> <p>I like the Contoso University sample applications because they're big enough to give me an idea of a typical server-side application and how our toolset can be applied, as well as a way for me to test out our toolchain without worrying about breaking our client's code.</p> <p>In a future series, I'll walk through each of these tools specifically on some of the motivations, but for now, I just like having something I can pull down and experiment with!</p> <p>We at Headspring do a number of other different kinds of systems, from Xamarin, WPF, React/Redux w/ ASP.NET Core Web API, so I'd love to build versions of Contoso University for those.</p> <p>However, the challenge is finding someone else that's already put in the work for a decent, full-featured sample app! If you find one for those platforms, please let me know because selfishly, I'd love to build living, working examples for our teams :)</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=TgXlFq5mKmQ:V7WO4PV8s6s:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=TgXlFq5mKmQ:V7WO4PV8s6s:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=TgXlFq5mKmQ:V7WO4PV8s6s:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=TgXlFq5mKmQ:V7WO4PV8s6s:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=TgXlFq5mKmQ:V7WO4PV8s6s:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/TgXlFq5mKmQ" height="1" width="1" alt=""/> Containers - What Are They Good For? Running Our CI Builds https://jimmybogard.com/containers-what-are-they-good-for-ci-builds/ Jimmy Bogard urn:uuid:b849f371-fc99-05ad-3d20-4779e925168a Tue, 24 Jul 2018 14:14:18 +0000 <p>In the <a href="https://jimmybogard.com/containers-what-are-they-good-for-crafting-our-build-environments/">last post</a>, I looked at creating a build environment, settling on a strategy where I used Docker Compose to create our build environment. From there, I need some way to actually <em>run</em> our build in an environment.</p> <p>Typically, my Windows build run in some sort of hosted agent,</p> <p>In the <a href="https://jimmybogard.com/containers-what-are-they-good-for-crafting-our-build-environments/">last post</a>, I looked at creating a build environment, settling on a strategy where I used Docker Compose to create our build environment. From there, I need some way to actually <em>run</em> our build in an environment.</p> <p>Typically, my Windows build run in some sort of hosted agent, whether it's <a href="https://www.appveyor.com/">AppVeyor</a> or VSTS. The major advantage here is there's some pool of build agents that I don't really need to worry about, and it can run our build.</p> <p>Things get a little modified in our containerized approach, as we still need something to kick off our build. There are containerized CI solutions out there, such as <a href="https://circleci.com/docker/">CircleCI</a>, but as far as I can tell, they're Linux/macOS-only. Not helpful for us, as we want to target Windows build environments.</p> <p>However, there are public build agents that <em>do</em> support Windows Docker containers, the first one being AppVeyor.</p> <h3 id="appveyordockerbuilds">AppVeyor Docker Builds</h3> <p>We've already got our basic CI build (from the last post), so now all that's left is executing it in an AppVeyor build. <a href="https://www.appveyor.com/docs/build-environment/#docker">AppVeyor does support Docker</a>, with base images already installed in the cache. All we need to do is create our build, in my case with an <code>appveyor.yml</code> file:</p> <pre><code class="language-yml">version: 1.0.{build} image: Visual Studio 2017 build_script: - ps: .\ci.ps1 </code></pre> <p>Nothing that exciting, as we've already built out our Docker-based build. Once we've got this ready, we can push up our code.</p> <p>....and wait.</p> <p>....and wait.</p> <p>....</p> <p>....and finally get a result! Failure :(</p> <pre><code>Build started git clone -q --branch=master https://github.com/jbogard/contosouniversitydocker.git C:\projects\contosouniversitydocker git checkout -qf eb996f64cbe61e3349bf901542e8037e17bc1b05 .\ci.ps1 Pulling test-db (microsoft/mssql-server-windows-developer:)... latest: Pulling from microsoft/mssql-server-windows-developer Digest: sha256:a3e77eb7ac136bf419269ab6a3f3387df5055d78b2b6ba2e930e1c6312b50e07 Status: Downloaded newer image for microsoft/mssql-server-windows-developer:latest Building ci Step 1/11 : FROM microsoft/powershell:nanoserver-1709 AS installer-env nanoserver-1709: Pulling from microsoft/powershell Service 'ci' failed to build: a Windows version 10.0.16299-based image is incompatible with a 10.0.14393 host Exec: </code></pre> <p>So we've got a problem here, our container image isn't <em>exactly</em> compatible with the AppVeyor host. Another problem we have is because we decided to use an off-the-shelf SQL Server image, it took 14 minutes to download that image.</p> <p>Yikes!</p> <p>There are some ways we can fix all this, but ultimately, it comes down to:</p> <ul> <li>Finding smaller images for our database</li> <li>Finding a supported image for our "starting point" images</li> </ul> <p>I can't really do anything about a smaller database, other than switching to another database. As we saw in a previous post, this could mean SQL Local DB, but we run into the problem that it's quite hard to install from the command line without using Chocolatey. Which requires PowerShell. Which requires .NET 4.5, which requires an older nanoserver image.</p> <p>As for the starting point image, one thing we can do to make our lives a little easier is to build our "build image" and push this to a container registry. That way our build doesn't have to build our CI image from scratch, instead it's all built, and ready to go.</p> <h3 id="addressingouroptions">Addressing our options</h3> <p>I don't want to walk through each (painful) step for me, but I tried a number of different options for running our build:</p> <ul> <li>AppVeyor, with Docker Compose</li> <li>VSTS, with Docker Compose</li> <li>VSTS, with Docker steps</li> <li>VSTS, with Docker steps and our images in <a href="https://azure.microsoft.com/en-us/services/container-registry/">Azure Container Registry (ACR)</a></li> <li>VSTS, with a custom agent hosted in <a href="https://azure.microsoft.com/en-us/services/kubernetes-service/">Azure Kubernetes Service (AKS)</a> and images in ACR</li> </ul> <p>In the last example, I ditched the hosted agents and went purely with <em>my own</em> hosted agent. This meant that my CI build image was no longer a blank image, but the <a href="https://hub.docker.com/r/microsoft/vsts-agent/">VSTS image</a> so that it could connect with VSTS and build. My Kubernetes configuration then looked like:</p> <pre><code class="language-yml">apiVersion: v1 kind: ReplicationController metadata: name: vsts-agent spec: replicas: 1 template: metadata: labels: app: vsts-agent version: "0.1" spec: containers: - name: vsts-agent image: jbogard.azurecr.io/vsts-agent:windows-10.0.14393 env: - name: VSTS_ACCOUNT valueFrom: secretKeyRef: name: vsts key: VSTS_ACCOUNT - name: VSTS_TOKEN valueFrom: secretKeyRef: name: vsts key: VSTS_TOKEN - name: VSTS_POOL value: k8s-windowsservercore volumeMounts: - mountPath: /var/run/docker.sock name: docker-volume imagePullSecrets: - name: vsts-agent-login nodeSelector: beta.kubernetes.io/os: windows volumes: - name: docker-volume hostPath: path: /var/run/docker.sock </code></pre> <p>Yikes. At this point, I basically gave up. My build times did finally go down (from 30 minutes to ~5) on Kubernetes, but now I was managing my own k8s cluster just to run a build. It wasn't elastic either, I couldn't spin up new agents on demand without introducing Azure Functions, web hooks, and a bunch of other things that would take me <em>forever</em> to get set up.</p> <h3 id="finalconclusion">Final conclusion</h3> <p>After all of this effort, I found that for the container experience on Windows, it works really well for local dependencies and building production-runnable instances.</p> <p>But as a build/CI environment, it's still a poor experience. Either I completely change my targets (going from Windows containers to Linux), or have always-on container hosts that can cache my images effectively.</p> <p>Where I still find hope for container images on Windows are for a better packaging and execution mechanism for my applications. Today, I largely use Octopus Deploy or VSTS and their artifact mechanisms to package my application for production:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/6/2018/Picture0033.png" alt=""></p> <p>Instead of using NuGet, a fancy zip file with my code and a manifest, what about a fully ready-to-run application? This is where I'd like to see my systems move towards - a better packaging mechanism that includes <em>everything</em> needed to run my application. Whether this is Docker or a better linker, I'm not sure yet, but something that packages my application and its runtime to ensure portability would reduce the amount of setup and headaches when I get to production.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=AbOHGQgxDuM:vBI0CSTkCkY:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=AbOHGQgxDuM:vBI0CSTkCkY:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=AbOHGQgxDuM:vBI0CSTkCkY:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=AbOHGQgxDuM:vBI0CSTkCkY:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=AbOHGQgxDuM:vBI0CSTkCkY:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/AbOHGQgxDuM" height="1" width="1" alt=""/> AutoMapper extensions for Microsoft DI 5.0 released https://jimmybogard.com/automapper-extensions-for-microsoft-di-5-0-released/ Jimmy Bogard urn:uuid:2cd3111b-450f-4a8a-c927-865cf4a8d2ce Thu, 19 Jul 2018 13:40:06 +0000 <p>Hot off the presses, with some pretty big breaking changes. For the unfamiliar, the <a href="https://www.nuget.org/packages/automapper.extensions.microsoft.dependencyinjection">AutoMapper.Extensions.Microsoft.DependencyInjection</a> package is the extension to the MS DI libraries used by ASP.NET Core and others. It lets you do:</p> <pre><code class="language-c#">services.AddMvc(); services.AddAutoMapper(); </code></pre> <p>And this package finds all AutoMapper configuration in</p> <p>Hot off the presses, with some pretty big breaking changes. For the unfamiliar, the <a href="https://www.nuget.org/packages/automapper.extensions.microsoft.dependencyinjection">AutoMapper.Extensions.Microsoft.DependencyInjection</a> package is the extension to the MS DI libraries used by ASP.NET Core and others. It lets you do:</p> <pre><code class="language-c#">services.AddMvc(); services.AddAutoMapper(); </code></pre> <p>And this package finds all AutoMapper configuration in the current AppDomain, or assemblies you specify, configure everything, and register the AutoMapper-specific services with the container.</p> <p>In the past, I've supported both the static- and instance-based means of using AutoMapper with this library. However, because of a number of issues, the static version is fundamentally incompatible with dependency injection, making it far to easy to make mistakes.</p> <p>This version removes the support for static usage of AutoMapper, and you now <em>must</em> inject an <code>IMapper</code> for instance mapping or <code>IConfigurationProvider</code> for LINQ projections (<code>ProjectTo</code>).</p> <p>It's a bit of a breaking change, but I think for the better as it removes this wonky static global instance of AutoMapper that's at odds with how anyone using ASP.NET Core/MS DI deals with any other dependency.</p> <p>To migrate, you'll need to replace all instances of using the static <code>Mapper</code> class with an <code>IMapper</code> dependency. I've also removed the configuration option to force the instance-based registration, so if you use that, just remove it.</p> <p>The moral of the story - if you're doing DI, don't go halfway.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=mB1xhCC17aI:_v0xvkyWXk8:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=mB1xhCC17aI:_v0xvkyWXk8:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=mB1xhCC17aI:_v0xvkyWXk8:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=mB1xhCC17aI:_v0xvkyWXk8:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=mB1xhCC17aI:_v0xvkyWXk8:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/mB1xhCC17aI" height="1" width="1" alt=""/> Containers - What Are They Good For? Crafting our Build Environments https://jimmybogard.com/containers-what-are-they-good-for-crafting-our-build-environments/ Jimmy Bogard urn:uuid:e70a3db9-cdb1-a4a4-d533-8781ec18a51b Wed, 20 Jun 2018 18:24:59 +0000 <p>In the <a href="https://jimmybogard.com/containers-what-are-they-good-for-build/">last post</a>, I looked at what comes "out of the box" with the Docker support in Visual Studio. While what we saw makes a lot sense for a good "F5" story, it's not that great for an actual build environment.</p> <p>In a normal build environment, we have not</p> <p>In the <a href="https://jimmybogard.com/containers-what-are-they-good-for-build/">last post</a>, I looked at what comes "out of the box" with the Docker support in Visual Studio. While what we saw makes a lot sense for a good "F5" story, it's not that great for an actual build environment.</p> <p>In a normal build environment, we have not only the components to build and package, but also to run our tests. And for us, tests mean we run not only unit tests, but integration tests that run against a database. Ideally, we'd also test our database migration scripts as well. Combining the two steps means our "build":</p> <ul> <li>Restores</li> <li>Builds</li> <li>Migrates the DB</li> <li>Runs unit tests</li> <li>Runs integration tests against the DB</li> <li>Packages</li> <li>Publishes</li> </ul> <p>I'm going to leave off the last two steps for now, and focus on the "Continuous Integration" part of our environment:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/1/2018/Picture0018.png" alt="CI/CD Environment"></p> <p>I want to focus on containers for the continuous integration part. Our big decision here is - what exactly is our build environment? We have a few options:</p> <ul> <li>Use the "kitchen sink" image from the VSTS team, just using what they've build as our image</li> <li>Start from default images and layer on the pieces we need into one single image</li> <li>Use Docker Compose to compose multiple images together into a build "environment"</li> </ul> <p>In our normal container-less environment, we run our build on the host machine. We now need to construct a build environment in a container, or multiple containers. Let's look first at constructing a build environment.</p> <h3 id="kitchensinkapproach">Kitchen Sink approach</h3> <p>In this approach, instead of trying to add each dependency on our own, we'll just use exactly what the build server uses. Not all cloud providers have this available, but VSTS lets you run build agents as Docker images. They've <a href="https://hub.docker.com/r/microsoft/vsts-agent/">got a <em>ton</em> available</a> and we can pick and choose our OS and other requirements.</p> <p>On top of that, they have two separate flavors of images:</p> <ul> <li>Base image with only VSTS agent connections</li> <li>Kitchen sink image with a ton of dependencies pre-installed</li> </ul> <p>There is a problem here, however. It's Ubuntu-only! We want a Windows image, not *nix one.</p> <p>There are older VSTS container images, however, <a href="https://github.com/Microsoft/vsts-agent-docker/blob/master/windows/servercore/10.0.14393/standard/VS2017/Dockerfile">they're very old</a>, and not maintained.</p> <p>Suppose we did go this route, what might it give us? Mainly, the ability to run VSTS agents as containers, and more intriguingly, the ability to scale up and down through container orchestrators. But since it's *nix-only, won't work for us. Kitchen sink, sunk.</p> <h3 id="singleimagebuiltup">Single image built up</h3> <p>Just working with a single image can be a bit nicer. We don't have to worry about composing multiple images together into a network, everything is self-contained. So our first step here is - what should be our starting image? To figure this out, let's see what our image requirements are:</p> <ul> <li>.NET Core 2.1 SDK</li> <li>Powershell 4 (build script)</li> <li>Some SQL, maybe SqlLocalDb? SQL Express?</li> </ul> <p>Not too bad, but we see already that we have some tough decisions. Container images are great if everything is already there, but because there's no "apt-get" in Windows, installing dependencies is...painful.</p> <p>First up, how about .NET Core SDK? Is there a build image available, or do we have to start even more fundamental? Well, we've already used a build image for local development, so we can use that as our starting point.</p> <p>However, the other two pieces <em>aren't</em> installed. No problem, right? Well, no, because Powershell 4 requires the full .NET Framework, and that's not easy to install via the command line. We could install Powershell Core, which is the cross-platform, .NET Core-based version of Powershell. That's not too bad to install into a container, provided <a href="https://github.com/PowerShell/PowerShell/blob/master/docker/release/nanoserver/Dockerfile">someone else has already figured it out</a>.</p> <p>My other option of course is to eliminate the Powershell dependency, but this means using just plain <code>build.cmd</code> files as my build script, and those are...ugly.</p> <p>And for SQL Server, there are images, but maybe we just want SQL Local DB? I couldn't find a way to do this strictly at the command line, but there is a <a href="https://chocolatey.org/packages/sqllocaldb">Chocolatey package</a>. However, then I'd need Chocolatey, and <a href="https://chocolatey.org/install#requirements">Chocolatey's requirements</a> are now more difficult for me, it needs the full .NET Framework. I'm trying to keep things lean and only have .NET Core on my build agent, so again, this is out.</p> <p>At this point, we're in a bit of a bind with a single image. I can certainly bring all the pieces together, but it seems like my dependencies are...many. Bringing in PowerShell 6, the .NET Core version, seems reasonable. But beyond that, adding the additional dependencies makes my container grow ever larger in size.</p> <p>So for now, single image is out.</p> <h3 id="dockercompose">Docker Compose</h3> <p>Right now I've been trying to get all my build dependencies on a single image, but what if we built our build "environment" not as a single image, but similar to our actual build environment? What if we built something like this:</p> <p><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/5/2018/Picture0032.png" alt="Docker Build Network with build and database images"></p> <p>Instead of trying to jam everything into one single image, what if we instead focus on two things:</p> <ul> <li>What is needed to build?</li> <li>What is needed to test?</li> </ul> <p>To build, I don't have a lot of dependencies. Really it's just .NET Core SDK, and PowerShell Core to kick things off. In a future release of the Docker .NET Core SDK containers, the team is looking at <a href="https://github.com/dotnet/dotnet-docker/issues/360#issuecomment-397327408">including PS Core installed</a> - but for now, we can just create an image to do this ourselves.</p> <p>Right now there isn't a public image that contains both of these dependencies, so we need to decide which is harder:</p> <ul> <li>Start with <a href="https://hub.docker.com/r/microsoft/powershell/">PowerShell Core image</a>, add .NET Core</li> <li>Start with <a href="https://hub.docker.com/r/microsoft/dotnet/">.NET Core 2.1 SDK image</a>, add PowerShell Core</li> </ul> <p>Both dockerfiles are publicly available, so I don't have to figure out <em>how</em> to install each. All things being equal, I'd really like to go with the smaller base image. However, there's a wrinkle here, in that in order to <em>add</em> either of these, I need something that can download a file from the web. If you look at each dockerfile, they both use the full <code>windowsservercore</code> image, with full .NET, as one of the intermediate, staged images.</p> <p>We'll see in the next post, but even though images are cached on the host machine, I do want to make sure I use the right cached images. Using an image that isn't cached, when the images are huuuuuuge, can crater a build.</p> <p>To be honest, neither option is that great, both the SDK and Server Core downloads are large, so let's just flip a coin and say "start with the Powershell Image". With this starting point, let's look at our dockerfile to create our "build" image:</p> <pre><code class="language-dockerfile"># escape=` # Installer image FROM microsoft/powershell:nanoserver-1709 AS installer-env SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"] # Retrieve .NET Core SDK ENV DOTNET_SDK_VERSION 2.1.300 RUN Invoke-WebRequest -OutFile dotnet.zip https://dotnetcli.blob.core.windows.net/dotnet/Sdk/$Env:DOTNET_SDK_VERSION/dotnet-sdk-$Env:DOTNET_SDK_VERSION-win-x64.zip; ` $dotnet_sha512 = '4aa6ff6aa51e1d71733944e10fd9e37647a58df7efbc76f432b8c3ffa3f617f9da36f72532175a1e765dbaf4598a14350017342d5f776dfe8e25d5049696d003'; ` if ((Get-FileHash dotnet.zip -Algorithm sha512).Hash -ne $dotnet_sha512) { ` Write-Host 'CHECKSUM VERIFICATION FAILED!'; ` exit 1; ` }; ` ` Expand-Archive dotnet.zip -DestinationPath dotnet; ` Remove-Item -Force dotnet.zip # SDK image FROM microsoft/powershell:nanoserver-1709 COPY --from=installer-env ["dotnet", "C:\\Program Files\\dotnet"] # In order to set system PATH, ContainerAdministrator must be used USER ContainerAdministrator RUN setx /M PATH "%PATH%;C:\Program Files\dotnet" USER ContainerUser # Configure Kestrel web server to bind to port 80 when present ENV ASPNETCORE_URLS=http://+:80 ` # Enable detection of running in a container DOTNET_RUNNING_IN_CONTAINER=true ` # Enable correct mode for dotnet watch (only mode supported in a container) DOTNET_USE_POLLING_FILE_WATCHER=true ` # Skip extraction of XML docs - generally not useful within an image/container - helps perfomance NUGET_XMLDOC_MODE=skip # Trigger first run experience by running arbitrary cmd to populate local package cache RUN dotnet help </code></pre> <p>Not that exciting, we're just copying the <a href="https://github.com/dotnet/dotnet-docker/blob/master/2.1/sdk/nanoserver-1709/amd64/Dockerfile">.NET Core 2.1 SDK dockerfile</a> and using the Powershell Core docker images instead of a plain nanoserver/windowsservercore image.</p> <p>To make sure I can run this OK, from a command line I'll make sure everything works:</p> <pre><code>&gt; docker build .\ -t contosouniversity/ci ... lots of messages &gt; docker run -it contosouniversity/ci pwsh -Command Write-Host "Hello World" Hello World &gt; docker run -it contosouniversity/ci pwsh -Command dotnet --version 2.1.300 </code></pre> <p>Good to go!</p> <p>Now that I have a build image that contains my dependencies, I need to craft a docker-compose file that stitches together my source from the host, the build image, and the database image I'm using in my "local running" environment:</p> <pre><code class="language-yaml">version: '3' services: ci: image: contosouniversity/ci build: context: . dockerfile: ci\Dockerfile volumes: - .\:c:\src working_dir: \src environment: - "ConnectionStrings:DefaultConnection=Server=test-db;Database=contosouniversity-test;User Id=sa;Password=Pass@word" depends_on: - test-db test-db: image: microsoft/mssql-server-windows-developer environment: - sa_password=Pass@word - ACCEPT_EULA=Y ports: - "6433:1433" networks: default: external: name: nat </code></pre> <p>I have two services, my build image <code>ci</code> and the database <code>test-db</code>. Instead of copying the source from the host to the build image, I simply mount the current directory to <code>C:\src</code>. With this altogether, I can run the build inside my Docker environment like:</p> <pre><code>docker-compose -f ./docker-compose.ci.yml -p contosouniversitydotnetcore-ci up -d --build --remove-orphans --force-recreate docker-compose -f ./docker-compose.ci.yml -p contosouniversitydotnetcore-ci run ci pwsh .\Build.ps1 </code></pre> <p>With this in place (inside a single Powershell script to make my life easier), I can run this build locally, and, eventually, my build succeeds!</p> <p>By using containers as my build environment, it means I have an absolutely controlled, deterministic, immutable build environment that I <em>know</em> won't fail based on some wonky environment settings.</p> <p>In our next post, we'll look at ways of running this build setup in some sort of hosted build environment.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=CvYzPmJTa1A:l0oVIFEdjl0:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=CvYzPmJTa1A:l0oVIFEdjl0:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=CvYzPmJTa1A:l0oVIFEdjl0:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=CvYzPmJTa1A:l0oVIFEdjl0:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=CvYzPmJTa1A:l0oVIFEdjl0:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/CvYzPmJTa1A" height="1" width="1" alt=""/> AutoMapper 7.0 Released https://jimmybogard.com/automapper-7-0-released/ Jimmy Bogard urn:uuid:2d70650d-acdf-0bca-cb8c-fa5738f0c5b5 Tue, 05 Jun 2018 18:55:41 +0000 <p>Today I pushed out the AutoMapper 7.0 release:</p> <p><a href="https://github.com/AutoMapper/AutoMapper/releases/tag/v7.0.0">Release notes</a></p> <p>This release is mainly bug fixes, but keeping with the last few major releases, has a few breaking API changes. The major breaking API change is breaking out the expression mapping (OData support) so that it can release independently</p> <p>Today I pushed out the AutoMapper 7.0 release:</p> <p><a href="https://github.com/AutoMapper/AutoMapper/releases/tag/v7.0.0">Release notes</a></p> <p>This release is mainly bug fixes, but keeping with the last few major releases, has a few breaking API changes. The major breaking API change is breaking out the expression mapping (OData support) so that it can release independently of the normal in-memory AutoMapper mapping. You can find this in the <a href="https://www.nuget.org/packages/automapper.extensions.expressionmapping">AutoMapper.Extensions.ExpressionMapping</a> package.</p> <p>I've also updated the <a href="https://www.nuget.org/packages/automapper.extensions.microsoft.dependencyinjection">AutoMapper.Extensions.Microsoft.DependencyInjection</a> package to this release (4.0.0).</p> <p>Enjoy!</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=1fKFVmxDjek:lS3Vi6a66g4:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=1fKFVmxDjek:lS3Vi6a66g4:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=1fKFVmxDjek:lS3Vi6a66g4:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=1fKFVmxDjek:lS3Vi6a66g4:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=1fKFVmxDjek:lS3Vi6a66g4:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/1fKFVmxDjek" height="1" width="1" alt=""/> MediatR 5.0 Released https://jimmybogard.com/mediatr-5-0-released/ Jimmy Bogard urn:uuid:53924c49-9a1c-7e6e-2d91-a8dc36dcbc79 Tue, 05 Jun 2018 14:07:23 +0000 <p>Another milestone for MediatR, I released 5.0 over the weekend. This release simplifies a number of interfaces and methods, but introduced some breaking changes along the way.</p> <p>The major change for MediatR in 5.0 is the combination of void requests to directly implement <code>IRequest&lt;Unit&gt;</code>. In</p> <p>Another milestone for MediatR, I released 5.0 over the weekend. This release simplifies a number of interfaces and methods, but introduced some breaking changes along the way.</p> <p>The major change for MediatR in 5.0 is the combination of void requests to directly implement <code>IRequest&lt;Unit&gt;</code>. In 4.x:</p> <pre><code class="language-c#">public interface IRequest&lt;T&gt; : IBaseRequest { } public interface IRequest : IBaseRequest { } </code></pre> <p>And in 5.0:</p> <pre><code class="language-c#">public interface IRequest&lt;T&gt; : IBaseRequest { } public interface IRequest : IRequest&lt;Unit&gt; { } </code></pre> <p>For the handlers, in 4.x:</p> <pre><code class="language-c#">public interface IRequestHandler&lt;in TRequest, TResponse&gt; where TRequest : IRequest&lt;TResponse&gt; { Task&lt;TResponse&gt; Handle(TRequest request, CancellationToken cancellationToken); } public interface IRequestHandler&lt;in TRequest&gt; where TRequest : IRequest { Task Handle(TRequest request, CancellationToken cancellationToken); } </code></pre> <p>And in 5.0:</p> <pre><code class="language-c#">public interface IRequestHandler&lt;in TRequest, TResponse&gt; where TRequest : IRequest&lt;TResponse&gt; { Task&lt;TResponse&gt; Handle(TRequest request, CancellationToken cancellationToken); } public interface IRequestHandler&lt;in TRequest&gt; : IRequestHandler&lt;TRequest, Unit&gt; where TRequest : IRequest&lt;Unit&gt; { } </code></pre> <p>While the original 4.x interfaces were convenient for void-based requests, having a completely separate interface meant that you really couldn't use pipeline behaviors or pre/post processors. With the 5.0 merging of interfaces, you can use the full feature set of MediatR without worrying about weird container problems.</p> <p>I've kept the helper base classes if you need them, and written up some <a href="https://github.com/jbogard/MediatR/releases/tag/v5.0.1">release notes</a> and <a href="https://github.com/jbogard/MediatR/wiki/Migration-Guide-4.x-to-5.0">migration guide</a>.</p> <p>With this release, I don't see any more changes in the interfaces for <code>IMediator</code>, the requests, or handlers.</p> <p>Enjoy!</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=kw4imzvzF8A:f0g7tu_CXg8:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=kw4imzvzF8A:f0g7tu_CXg8:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=kw4imzvzF8A:f0g7tu_CXg8:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=kw4imzvzF8A:f0g7tu_CXg8:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=kw4imzvzF8A:f0g7tu_CXg8:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/kw4imzvzF8A" height="1" width="1" alt=""/> Collaboration vs. Critique https://lostechies.com/derekgreer/2018/05/18/collaboration-vs-critique/ Los Techies urn:uuid:8a2d0bfb-9efe-2fd2-1e9b-6ba6d06055da Fri, 18 May 2018 17:00:00 +0000 While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams. The people aspect of software development, more often than not, tends to be the most difficult part of software engineering. Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs. The nature of software development tends to follow an empirical process model rather than a defined process model. That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset. Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation. This means that team members are often required to work together to determine how to proceed. Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation. One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique. <p>While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams. The people aspect of software development, more often than not, tends to be the most difficult part of software engineering. Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs. The nature of software development tends to follow an empirical process model rather than a defined process model. That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset. Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation. This means that team members are often required to work together to determine how to proceed. Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation. One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique.</p> <p>To help illustrate what I’ve seen happen countless times both in catch-up design sessions and code reviews, consider the following two scenarios:</p> <h3 id="scenario-1">Scenario 1</h3> <p>Tom and Sally are both developers on a team maintaining a large-scale application. Tom takes the next task in the development queue which happens to have some complex processes that will need to be addressed. Being the good development team that they are, both Tom and Sally are aware of the requirements of the application (i.e. how the app needs to work from the user’s perspective), but they have deferred design-level discussions until the time of implementation. After Tom gets into the process a little, seeing that the problem is non-trivial, he pings Sally to help him brainstorm different approaches to solving the problem. Tom and Sally have been working together for over a year and have become accustomed to these sort of ad-hoc design sessions. As they begin discussing the problem, they each start tossing ideas out on the proverbial table resulting in multiple approaches to compare and contrast. The nature of the discussion is such that neither Tom nor Sally are embarrassed or offended when the other points out flaws in the design because there’s a sense of safety in their mutual understanding that this is a brainstorming session and that neither have thought in depth about the solutions being set froth yet. Tom throws out a couple of ideas, but ends up shooting them down himself as he uses Sally as a sounding board for the ideas. Sally does the same, but toward the end of the conversation suggests a slight alteration to one of Tom’s initial suggestions that they think may make it work after all. They end the session both with a sense that they’ve worked together to arrive at the best solution.</p> <h3 id="scenario-2">Scenario 2</h3> <p>Bill and Jake are developers on another team. They tend to work in a more siloed fashion, but they do rely upon one another for help from time to time and they are required to do code reviews prior to their code being merged into the main branch of development. Bill takes the next task in the development queue and spends the better part of an afternoon working out a solution with a basic working skeleton of the direction he’s going. The next day he decides that it might be good to have Jake take a look at the design to make him aware of the direction. Seeing where Bill’s design misses a few opportunities to make the implementation more adaptable to changes in the future, Jake points out where he would have done things differently. Bill acknowledges that Jake’s suggestions would be better and would have probably been just as easy to implement from the beginning, but inwardly he’s a bit disappointed that Jake didn’t like his design as-is and that he has to do some rework. In the end, Bill is left with a feeling of critique rather than collaboration.</p> <p>Whether it’s a high-level UML diagram or working code, how one person tends to perceive feedback on the ideas comprising a potential solution has everything to do with timing. It can be the exact same feedback they would have received either way, but when the feedback occurs often makes a difference between whether it’s perceived as collaboration or critique. It’s all about when the conversation happens.</p> Testing Button Click in React with Jest https://derikwhittaker.blog/2018/05/07/testing-button-click-in-react-with-jest/ Maintainer of Code, pusher of bits… urn:uuid:a8e7d9fd-d718-a072-55aa-0736ac21bec4 Mon, 07 May 2018 17:01:59 +0000 When building React applications you will most likely find yourself using Jest as your testing framework.  Jest has some really, really cool features built in.  But when you use Enzyme you can take your testing to the nest level. One really cool feature is the ability to test click events via Enzyme to ensure your &#8230; <p><a href="https://derikwhittaker.blog/2018/05/07/testing-button-click-in-react-with-jest/" class="more-link">Continue reading <span class="screen-reader-text">Testing Button Click in React with&#160;Jest</span></a></p> <p>When building <a href="https://reactjs.org/" target="_blank" rel="noopener">React</a> applications you will most likely find yourself using <a href="https://facebook.github.io/jest" target="_blank" rel="noopener">Jest</a> as your testing framework.  Jest has some really, really cool features built in.  But when you use <a href="http://airbnb.io/enzyme/docs/guides/jest.html" target="_blank" rel="noopener">Enzyme</a> you can take your testing to the nest level.</p> <p>One really cool feature is the ability to test click events via Enzyme to ensure your code responds as expected.</p> <p>Before we get started you are going to want to make sure you have Jest and Enzyme installed in your application.</p> <ul> <li>Installing <a href="https://github.com/airbnb/enzyme/blob/master/docs/installation/README.md" target="_blank" rel="noopener">Enzyme</a></li> <li>Installing <a href="https://facebook.github.io/jest/docs/en/getting-started.html" target="_blank" rel="noopener">Jest</a></li> </ul> <p>Sample code under test</p> <p><img data-attachment-id="111" data-permalink="https://derikwhittaker.blog/2018/05/07/testing-button-click-in-react-with-jest/screen-shot-2018-05-07-at-12-52-56-pm/" data-orig-file="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640" data-orig-size="580,80" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="Screen Shot 2018-05-07 at 12.52.56 PM" data-image-description="" data-medium-file="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640?w=300" data-large-file="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640?w=580" class="alignnone size-full wp-image-111" src="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640" alt="Screen Shot 2018-05-07 at 12.52.56 PM" srcset="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png 580w, https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=150 150w, https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=300 300w" sizes="(max-width: 580px) 100vw, 580px" /></p> <p>What I would like to be able to do is pull the button out of my component and test the <code>onClick</code> event handler.</p> <div class="code-snippet"> <pre class="code-content"> // Make sure you have your imports setup correctly import React from 'react'; import { shallow } from 'enzyme'; it('When active link clicked, will push correct filter message', () =&gt; { let passedFilterType = ''; const handleOnTotalsFilter = (filterType) =&gt; { passedFilterType = filterType; }; const accounts = {}; const wrapper = shallow(&lt;MyComponent accounts={accounts} filterHeader="" onTotalsFilter={handleOnTotalsFilter} /&gt;); const button = wrapper.find('#archived-button'); button.simulate('click'); expect(passedFilterType).toBe(TotalsFilterType.archived); }); </pre> </div> <p>Lets take a look at the test above</p> <ol> <li>First we are going to create a callback (click handler) to catch the bubbled up values.</li> <li>We use Enzyme to create our component <code>MyComponent</code></li> <li>We use the .find() on our wrapped component to find our &lt;Button /&gt; by id</li> <li>After we get our button we can call .simulate(&#8216;click&#8217;) which will act as a user clicking the button.</li> <li>We can assert that the expected value bubbles up.</li> </ol> <p>As you can see, simulating a click event of a rendered component is very straight forward, yet very powerful.</p> <p>Till next time,</p> Lessons from a year of Golang https://lostechies.com/ryansvihla/2018/05/07/lessons-from-a-year-of-go/ Los Techies urn:uuid:e37d6484-2864-cc2a-034c-cac3d89dede7 Mon, 07 May 2018 13:16:00 +0000 I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive. <p>I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive.</p> <p>This isn’t meant to evaluate if you should pick up Go or tell you what you should think of it, this is strictly meant to help people out that are new to the language but experienced in Java, Python, Ruby, C#, etc and have read some basic Go getting started guide.</p> <h2 id="dependency-management">Dependency management</h2> <p>This is probably the feature most frequently talked about by newcomers to Go and with some justification, as dependency management been a rapidly shifting area that’s nothing like what experienced Java, C#, Ruby or Python developers are used to.</p> <p>I’ll cut to the chase the default tool now is <a href="https://github.com/golang/dep">Dep</a> all other tools I’ve used such as <a href="https://github.com/Masterminds/glide">Glide</a> or <a href="https://github.com/tools/godep">Godep</a> they’re now deprecated in favor of Dep, and while Dep has advanced rapidly there are some problems you’ll eventually run into (or I did):</p> <ol> <li>Dep hangs randomly and is slow, which is supposedly network traffic <a href="https://github.com/golang/dep/blob/c8be449181dadcb01c9118a7c7b592693c82776f/docs/failure-modes.md#hangs">but it happens to everyone I know with tons of bandwidth</a>. Regardless, I’d like an option to supply a timeout and report an error.</li> <li>Versions and transitive depency conflicts can be a real breaking issue in Go still. So without shading or it’s equivalent two package depending on different versions of a given package can break your build, there are a number or proposals to fix this but we’re not there yet.</li> <li>Dep has some goofy ways it resolves transitive dependencies and you may have to add explicit references to them in your Gopkg.toml file. You can see an example <a href="https://kubernetes.io/blog/2018/01/introducing-client-go-version-6/">here</a> under <strong>Updating dependencies – golang/dep</strong>.</li> </ol> <h3 id="my-advice">My advice</h3> <ul> <li>Avoid hangs by checking in your dependencies directly into your source repository and just using the dependency tool (dep, godep, glide it doesn’t matter) for downloading dependencies.</li> <li>Minimize transitive dependencies by keeping stuff small and using patterns like microservices when your dependency tree conflicts.</li> </ul> <h2 id="gopath">GOPATH</h2> <p>Something that takes some adjustment is you check out all your source code in one directory with one path (by default ~/go/src ) and include the path to the source tree to where you check out. Example:</p> <ol> <li>I want to use a package I found on github called jim/awesomeness</li> <li>I have to go to ~/go/src and mkdir -p github.com/jim</li> <li>cd into that and clone the package.</li> <li>When I reference the package in my source file it’ll be literally importing github.com/jim/awesomeness</li> </ol> <p>A better guide to GOPATH and packages is <a href="https://thenewstack.io/understanding-golang-packages/">here</a>.</p> <h3 id="my-advice-1">My advice</h3> <p>Don’t fight it, it’s actually not so bad once you embrace it.</p> <h2 id="code-structure">Code structure</h2> <p>This is a hot topic and there are a few standards for the right way to structure you code from projects that do “file per class” to giant files with general concept names (think like types.go and net.go). Also if you’re used to using a lot of sub package you’re gonna to have issues with not being able to compile if for example you have two sub packages reference one another.</p> <h3 id="my-advice-2">My Advice</h3> <p>In the end I was reasonably ok with something like the following:</p> <ul> <li>myproject/bin for generated executables</li> <li>myproject/cmd for command line code</li> <li>myproject/pkg for code related to the package</li> </ul> <p>Now whatever you do is fine, this was just a common idiom I saw, but it wasn’t remotely all projects. I also had some luck with just jamming everything into the top level of the package and keeping packages small (and making new packages for common code that is used in several places in the code base). If I ever return to using Go for any reason I will probably just jam everything into the top level directory.</p> <h2 id="debugging">Debugging</h2> <p>No debugger! There are some projects attempting to add one but Rob Pike finds them a crutch.</p> <h3 id="my-advice-3">My Advice</h3> <p>Lots of unit tests and print statements.</p> <h2 id="no-generics">No generics</h2> <p>Sorta self explanatory and it causes you a lot of pain when you’re used to reaching for these.</p> <h3 id="my-advice-4">My advice</h3> <p>Look at the code generation support which uses pragmas, this is not exactly the same as having generics but if you have some code that has a lot of boiler plate without them this is a valid alternative. See this official <a href="https://blog.golang.org/generate">Go Blog post</a> for more details.</p> <p>If you don’t want to use generation you really only have reflection left as a valid tool, which comes with all of it’s lack of speed and type safety.</p> <h2 id="cross-compiling">Cross compiling</h2> <p>If you have certain features or dependencies you may find you cannot take advantage of one of Go’s better features cross compilation.</p> <p>I ran into this when using the confluent-go/kafka library which depends on the C librdkafka library. It basically meant I had to do all my development in a Linux VM because almost all our packages relied on this.</p> <h3 id="my-advice-5">My Advice</h3> <p>Avoid C dependencies at all costs.</p> <h2 id="error-handling">Error handling</h2> <p>Go error handling is not exception base but return based, and it’s got a lot of common idioms around it:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>myValue, err := doThing() if err != nil { return -1, fmt.Errorf(“unable to doThing %v”, err) } </code></pre></div></div> <p>Needless to say this can get very wordy when dealing with deeply nested exceptions or when you’re interacting a lot with external systems. It is definitely a mind shift if you’re used to the throwing exceptions wherever and have one single place to catch all exceptions where they’re handled appropriately.</p> <h3 id="my-advice-6">My Advice</h3> <p>I’ll be honest I never totally made my peace with this. I had good training from experienced opensource contributors to major Go projects, read all the right blog posts, definitely felt like I’d heard enough from the community on why the current state of Go error handling was great in their opinions, but the lack of stack traces was a deal breaker for me.</p> <p>On the positive side, I found Dave Cheney’s advice on error handling to be the most practical and he wrote <a href="https://github.com/pkg/errors">a package</a> containing a lot of that advice, we found it invaluable as it provided those stack traces we all missed but you had to remember to use it.</p> <h2 id="summary">Summary</h2> <p>A lot of people really love Go and are very productive with it, I just was never one of those people and that’s ok. However, I think the advice in this post is reasonably sound and uncontroversial. So, if you find yourself needing to write some code it Go, give this guide a quick perusal and you’ll waste a lot less time than I did getting productive in developing applications in Go.</p> Raspberry Pi Kubernetes Cluster - Part 2 https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-2/ Jason Meridth urn:uuid:0aef121f-48bd-476f-e09d-4ca0aa2ac602 Thu, 03 May 2018 02:13:07 +0000 <p>Howdy again.</p> <p>Alright, my 8 port switch showed up so I was able to connect my raspberry 3B+ boards to my home network. I plugged it in with 6 1ft CAT5 cables I had in my catch-all box that all of us nerds have. I’d highly suggest flexible CAT 6 cables instead if you can get them, like <a href="https://www.amazon.com/Cat-Ethernet-Cable-Black-Connectors/dp/B01IQWGKQ6">here</a>. I ordered them and they showed up before I finished this post, so I am using the CAT6 cables.</p> <!--more--> <p>The IP addresses they will receive initialy from my home router via DHCP can be determined with nmap. Lets imagine my subnet is 192.168.1.0/24.</p> <p>Once I got them on the network I did the following:</p> <script src="https://gist.github.com/64e7b08729ffe779f77a7bda0221c6e9.js"> </script> <h3 id="install-raspberrian-os-on-sd-cards">Install Raspberrian OS On SD Cards</h3> <p>You can get the Raspberry Pi Stretch Lite OS from <a href="https://www.raspberrypi.org/downloads/raspbian/">here</a>.</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/raspberry_pi_stretch_lite.png" alt="Raspberry Pi Stretch Lite" /></p> <p>Then use the <a href="https://etcher.io/">Etcher</a> tool to install it to each of the 6 SD cards.</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/etcher.gif" alt="Etcher" /></p> <h4 id="important">IMPORTANT</h4> <p>Before putting the cards into the Raspberry Pis you need to add a <code class="highlighter-rouge">ssh</code> folder to the root of the SD cards. This will allow you to ssh to each Raspberry Pi with default credentials (username: <code class="highlighter-rouge">pi</code> and password <code class="highlighter-rouge">raspberry</code>). Example: <code class="highlighter-rouge">ssh pi@raspberry_pi_ip</code> where <code class="highlighter-rouge">raspberry_ip</code> is obtained from the nmap command above.</p> <p>Next post will be setting up kubernetes. Thank you for reading.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-2/">Raspberry Pi Kubernetes Cluster - Part 2</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on May 02, 2018.</p> Multi-Environment Deployments with React https://derikwhittaker.blog/2018/04/10/multi-environment-deployments-with-react/ Maintainer of Code, pusher of bits… urn:uuid:4c0ae985-09ac-6d2e-0429-addea1632ea3 Tue, 10 Apr 2018 12:54:17 +0000 If you are using Create-React-App to scaffold your react application there is built in support for changing environment variables based on the NODE_ENV values, this is done by using .env files.  In short this process works by having a .env, .env.production, .env.development set of files.  When you run/build your application CRA will set the NODE_ENV value &#8230; <p><a href="https://derikwhittaker.blog/2018/04/10/multi-environment-deployments-with-react/" class="more-link">Continue reading <span class="screen-reader-text">Multi-Environment Deployments with&#160;React</span></a></p> <p>If you are using <a href="https://github.com/facebook/create-react-app" target="_blank" rel="noopener">Create-React-App</a> to scaffold your react application there is <a href="https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/template/README.md#adding-development-environment-variables-in-env" target="_blank" rel="noopener">built in support</a> for changing environment variables based on the NODE_ENV values, this is done by using .env files.  In short this process works by having a .env, .env.production, .env.development set of files.  When you run/build your application <a href="https://github.com/facebook/create-react-app" target="_blank" rel="noopener">CRA</a> will set the NODE_ENV value to either <code>development</code> or <code>production</code> and based on these values the correct .env file will be used.</p> <p>This works great, when you have a simple deploy setup. But many times in enterprise level applications you need support for more than just 2 environments, many times it is 3-4 environments.  Common logic would suggest that you can accomplish this via the built in mechanism by having additional .env files and changing the NODE_ENV value to the value you care about.  However, CRA does not support this with doing an <code>eject</code>, which will eject all the default conventions and leave it to you to configure your React application.  Maybe this is a good idea, but in my case ejecting was not something I wanted to do.</p> <p>Because I did not want to do an <code>eject</code> I needed to find another solution, and after a fair amount of searching I found a solution that seems to work for me and my needs and is about the amount of effort as I wanted <img src="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f642.png" alt=" Raspberry Pi Kubernetes Cluster - Part 1 https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-1/ Jason Meridth urn:uuid:bd3470f6-97d5-5028-cf12-0751f90915c3 Sat, 07 Apr 2018 14:01:00 +0000 <p>Howdy</p> <p>This is going to be the first post about my setup of a Raspberry Pi Kubernetes Cluster. I saw a post by <a href="https://harthoover.com/kubernetes-1.9-on-a-raspberry-pi-cluster/">Hart Hoover</a> and it finally motivated me to purchase his “grocery list” and do this finally. I’ve been using <a href="https://kubernetes.io/docs/getting-started-guides/minikube/">Minikube</a> for local Kubernetes testing but it doesn’t give you multi-host testing abilities. I’ve also been wanting to get deeper into my Raspberry Pi knowledge. Lots of learning and winning.</p> <p>The items I bought were:</p> <ul> <li>Six <a href="https://smile.amazon.com/dp/B07BFH96M3">Raspberry Pi 3 Model B+ Motherboards</a></li> <li>Six <a href="https://smile.amazon.com/gp/product/B010Q57T02/">SanDisk Ultra 32GB microSDHC UHS-I Card with Adapter, Grey/Red, Standard Packaging (SDSQUNC-032G-GN6MA)</a></li> <li>One <a href="https://smile.amazon.com/gp/product/B011KLFERG/ref=oh_aui_detailpage_o02_s01?ie=UTF8&amp;psc=1">Sabrent 6-Pack 22AWG Premium 3ft Micro USB Cables High Speed USB 2.0 A Male to Micro B Sync and Charge Cables Black CB-UM63</a></li> <li>One <a href="https://smile.amazon.com/gp/product/B01L0KN8OS/ref=oh_aui_detailpage_o02_s01?ie=UTF8&amp;psc=1">AmazonBasics 6-Port USB Wall Charger (60-Watt) - Black</a></li> <li>One <a href="https://smile.amazon.com/gp/product/B01D9130QC/ref=oh_aui_detailpage_o02_s00?ie=UTF8&amp;psc=1">GeauxRobot Raspberry Pi 3 Model B 6-layer Dog Bone Stack Clear Case Box Enclosure also for Pi 2B B+ A+ B A</a></li> <li>One <a href="http://amzn.to/2gNzLzi">Black Box 8-Port Switch</a></li> </ul> <p>Here is the tweet when it all arrived:</p> <div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet"><p lang="en" dir="ltr">I blame <a href="https://twitter.com/hhoover?ref_src=twsrc%5Etfw">@hhoover</a> ;). I will be building my <a href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">@kubernetesio</a> cluster once the 6pi case shows up next Wednesday. The extra pi is to upgrade my <a href="https://twitter.com/RetroPieProject?ref_src=twsrc%5Etfw">@RetroPieProject</a>. Touch screen is an addition I want to try. Side project here I come. <a href="https://t.co/EebIKbsCeH">pic.twitter.com/EebIKbsCeH</a></p>&mdash; Jason Meridth (@jmeridth) <a href="https://twitter.com/jmeridth/status/980075584725422080?ref_src=twsrc%5Etfw">March 31, 2018</a></blockquote> <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </div> <p>I spent this morning finally putting it together.</p> <p>Here is me getting started on the “dogbone case” to hold all of the Raspberry Pis:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_2.jpg" alt="The layout" /></p> <p>The bottom and one layer above:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_3.jpg" alt="The bottom and one layer above" /></p> <p>And the rest:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_4.jpg" alt="3 Layers" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_11.jpg" alt="4 Layers" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_12.jpg" alt="5 Layers" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_13.jpg" alt="6 Layers and Finished" /></p> <p>Different angles completed:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_14.jpg" alt="Finished Angle 2" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_15.jpg" alt="Finished Angle 3" /></p> <p>And connect the power:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_16.jpg" alt="Power" /></p> <p>Next post will be on getting the 6 sandisk cards ready and putting them in and watching the Raspberry Pis boot up and get a green light. Stay tuned.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-1/">Raspberry Pi Kubernetes Cluster - Part 1</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on April 07, 2018.</p> Building AWS Infrastructure with Terraform: S3 Bucket Creation https://derikwhittaker.blog/2018/04/06/building-aws-infrastructure-with-terraform-s3-bucket-creation/ Maintainer of Code, pusher of bits… urn:uuid:cb649524-d882-220f-c253-406a54762705 Fri, 06 Apr 2018 14:28:49 +0000 If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure.  In the AWS word you can use the native CloudFormation solution, but honestly I find this painful and the docs very lacking.  Personally, I prefer Terraform by Hashicorp.  In my experience &#8230; <p><a href="https://derikwhittaker.blog/2018/04/06/building-aws-infrastructure-with-terraform-s3-bucket-creation/" class="more-link">Continue reading <span class="screen-reader-text">Building AWS Infrastructure with Terraform: S3 Bucket&#160;Creation</span></a></p> <p>If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure.  In the AWS word you can use the native <a href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=DChcSEwjD-Lry6KXaAhUMuMAKHTB8AYwYABAAGgJpbQ&amp;ohost=www.google.com&amp;cid=CAESQeD2aF3IUBPQj5YF9K0xmz0FNtIhnq3PzYAHFV6dMZVIirR_psuXDSgkzxZ0jXoyWfpECufNNfbp7JzHQ73TTrQH&amp;sig=AOD64_1b_L781SLpKXqLTFFYIk5Zv3BcHA&amp;q=&amp;ved=0ahUKEwi1l7Hy6KXaAhWD24MKHQXSCQ0Q0QwIJw&amp;adurl=" target="_blank" rel="noopener">CloudFormation</a> solution, but honestly I find this painful and the docs very lacking.  Personally, I prefer <a href="https://www.terraform.io/" target="_blank" rel="noopener">Terraform</a> by <a href="https://www.hashicorp.com/" target="_blank" rel="noopener">Hashicorp</a>.  In my experience the simplicity and easy of use, not to mention the stellar documentation make this the product of choice.</p> <p>This is the initial post in what I hope to be a series of post about how to use Terraform to setup/build AWS Infrastructure.</p> <p>Terrform Documentation on S3 Creation -&gt; <a href="https://www.terraform.io/docs/providers/aws/d/s3_bucket.html" target="_blank" rel="noopener">Here</a></p> <p>In this post I will cover 2 things</p> <ol> <li>Basic bucket setup</li> <li>Bucket setup as Static website</li> </ol> <p>Setting up a basic bucket we can use the following</p> <div class="code-snippet"> <pre class="code-content">resource "aws_s3_bucket" "my-bucket" { bucket = "my-bucket" acl = "private" tags { Any_Tag_Name = "Tag value for tracking" } } </pre> </div> <p>When looking at the example above the only 2 values that are required are bucket and acl.</p> <p>I have added the use of Tags to show you can add custom tags to your bucket</p> <p>Another way to setup an S3 bucket is to act as a Static Web Host.   Setting this up takes a bit more configuration, but not a ton.</p> <div class="code-snippet"> <pre class="code-content">resource "aws_s3_bucket" "my-website-bucket" { bucket = "my-website-bucket" acl = "public-read" website { index_document = "index.html" error_document = "index.html" } policy = &lt;&lt;POLICY { "Version": "2012-10-17", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-website-bucket/*" } ] } POLICY tags { Any_Tag_Name = "Tag value for tracking" } } </pre> </div> <p>The example above has 2 things that need to be pointed out.</p> <ol> <li>The website settings.  Make sure you setup the correct pages here for index/error</li> </ol> <p>The Policy settings.  Here I am using just basic policy.  You can of course setup any policy here you want/need.</p> <p>As you can see, setting up S3 buckets is very simple and straight forward.</p> <p><strong><em>*** Reminder: S3 bucket names MUST be globally unique ***</em></strong></p> <p>Till next time,</p> SSH - Too Many Authentication Failures https://blog.jasonmeridth.com/posts/ssh-too-many-authentication-failures/ Jason Meridth urn:uuid:d7fc1034-1798-d75e-1d61-84fac635dda4 Wed, 28 Mar 2018 05:00:00 +0000 <h1 id="problem">Problem</h1> <p>I started seeing this error recently and had brain farted on why.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">Received disconnect from 123.123.132.132: Too many authentication failures <span class="k">for </span>hostname</code></pre></figure> <p>After a bit of googling it came back to me. This is because I’ve loaded too many keys into my ssh-agent locally (<code class="highlighter-rouge">ssh-add</code>). Why did you do that? Well, because it is easier than specifying the <code class="highlighter-rouge">IdentityFile</code> on the cli when trying to connect. But there is a threshhold. This is set by the ssh host by the <code class="highlighter-rouge">MaxAuthTries</code> setting in <code class="highlighter-rouge">/etc/ssh/sshd_config</code>. The default is 6.</p> <h1 id="solution-1">Solution 1</h1> <p>Clean up the keys in your ssh-agent.</p> <p><code class="highlighter-rouge">ssh-add -l</code> lists all the keys you have in your ssh-agent <code class="highlighter-rouge">ssh-add -d key</code> deletes the key from your ssh-agent</p> <h1 id="solution-2">Solution 2</h1> <p>You can solve this on the command line like this:</p> <p><code class="highlighter-rouge">ssh -o IdentitiesOnly=yes -i ~/.ssh/example_rsa foo.example.com</code></p> <p>What is IdentitiesOnly? Explained in Solution 3 below.</p> <h1 id="solution-3-best">Solution 3 (best)</h1> <p>Specifiy, explicitly, which key goes to which host(s) in your <code class="highlighter-rouge">.ssh/config</code> file.</p> <p>You need to configure which key (“IdentityFile”) goes with which domain (or host). You also want to handle the case when the specified key doesn’t work, which would usually be because the public key isn’t in ~/.ssh/authorized_keys on the server. The default is for SSH to then try any other keys it has access to, which takes us back to too many attempts. Setting “IdentitiesOnly” to “yes” tells SSH to only try the specified key and, if that fails, fall through to password authentication (presuming the server allows it).</p> <p>Your ~/.ssh/config would look like:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host *.myhost.com IdentitiesOnly yes IdentityFile ~/.ssh/myhost Host secure.myhost.com IdentitiesOnly yes IdentityFile ~/.ssh/mysecurehost_rsa Host *.myotherhost.domain IdentitiesOnly yes IdentityFile ~/.ssh/myotherhost_rsa </code></pre></div></div> <p><code class="highlighter-rouge">Host</code> is the host the key can connect to <code class="highlighter-rouge">IdentitiesOnly</code> means only to try <em>this</em> specific key to connect, no others <code class="highlighter-rouge">IdentityFile</code> is the path to the key</p> <p>You can try multiple keys if needed</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host *.myhost.com IdentitiesOnly yes IdentityFile ~/.ssh/myhost_rsa IdentityFile ~/.ssh/myhost_dsa </code></pre></div></div> <p>Hope this helps someone else.</p> <p>Cheers!</p> <p><a href="https://blog.jasonmeridth.com/posts/ssh-too-many-authentication-failures/">SSH - Too Many Authentication Failures</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on March 28, 2018.</p> Clear DNS Cache In Chrome https://blog.jasonmeridth.com/posts/clear-dns-cache-in-chrome/ Jason Meridth urn:uuid:6a2c8c0b-c91b-5f7d-dbc7-8065f0a2f1fd Tue, 27 Mar 2018 20:42:00 +0000 <p>I’m blogging this because I keep forgetting how to do it. Yeah, yeah, I can google it. I run this blog so I know it is always available…..anywho.</p> <p>Go To:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chrome://net-internals/#dns </code></pre></div></div> <p>Click “Clear host cache” button</p> <p><img src="https://blog.jasonmeridth.com/images/clear_dns_cache_in_chrome.png" alt="clear_dns_cache_in_chrome" /></p> <p>Hope this helps someone else.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/clear-dns-cache-in-chrome/">Clear DNS Cache In Chrome</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on March 27, 2018.</p> Create Docker Container from Errored Container https://blog.jasonmeridth.com/posts/create-docker-container-from-errored-container/ Jason Meridth urn:uuid:33d5a6b5-4c48-ae06-deb6-a505edc6b427 Mon, 26 Mar 2018 03:31:00 +0000 <p>When I’m trying to “dockerize” an applciation I usually have to work through some wonkiness.</p> <p>To diagnose a container that has errored out, I, obviously, look at the logs via <code class="highlighter-rouge">docker logs -f [container_name]</code>. That is sometimes helpful. It will, at minimum tell me where I need to focus on the new container I’m going to create.</p> <p><img src="https://blog.jasonmeridth.com/images/diagnose.jpg" alt="diagnose" /></p> <p>Pre-requisites to being able to build a diagnosis container:</p> <ul> <li>You need to use <code class="highlighter-rouge">CMD</code>, <em>not</em> <code class="highlighter-rouge">ENTRYPOINT</code> in the Dockerfile <ul> <li>with <code class="highlighter-rouge">CMD</code> you’ll be able to start a shell, with <code class="highlighter-rouge">ENTRYPOINT</code> your diagnosis container will just keep trying to run that</li> </ul> </li> </ul> <p>To create a diagnosis container, do the following:</p> <ul> <li>Check your failed container ID by <code class="highlighter-rouge">docker ps -a</code></li> <li>Create docker image form the container with <code class="highlighter-rouge">docker commit</code> <ul> <li>example: <code class="highlighter-rouge">docker commit -m "diagnosis" [failed container id]</code></li> </ul> </li> <li>Check the newly create docker image ID by <code class="highlighter-rouge">docker images</code></li> <li><code class="highlighter-rouge">docker run -it [new container image id] sh</code> <ul> <li>this takes you into a container immediately after the error occurred.</li> </ul> </li> </ul> <p>Hope this helps someone else.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/create-docker-container-from-errored-container/">Create Docker Container from Errored Container</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on March 25, 2018.</p> Log Early, Log Often… Saved my butt today https://derikwhittaker.blog/2018/03/21/log-early-log-often-saved-my-butt-today/ Maintainer of Code, pusher of bits… urn:uuid:395d9800-e7ce-27fd-3fc1-5e68628bc161 Wed, 21 Mar 2018 13:16:03 +0000 In a prior posting (AWS Lambda:Log Early Log often, Log EVERYTHING) I wrote about the virtues and value about having really in depth logging, especially when working with cloud services.  Well today this logging saved my ASS a ton of detective work. Little Background I have a background job (Lambda that is called on a schedule) &#8230; <p><a href="https://derikwhittaker.blog/2018/03/21/log-early-log-often-saved-my-butt-today/" class="more-link">Continue reading <span class="screen-reader-text">Log Early, Log Often&#8230; Saved my butt&#160;today</span></a></p> <p>In a prior <a href="https://derikwhittaker.blog/2018/03/06/aws-lambda-log-early-log-often-log-everything/" target="_blank" rel="noopener">posting (AWS Lambda:Log Early Log often, Log EVERYTHING)</a> I wrote about the virtues and value about having really in depth logging, especially when working with cloud services.  Well today this logging saved my ASS a ton of detective work.</p> <p><strong>Little Background</strong><br /> I have a background job (Lambda that is called on a schedule) to create/update data cache in a <a href="https://aws.amazon.com/dynamodb/" target="_blank" rel="noopener">DynamoDB</a> table.  Basically this job will pull data from one data source and attempt to push it as create/update/delete to our Dynamo table.</p> <p>Today when I was running our application I noticed things were not loading right, in fact I had javascript errors because of null reference errors.  I knew that the issue had to be in our data, but was not sure what was wrong.  If I had not had a ton of logging (debug and info) I would have had to run our code locally and step though/debug code for hundreds of items of data.</p> <p>However, because of in depth logging I was able to quickly go to <a href="https://aws.amazon.com/cloudwatch/" target="_blank" rel="noopener">CloudWatch</a> and filter on a few key words and narrow hundreds/thousands of log entries down to 5.  Once I had these 5 entries I was able to expand a few of those entries and found the error within seconds.</p> <p>Total time to find the error was less than 5 minutes and I never opened a code editor or stepped into code.</p> <p>The moral of this story, because I log everything, including data (no PII of course) I was able to quickly find the source of the error.  Now to fix the code&#8230;.</p> <p>Till next time,</p> AWS Lambda: Log early, Log often, Log EVERYTHING https://derikwhittaker.blog/2018/03/06/aws-lambda-log-early-log-often-log-everything/ Maintainer of Code, pusher of bits… urn:uuid:6ee7f59b-7f4c-1312-bfff-3f9c46ec8701 Tue, 06 Mar 2018 14:00:58 +0000 In the world of building client/server applications logs are important.  They are helpful when trying to see what is going on in your application.  I have always held the belief  that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code. But lets &#8230; <p><a href="https://derikwhittaker.blog/2018/03/06/aws-lambda-log-early-log-often-log-everything/" class="more-link">Continue reading <span class="screen-reader-text">AWS Lambda: Log early, Log often, Log&#160;EVERYTHING</span></a></p> <p>In the world of building client/server applications logs are important.  They are helpful when trying to see what is going on in your application.  I have always held the belief  that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code.</p> <p>But lets be honest, in most cases when building client/server applications logs are an afterthought.  Often this is because you can pretty easily (in most cases) debug your application and step through the code.</p> <p>When building a <a href="https://aws.amazon.com/serverless/" target="_blank" rel="noopener">serverless</a> applications with technologies like <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener">AWS Lambda</a> functions (holds true for Azure Functions as well) your logging game really needs to step up.</p> <p>The reason for this is that you cannot really debug your Lambda in the wild (you can to some degree locally with AWS SAM or the Serverless framework).  Because of this you need produce detailed enough logs to allow you to easily determine the WHAT and WHERE.</p> <p>When I build my serverless functions I have a few guidelines I follow</p> <ol> <li>Info Log calls to methods, output argument data (make sure no <a href="https://en.wikipedia.org/wiki/Personally_identifiable_information" target="_blank" rel="noopener">PII</a>/<a href="https://en.wikipedia.org/wiki/Protected_health_information" target="_blank" rel="noopener">PHI</a>)</li> <li>Error Log any failures (in try/catch or .catch for promises)</li> <li>Debug Log any critical decision points</li> <li>Info Log exit calls at top level methods</li> </ol> <p>I also like to setup a simple and consistent format for my logs.  The example I follow for my Lambda logs is as seen below</p> <div class="code-snippet"> <pre class="code-content">timestamp: [logLevel] : [Class.Method] - message {data points} </pre> </div> <p>I have found that if I follow these general guidelines the pain of determine failure points in serverless environments is heavily reduced.</p> <p>Till next time,</p> Sinon Error: Attempted to wrap undefined property ‘XYZ as function https://derikwhittaker.blog/2018/02/27/sinon-error-attempted-to-wrap-undefined-property-xyz-as-function/ Maintainer of Code, pusher of bits… urn:uuid:b41dbd54-3804-6f6d-23dc-d2a04635033a Tue, 27 Feb 2018 13:45:29 +0000 I ran into a fun little error recently when working on a ReactJs application.  In my application I was using SinonJs to setup some spies on a method, I wanted to capture the input arguments for verification.  However, when I ran my test I received the following error. Attempted to wrap undefined property handlOnAccountFilter as &#8230; <p><a href="https://derikwhittaker.blog/2018/02/27/sinon-error-attempted-to-wrap-undefined-property-xyz-as-function/" class="more-link">Continue reading <span class="screen-reader-text">Sinon Error: Attempted to wrap undefined property &#8216;XYZ as&#160;function</span></a></p> <p>I ran into a fun little error recently when working on a <a href="https://reactjs.org/" target="_blank" rel="noopener">ReactJs</a> application.  In my application I was using <a href="http://sinonjs.org/" target="_blank" rel="noopener">SinonJs</a> to setup some spies on a method, I wanted to capture the input arguments for verification.  However, when I ran my test I received the following error.</p> <blockquote><p>Attempted to wrap undefined property handlOnAccountFilter as function</p></blockquote> <p>My method under test is setup as such</p> <div class="code-snippet"> <pre class="code-content">handleOnAccountFilter = (filterModel) =&gt; { // logic here } </pre> </div> <p>I was using the above syntax is the <a href="https://github.com/jeffmo/es-class-public-fields" target="_blank" rel="noopener">proposed class property</a> feature, which will automatically bind the <code>this</code> context of the class to my method.</p> <p>My sinon spy is setup as such</p> <div class="code-snippet"> <pre class="code-content">let handleOnAccountFilterSpy = null; beforeEach(() =&gt; { handleOnAccountFilterSpy = sinon.spy(AccountsListingPage.prototype, 'handleOnAccountFilter'); }); afterEach(() =&gt; { handleOnAccountFilterSpy.restore(); }); </pre> </div> <p>Everything looked right, but I was still getting this error.  It turns out that this error is due in part in the way that the Class Property feature implements the handlOnAccountFilter.  When you use this feature the method/property is added to the class as an instance method/property, not as a prototype method/property.  This means that sinon is not able to gain access to it prior to creating an instance of the class.</p> <p>To solve my issue I had to make a change in the implementation to the following</p> <div class="code-snippet"> <pre class="code-content">handleOnAccountFilter(filterModel) { // logic here } </pre> </div> <p>After make the above change I needed to determine how I wanted to bind <code>this</code> to my method (Cory show 5 ways to do this <a href="https://medium.freecodecamp.org/react-binding-patterns-5-approaches-for-handling-this-92c651b5af56" target="_blank" rel="noopener">here</a>).  I chose to bind <code>this</code> inside the constructor as below</p> <div class="code-snippet"> <pre class="code-content">constructor(props){ super(props); this.handleOnAccountFilter = this.handleOnAccountFilter.bind(this); } </pre> </div> <p>I am not a huge fan of having to do this (pun intended), but oh well.  This solved my issues.</p> <p>Till next time</p> Ensuring componentDidMount is not called in Unit Tests https://derikwhittaker.blog/2018/02/22/ensuring-componentdidmount-is-not-called-in-unit-tests/ Maintainer of Code, pusher of bits… urn:uuid:da94c1a3-2de4-a90c-97f5-d7361397a33c Thu, 22 Feb 2018 19:45:53 +0000 If you are building a ReactJs you will often times implement componentDidMount on your components.  This is very handy at runtime, but can pose an issue for unit tests. If you are building tests for your React app you are very likely using enzyme to create instances of your component.  The issue is that when enzyme creates &#8230; <p><a href="https://derikwhittaker.blog/2018/02/22/ensuring-componentdidmount-is-not-called-in-unit-tests/" class="more-link">Continue reading <span class="screen-reader-text">Ensuring componentDidMount is not called in Unit&#160;Tests</span></a></p> <p>If you are building a <a href="https://reactjs.org/" target="_blank" rel="noopener">ReactJs</a> you will often times implement <code>componentDidMount</code> on your components.  This is very handy at runtime, but can pose an issue for unit tests.</p> <p>If you are building tests for your React app you are very likely using <a href="http://airbnb.io/projects/enzyme/" target="_blank" rel="noopener">enzyme</a> to create instances of your component.  The issue is that when enzyme creates the component it invokes the lifecyle methods, like <code>componentDidMount</code>.  Sometimes we do not want this to be called, but how to suppress this?</p> <p>I have found 2 different ways to suppress/mock <code>componentDidMount</code>.</p> <p>Method one is to redefine <code>componentDidMount</code> on your component for your tests.  This could have interesting side effects so use with caution.</p> <div class="code-snippet"> <pre class="code-content"> describe('UsefullNameHere', () =&gt; { beforeAll(() =&gt; { YourComponent.prototype.componentDidMount = () =&gt; { // can omit or add custom logic }; }); }); </pre> </div> <p>Basically above I am just redefining the componentDidMount method on my component.  This works and allows you to have custom logic.  Be aware that when doing above you will have changed the implementation for your component for the lifetime of your test session.</p> <p>Another solution is to use a mocking framework like <a href="http://sinonjs.org/" target="_blank" rel="noopener">SinonJs</a>.  With Sinon you can stub out the <code>componentDidMount</code> implementation as seen below</p> <div class="code-snippet"> <pre class="code-content"> describe('UsefullNameHere', () =&gt; { let componentDidMountStub = null; beforeAll(() =&gt; { componentDidMountStub = sinon.stub(YourComponent.prototype, 'componentDidMount').callsFake(function() { // can omit or add custom logic }); }); afterAll(() =&gt; { componentDidMountStub.restore(); }); }); </pre> </div> <p>Above I am using .stub to redefine the method.  I also added .<a href="http://sinonjs.org/releases/v4.3.0/stubs/" target="_blank" rel="noopener">callsFake</a>() but this can be omitted if you just want to ignore the call.  You will want to make sure you restore your stub via the afterAll, otherwise you will have stubbed out the call for the lifetime of your test session.</p> <p>Till next time,</p> Los Techies Welcomes Derik Whittaker https://lostechies.com/derekgreer/2018/02/21/los-techies-welcomes-derik-whittaker/ Los Techies urn:uuid:adc9a1c8-48ea-3bea-1aa7-320d51db12a1 Wed, 21 Feb 2018 11:00:00 +0000 Los Techies would like to introduce, and extend a welcome to Derik Whittaker. Derik is a C# MVP, member of the AspInsiders group, community speaker, and Pluralsight author. Derik was previously a contributor at CodeBetter.com. Welcome, Derik! <p>Los Techies would like to introduce, and extend a welcome to Derik Whittaker. Derik is a C# MVP, member of the AspInsiders group, community speaker, and Pluralsight author. Derik was previously a contributor at <a href="http://codebetter.com/">CodeBetter.com</a>. Welcome, Derik!</p> Ditch the Repository Pattern Already https://lostechies.com/derekgreer/2018/02/20/ditch-the-repository-pattern-already/ Los Techies urn:uuid:7fab2063-d833-60ce-9e46-e4a413ec8391 Tue, 20 Feb 2018 21:00:00 +0000 One pattern that still seems particularly common among .Net developers is the Repository pattern. I began using this pattern with NHibernate around 2006 and only abandoned its use a few years ago. <p>One pattern that still seems particularly common among .Net developers is the <a href="https://martinfowler.com/eaaCatalog/repository.html">Repository pattern.</a> I began using this pattern with NHibernate around 2006 and only abandoned its use a few years ago.</p> <p>I had read several articles over the years advocating abandoning the Repository pattern in favor of other suggested approaches which served as a pebble in my shoe for a few years, but there were a few design principles whose application seemed to keep motivating me to use the pattern.  It wasn’t until a change of tooling and a shift in thinking about how these principles should be applied that I finally felt comfortable ditching the use of repositories, so I thought I’d recount my journey to provide some food for thought for those who still feel compelled to use the pattern.</p> <h2 id="mental-obstacle-1-testing-isolation">Mental Obstacle 1: Testing Isolation</h2> <p>What I remember being the biggest barrier to moving away from the use of repositories was writing tests for components which interacted with the database.  About a year or so before I actually abandoned use of the pattern, I remember trying to stub out a class derived from Entity Framework’s DbContext after reading an anti-repository blog post.  I don’t remember the details now, but I remember it being painful and even exploring use of a 3rd-party library designed to help write tests for components dependent upon Entity Framework.  I gave up after a while, concluding it just wasn’t worth the effort.  It wasn’t as if my previous approach was pain-free, as at that point I was accustomed to stubbing out particularly complex repository method calls, but as with many things we often don’t notice friction to which we’ve become accustomed for one reason or another.  I had assumed that doing all that work to stub out my repositories was what I should be doing.</p> <p>Another principle that I picked up from somewhere (maybe the big <a href="http://xunitpatterns.com/">xUnit Test Patterns</a> book? … I don’t remember) that seemed to keep me bound to my repositories was that <a href="http://aspiringcraftsman.com/2012/04/01/tdd-best-practices-dont-mock-others/">you shouldn’t write tests that depend upon dependencies you don’t own</a>.  I believed at the time that I should be writing tests for Application Layer services (which later morphed into discrete dispatched command handlers) and the idea of stubbing out either NHIbernate or Entity Framework violated my sensibilities.</p> <h2 id="mental-obstacle-2-the-dependency-inversion-principle-adherence">Mental Obstacle 2: The Dependency Inversion Principle Adherence</h2> <p>The Dependency Inversion Principle seems to be a source of confusion for many which stems in part from the similarity of wording with the practice of <a href="https://lostechies.com/derickbailey/2011/09/22/dependency-injection-is-not-the-same-as-the-dependency-inversion-principle/">Dependency Injection</a> as well as from the fact that the pattern’s formal definition reflects the platform from whence the principle was conceived (i.e. C++).  One might say that the abstract definition of the Dependency Inversion Principle was too dependent upon the details of its origin (ba dum tss).  I’ve written about the principle a few times (perhaps my most succinct being <a href="https://stackoverflow.com/a/1113937/1219618">this Stack Overflow answer</a>), but put simply, the Dependency Inversion Principle has at its primary goal the decoupling of the portions of your application which define <i>policy</i> from the portions which define <i>implementation</i>.  That is to say, this principle seeks to keep the portions of your application which govern what your application does (e.g. workflow, business logic, etc.) from being tightly coupled to the portions of your application which govern the low level details of how it gets done (e.g. persistence to an Sql Server database, use of Redis for caching, etc.).</p> <p>A good example of a violation of this principle, which I recall from my NHibernate days, was that once upon a time NHibernate was tightly coupled to log4net.  This was later corrected, but at one time the NHibernate assembly had a hard dependency on log4net.  You could use a different logging library for your own code if you wanted, and you could use binding redirects to use a different version of log4net if you wanted, but at one time if you had a dependency on NHibernate then you had to deploy the log4net library.  I think this went unnoticed by many due to the fact that most developers who used NHibernate also used log4net.</p> <p>When I first learned about the principle, I immediately recognized that it seemed to have limited advertized value for most business applications in light of what Udi Dahan labeled<a href="http://udidahan.com/2009/06/07/the-fallacy-of-reuse/"> The Fallacy Of ReUse</a>.  That is to say, <i>properly understood</i>, the Dependency Inversion Principle has as its primary goal the reuse of components and keeping those components decoupled from dependencies which would keep them from being easily reused with other implementation components, but your application and business logic isn’t something that is likely to ever be reused in a different context.  The take away from that is basically that the advertized value of adhering to the Dependency Inversion Principle is really more applicable to libraries like NHibernate, Automapper, etc. and not so much to that workflow your team built for Acme Inc.’s distribution system.  Nevertheless, the Dependency Inversion Principle had a practical value of implementing an architecture style Jeffrey Palermo labeled <a href="http://jeffreypalermo.com/blog/the-onion-architecture-part-1/">the Onion Architecture.</a> Specifically, in contrast to <a href="https://msdn.microsoft.com/en-us/library/ff650258.aspx"> traditional 3-layered architecture models</a> where UI, Business, and Data Access layers precluded using something like <a href="https://msdn.microsoft.com/en-us/library/ff648105.aspx?f=255&amp;MSPPError=-2147217396">Data Access Logic Components</a> to encapsulate an ORM to map data directly to entities within the Business Layer, inverting the dependencies between the Business Layer and the Data Access layer provided the ability for the application to interact with the database while also <i>seemingly </i>abstracting away the details of the data access technology used.</p> <p>While I always saw the fallacy in strictly trying to apply the Dependency Inversion Principle to invert the implementation details of how I got my data from my application layer so that I’d someday be able to use the application in a completely different context, it seemed the academically astute and in vogue way of doing Domain-driven Design at the time, seemed consistent with the GoF’s advice to program to an interface rather than an implementation, and provided an easier way to write isolation tests than trying to partially stub out ORM types.</p> <h2 id="the-catalyst">The Catalyst</h2> <p>For the longest time, I resisted using Entity Framework.  I had become fairly proficient at using NHibernate and I just saw it as plain stupid to use a framework that was years behind NHibernate in features and maturity, especially when it had such a steep learning curve.  A combination of things happened, though.  A lot of the NHibernate supporters (like many within the Alt.Net crowd) moved on to other platforms like Ruby and Node; anything with Microsoft’s name on it eventually seems to gain market share whether it’s better or not; and Entity Framework eventually did seem to mostly catch up with NHibernate in features, and surpassed it in some areas. So, eventually I found it impossible to avoid using Entity Framework which led to me trying to apply the same patterns I’d used before with this newer-to-me framework.</p> <p>To be honest, everything mostly worked, especially for the really simple stuff.  Eventually, though, I began to see little ways I had to modify my abstraction to accommodate differences in how Entity Framework did things from how NHibernate did things.  What I discovered was that, while my repositories allowed my application code to be physically decoupled from the ORM, the way I was using the repositories was in small ways semantically coupled to the framework.  I wish I had kept some sort of record every time I ran into something, as the only real thing I can recall now were motivations with certain design approaches to expose the SaveChanges method for <a href="https://lostechies.com/derekgreer/2015/11/01/survey-of-entity-framework-unit-of-work-patterns/"> Unit of Work implementations</a> I don’t want to make more of the semantic coupling argument against repositories than it’s worth, but observing little places where <a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">my abstractions were leaking</a>, combined with the pebble in my shoe of developers who I felt were far better than me were saying I shouldn’t use them lead me to begin rethinking things.</p> <h2 id="more-effective-testing-strategies">More Effective Testing Strategies</h2> <p>It was actually a few years before I stopped using repositories that I stopped stubbing out repositories.  Around 2010, I learned that you can use Test-Driven Development to achieve 100% test coverage for the code for which you’re responsible, but when you plug your code in for the first time with that team that wasn’t designing to the same specification and not writing any tests at all that things may not work.  It was then that I got turned on to Acceptance Test Driven Development.  What I found was that writing high-level subcutaneous tests (i.e. skipping the UI layer, but otherwise end-to-end) was overall easier, was possible to align with acceptance criteria contained within a user story, provided more assurance that everything worked as a whole, and was easier to get teams on board with.  Later on, I surmised that I really shouldn’t have been writing isolation tests for components which, for the most part, are just specialized facades anyway.  All an isolation test for a facade really says is “did I delegate this operation correctly” and if you’re not careful you can end up just writing a whole bunch of tests that basically just validate whether you correctly configured your mocking library.</p> <p>So, by the time I started rethinking my use of repositories, I had long since stopped using them for test isolation.</p> <h2 id="taking-the-plunge">Taking the Plunge</h2> <p>It was actually about a year after I had become convinced that repositories were unnecessary, useless abstractions that I started working with a new codebase I had the opportunity to steer.  Once I eliminated them from the equation, everything got so much simpler.   Having been repository-free for about two years now, I think I’d have a hard time joining a team that had an affinity for them.</p> <h2 id="conclusion">Conclusion</h2> <p>If you’re still using repositories and you don’t have some other hangup you still need to get over like writing unit tests for your controllers or application services then give the repository-free lifestyle a try.  I bet you’ll love it.</p> Using Manual Mocks to test the AWS SDK with Jest https://derikwhittaker.blog/2018/02/20/using-manual-mocks-to-test-the-aws-sdk-with-jest/ Maintainer of Code, pusher of bits… urn:uuid:3a424860-3707-7327-2bb1-a60b9f3be47d Tue, 20 Feb 2018 13:56:45 +0000 Anytime you build Node applications it is highly suggested that your cover your code with tests.  When your code interacts with 3rd party API&#8217;s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests &#8230; <p><a href="https://derikwhittaker.blog/2018/02/20/using-manual-mocks-to-test-the-aws-sdk-with-jest/" class="more-link">Continue reading <span class="screen-reader-text">Using Manual Mocks to test the AWS SDK with&#160;Jest</span></a></p> <p>Anytime you build Node applications it is highly suggested that your cover your code with tests.  When your code interacts with 3rd party API&#8217;s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests not unit tests.</p> <p>If you are using <a href="http://bit.ly/jest-get-started" target="_blank" rel="noopener">Jest</a>, one solution is utilize the built in support for <a href="http://bit.ly/jest-manual-mocks" target="_blank" rel="noopener">manual mocks.</a>  I have found the usage of manual mocks invaluable while testing 3rd party API&#8217;s such as the AWS.  Keep in mind just because I am using manual mocks this will remove the need for using libraries like <a href="http://bit.ly/sinon-js" target="_blank" rel="noopener">SinonJs</a> (a JavaScript framework for creating stubs/mocks/spies).</p> <p>The way that manual mocks work in Jest is as follows (from the Jest website&#8217;s documentation).</p> <blockquote><p><em>Manual mocks are defined by writing a module in a <code>__mocks__/</code> subdirectory immediately adjacent to the module. For example, to mock a module called <code>user</code> in the <code>models</code> directory, create a file called <code>user.js</code> and put it in the <code>models/__mocks__</code> directory. Note that the <code>__mocks__</code> folder is case-sensitive, so naming the directory <code>__MOCKS__</code> will break on some systems. If the module you are mocking is a node module (eg: <code>fs</code>), the mock should be placed in the <code>__mocks__</code> directory adjacent to <code>node_modules</code> (unless you configured <a href="https://facebook.github.io/jest/docs/en/configuration.html#roots-array-string"><code>roots</code></a> to point to a folder other than the project root).</em></p></blockquote> <p>In my case I want to mock out the usage of the <a href="http://bit.ly/npm-aws-sdk" target="_blank" rel="noopener">AWS-SDK</a> for <a href="http://bit.ly/aws-sdk-node" target="_blank" rel="noopener">Node</a>.</p> <p>To do this I created a __mocks__ folder at the root of my solution.  I then created a <a href="http://bit.ly/gist-aws-sdk-js" target="_blank" rel="noopener">aws-sdk.js</a> file inside this folder.</p> <p>Now that I have my mocks folder created with a aws-sdk.js file I am able to consume my manual mock in my jest test by simply referencing the aws-sdk via a <code>require('aws-sdk')</code> command.</p> <div class="code-snippet"> <pre class="code-content">const AWS = require('./aws-sdk'); </pre> </div> <p>With declaration of AWS above my code is able to a use the <a href="http://bit.ly/npm-aws-sdk" target="_blank" rel="noopener">NPM </a>package during normal usage, or my aws-sdk.js mock when running under the Jest context.</p> <p>Below is a small sample of the code I have inside my aws-sdk.js file for my manual mock.</p> <div class="code-snippet"> <pre class="code-content">const stubs = require('./aws-stubs'); const AWS = {}; // This here is to allow/prevent runtime errors if you are using // AWS.config to do some runtime configuration of the library. // If you do not need any runtime configuration you can omit this. AWS.config = { setPromisesDependency: (arg) =&gt; {} }; AWS.S3 = function() { } // Because I care about using the S3 service's which are part of the SDK // I need to setup the correct identifier. // AWS.S3.prototype = { ...AWS.S3.prototype, // Stub for the listObjectsV2 method in the sdk listObjectsV2(params){ const stubPromise = new Promise((resolve, reject) =&gt; { // pulling in stub data from an external file to remove the noise // from this file. See the top line for how to pull this in resolve(stubs.listObjects); }); return { promise: () =&gt; { return stubPromise; } } } }; // Export my AWS function so it can be referenced via requires module.exports = AWS; </pre> </div> <p>A few things to point out in the code above.</p> <ol> <li>I chose to use the <a href="http://bit.ly/sdk-javascript-promises" target="_blank" rel="noopener">promise</a>s implementation of the listObjectsV2.  Because of this I need to return a promise method as my result on my listObjectsV2 function.  I am sure there are other ways to accomplish this, but this worked and is pretty easy.</li> <li>My function is returning stub data, but this data is described in a separate file called aws-stubs.js which sites along side of my aws-sdk.js file.  I went this route to remove the noise of having the stub data inside my aws-adk file.  You can see a full example of this <a href="http://bit.ly/gist-aws-stub-data" target="_blank" rel="noopener">here</a>.</li> </ol> <p>Now that I have everything setup my tests will no longer attempt to hit the actually aws-sdk, but when running in non-test mode they will.</p> <p>Till next time,</p> Configure Visual Studio Code to debug Jest Tests https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/ Maintainer of Code, pusher of bits… urn:uuid:31928626-b984-35f6-bf96-5bfb71e16208 Fri, 16 Feb 2018 21:33:03 +0000 If you have not given Visual Studio Code a spin you really should, especially if  you are doing web/javascript/Node development. One super awesome feature of VS Code is the ability to easily configure the ability to debug your Jest (should work just fine with other JavaScript testing frameworks) tests.  I have found that most of &#8230; <p><a href="https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/" class="more-link">Continue reading <span class="screen-reader-text">Configure Visual Studio Code to debug Jest&#160;Tests</span></a></p> <p>If you have not given <a href="https://code.visualstudio.com/" target="_blank" rel="noopener">Visual Studio Code</a> a spin you really should, especially if  you are doing web/javascript/Node development.</p> <p>One super awesome feature of VS Code is the ability to easily configure the ability to debug your <a href="https://facebook.github.io/jest/" target="_blank" rel="noopener">Jest </a>(should work just fine with other JavaScript testing frameworks) tests.  I have found that most of the time I do not need to actually step into the debugger when writing tests, but there are times that using <code>console.log</code> is just too much friction and I want to step into the debugger.</p> <p>So how do we configure VS Code?</p> <p>First you  will need to install the <a href="https://www.npmjs.com/package/jest-cli" target="_blank" rel="noopener">Jest-Cli</a> NPM package (I am assuming you already have Jest setup to run your tests, if you do not please read the <a href="https://facebook.github.io/jest/docs/en/getting-started.html" target="_blank" rel="noopener">Getting-Started</a> docs).  If you fail to do this step you will get the following error in Code when you try to run the debugger.</p> <p><img data-attachment-id="78" data-permalink="https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/jestcli/" data-orig-file="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640" data-orig-size="702,75" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="JestCLI" data-image-description="" data-medium-file="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640?w=300" data-large-file="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640?w=640" class="alignnone size-full wp-image-78" src="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640" alt="JestCLI" srcset="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640 640w, https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=150 150w, https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=300 300w, https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png 702w" sizes="(max-width: 640px) 100vw, 640px" /></p> <p>After you have Jest-Cli installed you will need to configure VS Code for debugging.  To do this open up the configuration by clicking Debug -&gt; Open Configurations.  This will open up a file called launch.json.</p> <p>Once launch.json is open add the following configuration</p> <div class="code-snippet"> <pre class="code-content"> { "name": "Jest Tests", "type": "node", "request": "launch", "program": "${workspaceRoot}/node_modules/jest-cli/bin/jest.js", "stopOnEntry": false, "args": ["--runInBand"], "cwd": "${workspaceRoot}", "preLaunchTask": null, "runtimeExecutable": null, "runtimeArgs": [ "--nolazy" ], "env": { "NODE_ENV": "development" }, "console": "internalConsole", "sourceMaps": false, "outFiles": [] } </pre> </div> <p>Here is a gist of a working <a href="https://gist.github.com/derikwhittaker/331d4a5befddf7fc6b2599f1ada5d866" target="_blank" rel="noopener">launch.json</a> file.</p> <p>After you save the file you are almost ready to start your debugging.</p> <p>Before you can debug you will want to open the debug menu (the bug icon on the left toolbar).   This will show a drop down menu with different configurations.  Make sure &#8216;Jest Test&#8217; is selected.</p> <p><img data-attachment-id="79" data-permalink="https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/jesttest/" data-orig-file="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640" data-orig-size="240,65" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="JestTest" data-image-description="" data-medium-file="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640?w=240" data-large-file="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640?w=240" class="alignnone size-full wp-image-79" src="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640" alt="JestTest" srcset="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png 240w, https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=150 150w" sizes="(max-width: 240px) 100vw, 240px" /></p> <p>If you have this setup correctly you should be able to set breakpoints and hit F5.</p> <p>Till next time,</p> On Migrating Los Techies to Github Pages https://lostechies.com/derekgreer/2018/02/16/on-migrating-lostechies-to-github-pages/ Los Techies urn:uuid:74de4506-44e0-f605-61cb-8ffe972f6787 Fri, 16 Feb 2018 20:00:00 +0000 We recently migrated Los Techies from a multi-site installation of WordPress to Github Pages, so I thought I’d share some of the more unique portions of the process. For a straightforward guide on migrating from WordPress to Github Pages, Tomomi Imura has published an excellent guide available here that covers exporting content, setting up a new Jekyll site (what Github Pages uses as its static site engine), porting the comments, and DNS configuration. The purpose of this post is really just to cover some of the unique aspects that related to our particular installation. <p>We recently migrated Los Techies from a multi-site installation of WordPress to Github Pages, so I thought I’d share some of the more unique portions of the process. For a straightforward guide on migrating from WordPress to Github Pages, Tomomi Imura has published an excellent guide available <a href="https://girliemac.com/blog/2013/12/27/wordpress-to-jekyll/">here</a> that covers exporting content, setting up a new Jekyll site (what Github Pages uses as its static site engine), porting the comments, and DNS configuration. The purpose of this post is really just to cover some of the unique aspects that related to our particular installation.</p> <h2 id="step-1-exporting-content">Step 1: Exporting Content</h2> <p>Having recently migrated <a href="http://aspiringcraftsman.com">my personal blog</a> from WordPress to Github Pages using the aforementioned guide, I thought the process of doing the same for Los Techies would be relatively easy. Unfortunately, due to the fact that we had a woefully out-of-date installation of WordPress, migrating Los Techies proved to be a bit problematic. First, the WordPress to Jekyll Exporter plugin wasn’t compatible with our version of WordPress. Additionally, our installation of WordPress couldn’t be upgraded in place for various reasons. As a result, I ended up taking the rather labor-intensive path of exporting each author’s content using the default WordPress XML export and then, for each author, importing into an up-to-date installation of WordPress using the hosting site with which I previously hosting my personal blog, exporting the posts using the Jekyll Exporter plugin, and then deleting the posts in preparation for the next iteration. This resulted in a collection of zipped, mostly ready posts for each author.</p> <h2 id="step-2-configuring-authors">Step 2: Configuring Authors</h2> <p>Our previous platform utilized the multi-site features of WordPress to facilitate a single site with multiple contributors. By default, Jekyll looks for content within a special folder in the root of the site named _posts, but there are several issues with trying to represent multiple contributors within the _posts folder. Fortunately Jekyll has a feature called Collections which allows you to set up groups of posts which can each have their own associated configuration properties. Once each of the author’s posts were copied to corresponding collection folders, a series of scripts were written to create author-specific index.html, archive.html, and tags.html files which are used by a custom post layout. Additionally, due to the way the WordPress content was exported, the permalinks generated for each post did not reflect the author’s subdirectory, so another script was written to strip out all the generated permalinks.</p> <h2 id="step-3-correcting-liquid-errors">Step 3: Correcting Liquid Errors</h2> <p>Jekyll uses a language called Liquid as its templating engine. Once all the content was in place, all posts which contained double curly braces were interpreted as Liquid commands which ended up breaking the build process. For that, each offending post had to be edited to wrap the content in Liquid directives {% raw %} … {% endraw %} to keep the content from being interpreted by the Liquid parser. Additionally, there were a few other odd things which were causing issues (such as posts with non-breaking space characters) for which more scripts were written to modify the posts to non-offending content.</p> <h2 id="step-4-enabling-disqus">Step 4: Enabling Disqus</h2> <p>The next step was to get Disqus comments working for the posts. By default, Disqus will use the page URL as the page identifier, so as long as the paths match then enabling Disqus should just work. The WordPress Disqus plugin we were using utilized a unique post id and guid as the Disqus page identifier, so the Disqus javascript had to be configured to use these properties. These values were preserved by the Jekyll exporter, but unfortunately the generated id property in the Jekyll front matter was getting internally overridden by Jekyll so another script had to be written to modify all the posts to rename the properties used for these values. Properties were added to the Collection configuration in the main _config.yml to designate the Disqus shortname for each author and allow people to toggle whether disqus was enabled or disabled for their posts.</p> <h2 id="step-5-converting-gists">Step 5: Converting Gists</h2> <p>Many authors at Los Techies used a Gist WordPress plugin to embed code samples within their posts. Github Pages supports a jekyll-gist plugin, so another script was written to modify all the posts to use Liquid syntax to denote the gists. This mostly worked, but there were still a number of posts which had to be manually edited to deal with different ways people were denoting their gists. In retrospect, it would have been better to use JavaScript rather than the Jekyll gist plugin due to the size of the Los Techies site. Every plugin you use adds time to the overall build process which can become problematic as we’ll touch on next.</p> <h2 id="step-6-excessive-build-time-mitigation">Step 6: Excessive Build-time Mitigation</h2> <p>The first iteration of the conversion used the Liquid syntax for generating the sidebar content which lists recent site-wide posts, recent author-specific posts, and the list of contributing authors. This resulted in extremely long build times, but it worked and who cares once the site is rendered, right? Well, what I found out was that Github has a hard cut off of 10 minutes for Jekyll site builds. If your site doesn’t build within 10 minutes, the process gets killed. At first I thought “Oh no! After all this effort, Github just isn’t going to support a site our size!” I then realized that rather than having every page loop over all the content, I could create a Jekyll template to generate JSON content one time and then use JavaScript to retrieve the content and dynamically generate the sidebar DOM elements. This sped up the build significantly, taking the build from close to a half-hour to just a few minutes.</p> <h2 id="step-8-converting-wordpress-uploaded-content">Step 8: Converting WordPress Uploaded Content</h2> <p>Another headache that presented itself is how WordPress represented uploaded content. Everything that anyone had ever uploaded to the site for images and downloads used within their posts were stored in a cryptic folder structure. Each folder had to be interrogated to see which files contained therein matched what author, the folder structure had to be reworked to accommodate the nature of the Jekyll site, and more scripts had to be written to edit everyone’s posts to change paths to the new content. Of course, the scripts only worked for about 95% of the posts, a number of posts had to be edited manually to fix things like non-printable characters being used in file names, etc.</p> <h2 id="step-9-handling-redirects">Step 9: Handling Redirects</h2> <p>The final step to get the initial version of the conversion complete was to handle redirects which were formally being handled by .httpacess. The Los Techies site started off using Community Server prior to migrating to WordPress and redirects were set up using .httpaccess to maintain the paths to all the previous content locations. Github Pages doesn’t support .httpaccess, but it does support a Jekyll redirect plugin. Unfortunately, it requires adding a redirect property to each post requiring a redirect and we had several thousand, so I had to write another script to read the .httpaccess file and figure out which post went with each line. Another unfortunate aspect of using the Jekyll redirect plugin is that it adds overhead to the build time which, as discussed earlier, can become an issue.</p> <h2 id="step-10-enabling-aggregation">Step 10: Enabling Aggregation</h2> <p>Once the conversion was complete, I decided to dedicate some time to figuring out how we might be able to add the ability to aggregate posts from external feeds. The first step to this was finding a service that could aggregate feeds together. You might think there would be a number of things that do this, and while I did find at least a half-dozen services, there were only a couple I found that allowed you to maintain a single feed and add/remove new feeds while preserving the aggregated feed. Most seemed to only allow you to do a one-time aggregation. For this I settled on a site named <a href="http://feed.informer.com">feed.informer.com</a>. Next, I replaced the landing page with JavaScript that dynamically built the site from the aggregated feed along with replacing the recent author posts section that did the same and a special external template capable of making an individual post look like it’s actually hosted on Los Techies. The final result was a site that displays a mixture of local content along with aggregated content.</p> <h2 id="conclusion">Conclusion</h2> <p>Overall, the conversion was way more work than I anticipated, but I believe worth the effort. The site is now much faster than it used to be and we aren’t having to pay a hosting service to host our site.</p> Going Async with Node AWS SDK with Express https://derikwhittaker.blog/2018/02/13/going-async-with-node-aws-sdk-with-express/ Maintainer of Code, pusher of bits… urn:uuid:d4750cda-8c6e-8b2f-577b-78c746ee6ebd Tue, 13 Feb 2018 13:00:30 +0000 When building applications in Node/Express you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary.  The 'old school' way was to use call backs, which often led to callback hell.  Than came along Promises which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything.  Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for async/await was introduced and this really has cleaned up and enhanced the readability of your code. <p>When building applications in <a href="https://nodejs.org/en/" target="_blank" rel="noopener">Node</a>/<a href="http://expressjs.com/" target="_blank" rel="noopener">Express </a>you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary.  The &#8216;old school&#8217; way was to use call backs, which often led to <a href="http://callbackhell.com/" target="_blank" rel="noopener">callback hell</a>.  Than came along <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promises</a> which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything.  Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function" target="_blank" rel="noopener">async</a>/<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await" target="_blank" rel="noopener">await</a> was introduced and this really has cleaned up and enhanced the readability of your code.</p> <p>Having the ability to use async/await is great, and is supported out of the box w/ Express.  But what do you do when you using a library which still wants to use promises or callbacks? The case in point for this article is <a href="https://aws.amazon.com/sdk-for-node-js/" target="_blank" rel="noopener">AWS Node SDK</a>.</p> <p>By default if you read through the AWS SDK documentation the examples lead you to believe that you need to use callbacks when implementing the SDK.  Well this can really lead to some nasty code in the world of Node/Express.  However, as of <a href="https://aws.amazon.com/blogs/developer/support-for-promises-in-the-sdk/" target="_blank" rel="noopener">v2.3.0</a> of the AWS SDK there is support for Promises.  This is much cleaner than using callbacks, but still poses a bit of an issue if you want to use async/await in your Express routes.</p> <p>However, with a bit of work you can get your promise based AWS calls to play nicely with your async/await based Express routes.  Lets take a look at how we can accomplish this.</p> <p>Before you get started I am going to make a few assumptions.</p> <ol> <li>You already have a Node/Express application setup</li> <li>You already have the AWS SDK for Node installed, if not read <a href="https://aws.amazon.com/sdk-for-node-js/" target="_blank" rel="noopener">here</a></li> </ol> <p>The first thing we are going to need to do is add reference to our AWS SDK and configure it to use promises.</p> <div class="code-snippet"> <pre class="code-content">const AWS = require('aws-sdk'); AWS.config.setPromisesDependency(null); </pre> </div> <p>After we have our SDK configured we can implement our route handler.  In my example here I am placing all the logic inside my handler.  In a real code base I would suggest better deconstruction of this code into smaller parts.</p> <div class="code-snippet"> <pre class="code-content">const express = require('express'); const router = express.Router(); const s3 = new AWS.S3(); router.get('/myRoute', async (req, res) =&gt; { const controller = new sitesController(); const params = req.params; const params = { Bucket: "bucket_name_here" }; let results = {}; var listPromise = s3.listObjects(params).promise(); listPromise.then((data) =&gt; { results = data; }); await Promise.all([listPromise]); res.json({data: results }) }) module.exports = router; </pre> </div> <p>Lets review the code above and call out a few important items.</p> <p>The first thing to notice is the addition of the <code>async</code> keyword in my route handler.  This is what allows us to use async/await in Node/Express.</p> <p>The next thing to look at is how I am calling the s3.listObjects.  Notice I am <strong>NOT </strong>providing a callback to the method, but instead I am chaining with .promise().  This is what instructs the SDK to use promises vs callbacks.  Once I have my callback I chain a &#8216;then&#8217; in order to handle my response.</p> <p>The last thing to pay attention to is the line with <code>await Promise.All([listPromise]);</code> This is the magic forces our route handler to not return prior to the resolution of all of our Promises.  Without this your call would exit prior to the listObjects call completing.</p> <p>Finally, we are simply returning our data from the listObjects call via <code>res.json</code> call.</p> <p>That&#8217;s it, pretty straight forward, once you learn that the AWS SDK supports something other than callbacks.</p> <p>Till next time,</p> Unable To Access Mysql With Root and No Password After New Install On Ubuntu https://blog.jasonmeridth.com/posts/unable-to-access-mysql-with-root-and-no-password-after-new-install-on-ubuntu/ Jason Meridth urn:uuid:f81a51eb-8405-7add-bddb-f805b183347e Wed, 31 Jan 2018 00:13:00 +0000 <p>This bit me in the rear end again today. Had to reinstall mysql-server-5.7 for other reasons.</p> <p>You just installed <code class="highlighter-rouge">mysql-server</code> locally for your development environment on a recent version of Ubuntu (I have 17.10 artful installed). You did it with a blank password for <code class="highlighter-rouge">root</code> user. You type <code class="highlighter-rouge">mysql -u root</code> and you see <code class="highlighter-rouge">Access denied for user 'root'@'localhost'</code>.</p> <p><img src="https://blog.jasonmeridth.com/images/wat.png" alt="wat" /></p> <p>Issue: Because you chose to not have a password for the <code class="highlighter-rouge">root</code> user, the <code class="highlighter-rouge">auth_plugin</code> for my MySQL defaulted to <code class="highlighter-rouge">auth_socket</code>. That means if you type <code class="highlighter-rouge">sudo mysql -u root</code> you will get in. If you don’t, then this is NOT the fix for you.</p> <p>Solution: Change the <code class="highlighter-rouge">auth_plugin</code> to <code class="highlighter-rouge">mysql_native_password</code> so that you can use the root user in the database.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo mysql -u root mysql&gt; USE mysql; mysql&gt; UPDATE user SET plugin='mysql_native_password' WHERE User='root'; mysql&gt; FLUSH PRIVILEGES; mysql&gt; exit; $ sudo systemctl restart mysql $ sudo systemctl status mysql </code></pre></div></div> <p><strong>NB</strong> ALWAYS set a password for mysql-server in staging/production.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/unable-to-access-mysql-with-root-and-no-password-after-new-install-on-ubuntu/">Unable To Access Mysql With Root and No Password After New Install On Ubuntu</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on January 30, 2018.</p> New Job https://blog.jasonmeridth.com/posts/new-job/ Jason Meridth urn:uuid:102e69a7-2b63-e750-2fa5-f46372d4d7c1 Mon, 08 Jan 2018 18:13:00 +0000 <p>Well, it is a new year and I’ve started a new job. I am now a Senior Software Engineer at <a href="https://truelinkfinancial.com">True Link Financial</a>.</p> <p><img src="https://blog.jasonmeridth.com/images/tllogo.png" alt="true link financial logo" /></p> <p>After interviewing with the co-founders Kai and Claire and their team, I knew I wanted to work here.</p> <p><strong>TL;DR</strong>: True Link: We give elderly and disable (really, anyone) back their financial freedom where they may not usually have it.</p> <p>Longer Version: Imagine you have an elderly family member who may start showing signs of dimensia. You can give them a True Link card and administer their card. You link it to their bank account or another source of funding and you can set limitations on when, where and how the card can be used. The family member feels freedom by not having to continually ask for money but is also protected by scammers and non-friendly people (yep, they exist).</p> <p>The customer service team, the marketing team, the product team, the engineering team and everyone else at True Link are amazing.</p> <p>For any nerd readers, the tech stack is currently Rails, React, AWS, Ansible. We’ll be introducing Docker and Kubernetes soon hopefully, but always ensuring the right tools for the right job.</p> <p>Looking forward to 2018.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/new-job/">New Job</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on January 08, 2018.</p> Docker Daemon Error When Running Docker Compose https://blog.jasonmeridth.com/posts/docker-daemon-error-when-running-docker-compose/ Jason Meridth urn:uuid:d21aa316-47d6-9a53-8329-179811ac6249 Tue, 02 Jan 2018 18:11:00 +0000 <p><img src="https://blog.jasonmeridth.com/images/why-docker-why-i-just-want-to-deploy.jpg" alt="docker why" /></p> <p>TL;DR Make sure you don’t have any old mounted volumes around if you see the error below.</p> <p>I just got the following error when trying to run <code class="highlighter-rouge">docker-compose up -d</code></p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running? If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable. </code></pre></div></div> <p>Please note I have a database using a mounted volume. The issue was that the previous mount was still present. Once I deleted that, <code class="highlighter-rouge">docker-compose up -d</code> would work jut fine.</p> <p>If anyone knows a more elegant way to handle this, I’m open to it.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/docker-daemon-error-when-running-docker-compose/">Docker Daemon Error When Running Docker Compose</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on January 02, 2018.</p> Lenovo Thinkpad - Swap ctrl and fn keys https://blog.jasonmeridth.com/posts/lenovo-thinkpad-swap-ctrl-and-fn-keys/ Jason Meridth urn:uuid:c989249f-12fc-b79e-147e-079eeb6c8de0 Wed, 22 Nov 2017 16:18:00 +0000 <p>I just got a new laptop. It is a Lenovo Thinkpad X1 Carbon 5th Gen.</p> <p>Ubuntu 17.10 16 GB RAM 1 TB SSD 64-bit i7 Pentium USB-C power</p> <p>I’m in love.</p> <p><img src="https://blog.jasonmeridth.com/images/lenovo-thinkpad-box.jpg" alt="lenovo thinkpad box" /></p> <p><img src="https://blog.jasonmeridth.com/images/lenovo-thinkpad.jpg" alt="lenovo thinkpad" /></p> <p><img src="https://blog.jasonmeridth.com/images/lenovo-thinkpad-fn-ctrl-keys.jpg" alt="lenovo thinkpad fn ctrl keys" /></p> <p>My only gripe is that the <code class="highlighter-rouge">Fn</code> key is on the far left bottom of the keyboard. I prefer that to be the <code class="highlighter-rouge">Ctrl</code> key due to copy/paste and other keyboard commands I use often. I also have very big hands and my pinky isn’t made to “find” the <code class="highlighter-rouge">Ctrl</code> key to the right of the <code class="highlighter-rouge">Fn</code> key.</p> <p>I currently use <code class="highlighter-rouge">Dconf</code> for mapping of keys in Ubuntu (currently using 17.10). I learned today that the <code class="highlighter-rouge">fn</code> key on keyboards is not managed by the operating system, which makes sense. Lenovo in all of its glorious-ness has a BIOS option to swap the <code class="highlighter-rouge">Fn</code> and <code class="highlighter-rouge">Ctrl</code> keys.</p> <p><img src="https://blog.jasonmeridth.com/images/lenovo-bios-fn-ctrl-swap.jpg" alt="lenovo bios fn ctrl swap" /></p> <p>Thank you Lenovo.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/lenovo-thinkpad-swap-ctrl-and-fn-keys/">Lenovo Thinkpad - Swap ctrl and fn keys</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on November 22, 2017.</p> Ubuntu - set caps lock to escape https://blog.jasonmeridth.com/posts/ubuntu-set-caps-lock-to-escape/ Jason Meridth urn:uuid:c4b489be-0841-6d35-7aa1-469f1440156b Wed, 22 Nov 2017 16:05:00 +0000 <p>I just got a new laptop and had to google again on how to set caps lock key to escape (I’m a Vim user).</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install dconf-tools dconf write /org/gnome/desktop/input-sources/xkb-options "['caps:escape']" </code></pre></div></div> <p>To know your options, use the following command:</p> <p><code class="highlighter-rouge">man xkeyboard-config</code></p> <p>(MAN pages are your friend; man is short for manual)</p> <p>You can also now use the <code class="highlighter-rouge">Dconf</code> GUI editor if you must (SHAME! ;) )</p> <p>Type <code class="highlighter-rouge">Dconf</code> in Unity or Gnome app opener and go to the following location:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>`org` →- `gnome` → `desktop` → `input-sources` → `xkb-options` </code></pre></div></div> <p>Add <code class="highlighter-rouge">['caps:escape']</code> to <code class="highlighter-rouge">Custom Value</code> textbox.</p> <p><img src="https://blog.jasonmeridth.com/images/dconf-caps-lock-to-escape.png" alt="dconf caps lock to escape" /></p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/ubuntu-set-caps-lock-to-escape/">Ubuntu - set caps lock to escape</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on November 22, 2017.</p> Cleanup Docker https://blog.jasonmeridth.com/posts/cleanup-docker/ Jason Meridth urn:uuid:77971466-e377-283d-29be-79ef44c5da32 Mon, 13 Nov 2017 03:15:00 +0000 <p>I keep having friends who have experienced the <code class="highlighter-rouge">no space left on device</code> when trying to build images.</p> <p>I have aliases for most of my container/image/volume cleanup:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lias dka='dkc;dki;dkv' alias dkc='docker ps -aq | xargs docker rm -f' alias dki='docker images -aq | xargs docker rmi -f' alias dkv='docker volume ls -qf dangling=true | xargs docker volume rm' </code></pre></div></div> <p>I use <code class="highlighter-rouge">dka</code> all the time.</p> <p>There is also the <code class="highlighter-rouge">docker system prune -a</code> command that works.</p> <p>I’ve also had to unmount my local aufs volume on my ubuntu laptop via:</p> <p><code class="highlighter-rouge">sudo umount -l /var/lib/docker/aufs &amp;&amp; sudo rm -rf /var/lib/docker/aufs</code></p> <p>and all things are cleaned up. Hope this helps someone else.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/cleanup-docker/">Cleanup Docker</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on November 12, 2017.</p> OnePlus 5 https://blog.jasonmeridth.com/posts/one-plus-5/ Jason Meridth urn:uuid:926e9e6f-286f-f817-45a2-0a249af8a8f9 Mon, 03 Jul 2017 17:02:00 +0000 <p>I’ve had a Nexus 6 for the last 2 years and was finally due for a phone upgrade. I went through a pretty good fiasco with Google store trying to purchase a Google Pixel XL earlier this year so I decided to wait for the One Plus 5 (release late June). I ordered it the first hour it was announced.</p> <p>Features that I’m loving:</p> <ul> <li>8 GB RAM</li> <li>128 GB hard drive</li> <li>dash charge</li> <li>USB C</li> <li>OxygenOS (Android fork)</li> <li>headphone jack</li> <li>fingerprint authentication</li> <li>dual camera (allows for portrait mode) 16 MP</li> <li>front camera 16 MP</li> <li>Dual SIM support</li> </ul> <p>More detailed specs can be found <a href="https://oneplus.net/5/specs">here</a></p> <p>Features I’m adjusting to:</p> <ul> <li>no Google Phone app install allowed</li> <li>no Google Contacts app install allowed</li> </ul> <p>I’ll adjust to those over time.</p> <p>I did entertain the iPhone 7 for a bit also but am not a fan of iTunes. The iPhone integration with Google apps has gotten much better since the last time I looked though.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/one-plus-5/">OnePlus 5</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on July 03, 2017.</p> Hello, React! – A Beginner’s Setup Tutorial https://lostechies.com/derekgreer/2017/05/25/hello-react-a-beginners-setup-tutorial/ Los Techies urn:uuid:896513a4-c41d-c8ea-820b-fbc3e2b5a442 Thu, 25 May 2017 08:00:32 +0000 React has been around for a few years now and there are quite a few tutorials available. Unfortunately, many are outdated, overly complex, or gloss over configuration for getting started. Tutorials which side-step configuration by using jsfiddle or code generator options are great when you’re wanting to just focus on the framework features itself, but many tutorials leave beginners struggling to piece things together when you’re ready to create a simple react application from scratch. This tutorial is intended to help beginners get up and going with React by manually walking through a minimal setup process. <p>React has been around for a few years now and there are quite a few tutorials available. Unfortunately, many are outdated, overly complex, or gloss over configuration for getting started. Tutorials which side-step configuration by using jsfiddle or code generator options are great when you’re wanting to just focus on the framework features itself, but many tutorials leave beginners struggling to piece things together when you’re ready to create a simple react application from scratch. This tutorial is intended to help beginners get up and going with React by manually walking through a minimal setup process.</p> <h2 id="a-simple-tutorial">A Simple Tutorial</h2> <p>This tutorial is merely intended to help walk you through the steps to getting a simple React example up and running. When you’re ready to dive into actually learning the React framework, a great list of tutorials can be found <a href="http://andrewhfarmer.com/getting-started-tutorials/">here.</a></p> <p>There are a several build, transpiler, or bundling tools from which to select when working with React. For this tutorial, we’ll be using be using Node, NPM, Webpack, and Babel.</p> <h2 id="step-1-install-node">Step 1: Install Node</h2> <p>Download and install Node for your target platform. Node distributions can be obtained <a href="https://nodejs.org/en/">here</a>.</p> <h2 id="step-2-create-a-project-folder">Step 2: Create a Project Folder</h2> <p>From a command line prompt, create a folder where you plan to develop your example.</p> <pre>$&gt; mkdir hello-react </pre> <h2 id="step-3-initialize-project">Step 3: Initialize Project</h2> <p>Change directory into the example folder and use the Node Package Manager (npm) to initialize the project:</p> <pre>$&gt; cd hello-react $&gt; npm init --yes </pre> <p>This results in the creation of a package.json file. While not technically necessary for this example, creating this file will allow us to persist our packaging and runtime dependencies.</p> <h2 id="step-4-install-react">Step 4: Install React</h2> <p>React is broken up into a core framework package and a package related to rendering to the Document Object Model (DOM).</p> <p>From the hello-react folder, run the following command to install these packages and add them to your package.json file:</p> <pre>$&gt; npm install --save-dev react react-dom </pre> <h2 id="step-5-install-babel">Step 5: Install Babel</h2> <p>Babel is a transpiler, which is to say it’s a tool from converting one language or language version to another. In our case, we’ll be converting EcmaScript 2015 to EcmaScript 5.</p> <p>From the hello-react folder, run the following command to install babel:</p> <pre>$&gt; npm install --save-dev babel-core </pre> <h2 id="step-6-install-webpack">Step 6: Install Webpack</h2> <p>Webpack is a module bundler. We’ll be using it to package all of our scripts into a single script we’ll include in our example Web page.</p> <p>From the hello-react folder, run the following command to install webpack globally:</p> <pre>$&gt; npm install webpack --global </pre> <h2 id="step-7-install-babel-loader">Step 7: Install Babel Loader</h2> <p>Babel loader is a Webpack plugin for using Babel to transpile scripts during the bundling process.</p> <p>From the hello-react folder, run the following command to install babel loader:</p> <pre>$&gt; npm install --save-dev babel-loader </pre> <h2 id="step-8-install-babel-presets">Step 8: Install Babel Presets</h2> <p>Babel presets are collections of plugins needed to support a given feature. For example, the latest version of babel-preset-es2015 at the time this writing will install 24 plugins which enables Babel to transpile ECMAScript 2015 to ECMAScript 5. We’ll be using presets for ES2015 as well as presets for React. The React presets are primarily needed for processing of <a href="https://facebook.github.io/react/docs/introducing-jsx.html">JSX</a>.</p> <p>From the hello-react folder, run the following command to install the babel presets for both ES2015 and React:</p> <pre>$&gt; npm install --save-dev babel-preset-es2015 babel-preset-react </pre> <h2 id="step-9-configure-babel">Step 9: Configure Babel</h2> <p>In order to tell Babel which presets we want to use when transpiling our scripts, we need to provide a babel config file.</p> <p>Within the hello-react folder, create a file named .babelrc with the following contents:</p> <pre>{ "presets" : ["es2015", "react"] } </pre> <h2 id="step-10-configure-webpack">Step 10: Configure Webpack</h2> <p>In order to tell Webpack we want to use Babel, where our entry point module is, and where we want the output bundle to be created, we need to create a Webpack config file.</p> <p>Within the hello-react folder, create a file named webpack.config.js with the following contents:</p> <pre>const path = require('path'); module.exports = { entry: './app/index.js', output: { path: path.resolve('dist'), filename: 'index_bundle.js' }, module: { rules: [ { test: /\.js$/, loader: 'babel-loader', exclude: /node_modules/ } ] } } </pre> <h2 id="step-11-create-a-react-component">Step 11: Create a React Component</h2> <p>For our example, we’ll just be creating a simple component which renders the text “Hello, React!”.</p> <p>First, create an app sub-folder:</p> <pre>$&gt; mkdir app </pre> <p>Next, create a file named app/index.js with the following content:</p> <pre>import React from 'react'; import ReactDOM from 'react-dom'; class HelloWorld extends React.Component { render() { return ( &lt;div&gt; Hello, React! &lt;/div&gt; ) } }; ReactDOM.render(&lt;HelloWorld /&gt;, document.getElementById('root')); </pre> <p>Briefly, this code includes the react and react-dom modules, defines a HelloWorld class which returns an element containing the text “Hello, React!” expressed using <a href="https://facebook.github.io/react/docs/introducing-jsx.html">JSX syntax</a>, and finally renders an instance of the HelloWorld element (also using JSX syntax) to the DOM.</p> <p>If you’re completely new to React, don’t worry too much about trying to fully understand the code. Once you’ve completed this tutorial and have an example up and running, you can move on to one of the aforementioned tutorials, or work through <a href="https://facebook.github.io/react/docs/hello-world.html">React’s Hello World example</a> to learn more about the syntax used in this example.</p> <div class="note"> <p> Note: In many examples, you will see the following syntax: </p> <pre> var HelloWorld = React.createClass( { render() { return ( &lt;div&gt; Hello, React! &lt;/div&gt; ) } }); </pre> <p> This syntax is how classes were defined in older versions of React and will therefore be what you see in older tutorials. As of React version 15.5.0 use of this syntax will produce the following warning: </p> <p style="color: red"> Warning: HelloWorld: React.createClass is deprecated and will be removed in version 16. Use plain JavaScript classes instead. If you&#8217;re not yet ready to migrate, create-react-class is available on npm as a drop-in replacement. </p> </div> <h2 id="step-12-create-a-webpage">Step 12: Create a Webpage</h2> <p>Next, we’ll create a simple html file which includes the bundled output defined in step 10 and declare a &lt;div&gt; element with the id “root” which is used by our react source in step 11 to render our HelloWorld component.</p> <p>Within the hello-react folder, create a file named index.html with the following contents:</p> <pre>&lt;html&gt; &lt;div id="root"&gt;&lt;/div&gt; &lt;script src="./dist/index_bundle.js"&gt;&lt;/script&gt; &lt;/html&gt; </pre> <h2 id="step-13-bundle-the-application">Step 13: Bundle the Application</h2> <p>To convert our app/index.js source to ECMAScript 5 and bundle it with the react and react-dom modules we’ve included, we simply need to execute webpack.</p> <p>Within the hello-react folder, run the following command to create the dist/index_bundle.js file reference by our index.html file:</p> <pre>$&gt; webpack </pre> <h2 id="step-14-run-the-example">Step 14: Run the Example</h2> <p>Using a browser, open up the index.html file. If you’ve followed all the steps correctly, you should see the following text displayed:</p> <pre>Hello, React! </pre> <h2 id="conclusion">Conclusion</h2> <p>Congratulations! After completing this tutorial, you should have a pretty good idea about the steps involved in getting a basic React app up and going. Hopefully this will save some absolute beginners from spending too much time trying to piece these steps together.</p> Up into the Swarm https://lostechies.com/gabrielschenker/2017/04/08/up-into-the-swarm/ Los Techies urn:uuid:844f7b20-25e5-e658-64f4-e4d5f0adf614 Sat, 08 Apr 2017 20:59:26 +0000 Last Thursday evening I had the opportunity to give a presentation at the Docker Meetup in Austin TX about how to containerize a Node JS application and deploy it into a Docker Swarm. I also demonstrated techniques that can be used to reduce friction in the development process when using containers. <p>Last Thursday evening I had the opportunity to give a presentation at the Docker Meetup in Austin TX about how to containerize a Node JS application and deploy it into a Docker Swarm. I also demonstrated techniques that can be used to reduce friction in the development process when using containers.</p> <p>The meeting was recorded but unfortunately sound only is available after approximately 16 minutes. You might want to just scroll forward to this point.</p> <p>Video: <a href="https://youtu.be/g786WiS5O8A">https://youtu.be/g786WiS5O8A</a></p> <p>Slides and code: <a href="https://github.com/gnschenker/pets-node">https://github.com/gnschenker/pets-node</a></p> New Year, New Blog https://lostechies.com/jimmybogard/2017/01/26/new-year-new-blog/ Los Techies urn:uuid:447d30cd-e297-a888-7ccc-08c46f5a1688 Thu, 26 Jan 2017 03:39:05 +0000 One of my resolutions this year was to take ownership of my digital content, and as such, I’ve launched a new blog at jimmybogard.com. I’m keeping all my existing content on Los Techies, where I’ve been humbled to be a part of for the past almost 10 years. Hundreds of posts, thousands of comments, and innumerable wrong opinions on software and systems, it’s been a great ride. <p>One of my resolutions this year was to take ownership of my digital content, and as such, I’ve launched a new blog at <a href="https://jimmybogard.com/">jimmybogard.com</a>. I’m keeping all my existing content on <a href="https://jimmybogard.lostechies.com/">Los Techies</a>, where I’ve been humbled to be a part of for the past <a href="http://grabbagoft.blogspot.com/2007/11/joining-los-techies.html">almost 10 years</a>. Hundreds of posts, thousands of comments, and innumerable wrong opinions on software and systems, it’s been a great ride.</p> <p>If you’re still subscribed to my FeedBurner feed – nothing to change, you’ll get everything as it should. If you’re only subscribed to the Los Techies feed…well you’ll need to <a href="http://feeds.feedburner.com/GrabBagOfT">subscribe to my feed</a> now.</p> <p>Big thanks to everyone at Los Techies that’s put up with me over the years, especially our site admin <a href="https://jasonmeridth.com/">Jason</a>, who has become far more knowledgable about WordPress than he ever probably wanted.</p> Containers – Cleanup your house revisited https://lostechies.com/gabrielschenker/2016/12/12/containers-cleanup-your-house-revisited/ Los Techies urn:uuid:696d80e9-3827-df0d-6fdf-ab1f51274e7b Mon, 12 Dec 2016 21:10:02 +0000 In version 1.13 Docker has added some useful commands to the CLI that make it easier to keep our environment clean. As you might have experienced yourself over time our development environment gets really cluttered with unused containers, dangling Docker images, abandoned volumes and forgotten networks. All these obsolete items take aways precious resources and ultimately lead to an unusable environment. In a previous post I have shown how we can keep our house clean by using various commands like <p>In version 1.13 Docker has added some useful commands to the CLI that make it easier to keep our environment clean. As you might have experienced yourself over time our development environment gets really cluttered with unused containers, dangling Docker images, abandoned volumes and forgotten networks. All these obsolete items take aways precious resources and ultimately lead to an unusable environment. In a <a href="https://lostechies.com/gabrielschenker/2016/08/14/containers-clean-up-your-house/">previous post</a> I have shown how we can keep our house clean by using various commands like</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker rm -f $(docker ps -aq) </code></pre></div></div> <p>to forcibly remove all running, stopped and terminated containers. Similarly we learned commands that allowed us to remove dangling images, networks and volumes.</p> <p>Although the commands I described solved the problem they were proprietary, verbose or difficult to remember. The new commands introduced are straight forward and easy to use. Let’s try them out.</p> <blockquote> <p>If you like this article then you can find more posts about containers in general and Docker in specific in <a href="https://lostechies.com/gabrielschenker/2016/08/26/containers-an-index/">this</a> table of content.</p> </blockquote> <h1 id="management-commands">Management Commands</h1> <p>To un-clutter the CLI a bit Docker 1.13 introduces new management commands. The list of those are</p> <ul> <li>system</li> <li>container</li> <li>image</li> <li>plugin</li> <li>secret</li> </ul> <p>Older versions of Docker already had <code class="highlighter-rouge">network, node, service, swarm</code> and <code class="highlighter-rouge">volume</code>.</p> <p>These new commands group subcommands that were previously directly implemented as root commands. Let me give an example</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker exec -it [container-name] [some-command] </code></pre></div></div> <p>The <code class="highlighter-rouge">exec</code> command is now a subcommand under <code class="highlighter-rouge">container</code>. Thus the equivalent of the above command is</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker container exec -it [container-name] [some-command] </code></pre></div></div> <p>I would assume that for reasons of backwards compatibility the old syntax will stick around with us for the time being.</p> <h1 id="docker-system">Docker System</h1> <p>There is a new management command <code class="highlighter-rouge">system</code>. It has 4 possible subcommands <code class="highlighter-rouge">df, events, info</code> and <code class="highlighter-rouge">prune</code>. The command</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker system df </code></pre></div></div> <p>gives us an overview of the overall disk usage of Docker. This include images, containers and (local) volumes. So we can now at any time stay informed about how much resources Docker consumes.</p> <p>If the previous command shows us that we’re using too much space we might as well start to cleanup. Our next command does exactly that. It is a <em>do-it-all</em> type of command</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker system prune </code></pre></div></div> <p>This command removes everything that is currently not used, and it does it in the correct sequence so that a maximum outcome is achieved. First unused containers are removed, then volumes and networks and finally dangling images. We have to confirm the operation though by answering with <code class="highlighter-rouge">y</code>. If we want to use this command in a script we can use the parameter <code class="highlighter-rouge">--force</code> or <code class="highlighter-rouge">-f</code> to instruct Docker not to ask for confirmation.</p> <h1 id="docker-container">Docker Container</h1> <p>We already know many of the subcommands of <code class="highlighter-rouge">docker container</code>. They were previously (and still are) direct subcommands of <code class="highlighter-rouge">docker</code>. We can get the full list of subcommands like this</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker container --help </code></pre></div></div> <p>In the list we find again a <code class="highlighter-rouge">prune</code> command. If we use it we only remove unused containers. Thus the command is much more limited than the <code class="highlighter-rouge">docker system prune</code> command that we introduced in the previous section. Using the <code class="highlighter-rouge">--force</code> or <code class="highlighter-rouge">-f</code> flag we can again instruct the CLI not to ask us for confirmation</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker container prune --force </code></pre></div></div> <h1 id="docker-network">Docker Network</h1> <p>As you might expect, we now also have a <code class="highlighter-rouge">prune</code> command here.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker network prune </code></pre></div></div> <p>removes all orphaned networks</p> <h1 id="docker-volume">Docker Volume</h1> <p>And once again we find a new <code class="highlighter-rouge">prune</code> command for volumes too.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker volume prune </code></pre></div></div> <p>removes all (local) volumes that are not used by at least one container.</p> <h1 id="docker-image">Docker Image</h1> <p>Finally we have the new image command which of course gets a <code class="highlighter-rouge">prune</code> subcommand too. We have the flag <code class="highlighter-rouge">--force</code> that does the same job as in the other samples and we have a flag <code class="highlighter-rouge">--all</code> that not just removes dangling images but all unused ones. Thus</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker image prune --force --all </code></pre></div></div> <p>removes all images that are unused and does not ask us for confirmation.</p> <h1 id="summary">Summary</h1> <p>Not only has Docker v 1.13 brought some needed order into the zoo of Docker commands by introducing so called admin commands but also we find some very helpful commands to clean up our environment from orphaned items. My favorite command will most probably be the <code class="highlighter-rouge">docker system prune</code> as I always like an uncluttered environment.</p> Mapping Caps Lock to Esc Is Native to OSX Now https://blog.jasonmeridth.com/posts/mapping-caps-lock-to-esc-on-osx-is-native-now/ Jason Meridth urn:uuid:bc5e8b8b-0d6e-ba6a-b9db-e36d6a6e3259 Mon, 31 Oct 2016 20:12:00 +0000 <p>I have been using <a href="https://pqrs.org/osx/karabiner/seil.html.en">Seil</a> for a few years now on OSX to map Caps Lock to Esc. I use Vim for my development and letting my left pinky tap the Caps Lock key instead of Esc allows me to keep my hands on the home row and move much quicker. I also can’t remember the last time I actually needed the Caps Lock key. Well as of 10.12.1 (macOS Sierra Update) you can do this mapping in System Preferences.</p> <p>Thank you to my co-worker <a href="https://twitter.com/kweerious">Dedi</a> for letting me know about this.</p> <p>Go to System Preferences from the Apple menu:</p> <p><img src="https://blog.jasonmeridth.com/images/system_preferences.png" alt="system preferences" /></p> <p>Go to keyboard: <img src="https://blog.jasonmeridth.com/images/keyboard.png" alt="Keyboard" /></p> <p>Go to “Modifier Keys” button on bottom right: <img src="https://blog.jasonmeridth.com/images/modifier_keys.png" alt="Modifier Keys Button" /></p> <p>Change Caps Lock Key to Escape: <img src="https://blog.jasonmeridth.com/images/caps_lock_to_esc.png" alt="Change Caps Lock Key to Esc" /></p> <p><a href="https://blog.jasonmeridth.com/posts/mapping-caps-lock-to-esc-on-osx-is-native-now/">Mapping Caps Lock to Esc Is Native to OSX Now</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on October 31, 2016.</p> Details HTML Section In Github Issues and Gists https://blog.jasonmeridth.com/posts/details-html-section-in-github-issues-and-gists/ Jason Meridth urn:uuid:824c5783-0603-a791-ceb8-6c3fb80638dd Thu, 28 Jul 2016 16:01:00 +0000 <p>I recently became aware of using the <code class="highlighter-rouge">&lt;details&gt;&lt;/details&gt;</code> and <code class="highlighter-rouge">&lt;summary&gt;...&lt;/summary&gt;</code> tags in Github issues and Gists.</p> <p><a href="https://github.com/jmeridth/jmeridth.github.io/issues/3">Here</a> is an example.</p> <p>I will definitely be using this more when posting big logs or stack traces.</p> <p>Kudos:</p> <ul> <li><a href="https://twitter.com/rhein_wein">Laura Frank</a> for showing me this feature</li> <li><a href="https://twitter.com/dahlbyk">Keith Dahlby</a> for letting me know it doesn’t currently work in Firefox</li> <li><a href="https://twitter.com/mhinze">Matt Hinze</a> for showing me <a href="https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d">this</a> gist where I learned of <code class="highlighter-rouge">&lt;summary&gt;...&lt;/summary&gt;</code></li> <li><a href="https://twitter.com/ericclemmons">Eric Clemmons</a> for the awesome <a href="https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d">gist</a> that does a great job explaining the feature</li> </ul> <p><a href="https://blog.jasonmeridth.com/posts/details-html-section-in-github-issues-and-gists/">Details HTML Section In Github Issues and Gists</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on July 28, 2016.</p> Stop and Remove All Docker Containers https://blog.jasonmeridth.com/posts/stop-remove-all-docker-containers/ Jason Meridth urn:uuid:9104fd13-29be-163e-6a64-15155d737189 Wed, 06 Jul 2016 21:44:00 +0000 <p>Command remove all docker containers:</p> <p><code class="highlighter-rouge">docker stop '$(docker ps -a -q)' &amp;&amp; docker rm '$(docker ps -a -q)'</code></p> <p><code class="highlighter-rouge">docker ps -a -q</code> lists all container IDs</p> <p><a href="https://blog.jasonmeridth.com/posts/stop-remove-all-docker-containers/">Stop and Remove All Docker Containers</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on July 06, 2016.</p>