Los Techies http://feed.informer.com/digests/ZWDBOR7GBI/feeder Los Techies Respective post owners and feed distributors Thu, 08 Feb 2018 14:40:57 +0000 Feed Informer http://feed.informer.com/ Constrained Open Generics Support Merged in .NET Core DI Container https://jimmybogard.com/constrained-open-generics-support-merged-in-net-core-di-container/ Jimmy Bogard urn:uuid:10eb6071-54da-8a50-1269-fe4741874a83 Tue, 25 Aug 2020 13:51:00 +0000 <p>Continuing the trend of extremely long blog titles...years ago I <a href="https://github.com/aspnet/DependencyInjection/issues/471">opened a pull request for supporting constrained open generics</a> in the built-in Microsoft.Extensions.DependencyInjection container. And another. And another....and another. But I did not give up hope - after porting through a couple repo moves and consolidations,</p> <p>Continuing the trend of extremely long blog titles...years ago I <a href="https://github.com/aspnet/DependencyInjection/issues/471">opened a pull request for supporting constrained open generics</a> in the built-in Microsoft.Extensions.DependencyInjection container. And another. And another....and another. But I did not give up hope - after porting through a couple repo moves and consolidations, this <a href="https://github.com/dotnet/runtime/pull/39540">feature is now merged</a> and will get released with .NET 5 (huzzah).</p><p>So what is this mouthful going to get you? Well, for users of <a href="https://github.com/jbogard/MediatR">MediatR</a>, quite a lot. Something I see quite a lot when building generic-oriented code are declarations of <code>IFoo&lt;T&gt;</code> where you have a method accepting that <code>T</code>. For example, <a href="https://fluentvalidation.net/">Fluent Validation</a> defines an <code>IValidator&lt;T&gt;</code> interface that is (roughly): </p><pre><code class="language-csharp">public interface IValidator&lt;in T&gt; { ValidationResult Validate(T instance); }</code></pre><p>Notice the contravariance here - we can use more derived types in the T parameter. I often do this in validation when I have many different screens that share some common trait, or validation:</p><pre><code class="language-csharp">public class UserInfoValidator : IValidator&lt;IContainUserInfo&gt; { public ValidationResult Validate(IContainUserInfo instance) { } }</code></pre><p>All DTOs that then want this common validation only need to implement the <code>IContainUserInfo</code> interface.</p><p>This is all well and good, but there is a slight problem here. My type parameter has lost its original type information, because I've closed the <code>IValidator</code> interface, so if I want to plug in additional dependencies based on that type, it's gone.</p><p>Constraining instead of closing the generic type preserves the generic parameter:</p><pre><code class="language-csharp">public class UserInfoValidator&lt;T&gt; : IValidator&lt;T&gt; where T : IContainUserInfo { public ValidationResult Validate(T instance) { // I can still use T's members of IContainUserInfo } }</code></pre><p>What's important here is that because I still have the type parameter, I can now include dependencies here that still can be based on the type T. For example, I could do something like:</p><pre><code class="language-csharp">public class UserInfoValidator&lt;T&gt; : IValidator&lt;T&gt; where T : IContainUserInfo { public UserInfoValidator(IEnumerable&lt;IPermission&lt;T&gt;&gt; permissions) =&gt; _permissions = permissions;</code></pre><p>In this example, I depend on another collection of permission dependencies to validate my user information. However, because my generic type is still open, at runtime, only the <code>IPermission&lt;T&gt;</code> instances that satisfy <code>T</code> will be satisfied and supplied.</p><p>There's a gap in .NET to make this sort of check much easier, and I've <a href="https://github.com/dotnet/runtime/issues/28033">opened an API proposal to help</a>. Please vote if this proposal interests you!</p><p>Constrained generics are a powerful tool to apply cross-cutting concerns to generics in dependency injection, giving us a "DI-enabled" sort of pattern matching. The pull request that merged provides a simple fix (don't blow up).</p><h3 id="what-took-so-long">What Took So Long?</h3><p>Great question, especially since the <a href="https://github.com/dotnet/runtime/pull/39540">change is so small</a> (adding a try-catch), but it exposed a challenge that the MS team has to worry about. The built-in container is a <a href="https://blog.ploeh.dk/2014/05/19/conforming-container/">conforming container</a> - that is, it provides some default behaviors that ASP.NET Core requires, and if you want to replace that container, you need to conform to those behaviors.</p><p>This is a challenge for the DI container library authors, who have built their own libraries over the years, independently, and now need to conform to the needs and design of the MS.Ext.DI container. Conversely, external library authors (such as myself) want to build on top of the default DI container without needing to provide a bunch of library-specific extensions (MediatR.Autofac, MediatR.Lamar, MediatR.TinyIoc etc).</p><p>But what happens if you want to depend on a feature that:</p><ul><li>Exists in most or all popular containers</li><li>Does not exist in the conforming container (MS.Ext.DI)</li><li>Is not needed by ASP.NET Core (the primary user)</li></ul><p>That's where constrained open generics sat. It's not needed by the primary users (various .NET hosts) but is needed, or wanted, by a large number of people that <em>use</em> these .NET hosts built on top of the conforming container.</p><p>So, it's a risk - so what I wound up needing to do was:</p><ul><li>Show that the majority (or all) of the containers support this behavior</li><li>Document this behavior explicitly with a set of <a href="https://github.com/dotnet/runtime/pull/39540/files#diff-aba9089ea38c7a8b973e5c0a97ca114dR594">specification tests</a></li></ul><p>These specification tests run against the built-in container plus 8 popular ones (Autofac, Unity, Lamar etc.)</p><p>With this in place, I could show the expected behavior of this feature in DI containers, and prove that yes indeed, all the listed containers do support this feature.</p><p>And with that, and a little <code><a href="https://github.com/dotnet/runtime/pull/39540/files#diff-f70d1705c4fa0c61a95683531fb4245bR242">try..catch</a></code>, this feature will drop in .NET 5.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Ti4bHePbbvI:UdQlwDUSKjc:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Ti4bHePbbvI:UdQlwDUSKjc:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Ti4bHePbbvI:UdQlwDUSKjc:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Ti4bHePbbvI:UdQlwDUSKjc:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Ti4bHePbbvI:UdQlwDUSKjc:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/Ti4bHePbbvI" height="1" width="1" alt=""/> Diagnostics and Instrumentation Packages Targeting Open Telemetry Beta for MongoDB and NServiceBus Published https://jimmybogard.com/diagnostics-and-instrumentation-packages-targeting-open-telemetry-beta-for-mongodb-and-nservicebus-published/ Jimmy Bogard urn:uuid:6fa46905-1a3e-5d4b-b4d5-3b2973bf58e9 Thu, 06 Aug 2020 19:47:16 +0000 <p>That's...a long title. I've published release packages for System.Diagnostics support for NServiceBus and MongoDB:</p><ul><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics/">NServiceBus.Extensions.Diagnostics</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.DiagnosticSources/">MongoDB.Driver.Core.Extensions.DiagnosticSources</a></li></ul><p>And published new beta packages targeting the <a href="https://medium.com/opentelemetry/opentelemetry-net-beta-released-e1b070f0a5bc">beta release of OpenTelemetry</a>:</p><ul><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics.OpenTelemetry/">NServiceBus.Extensions.Diagnostics.OpenTelemetry</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.OpenTelemetry/">MongoDB.Driver.Core.Extensions.OpenTelemetry</a></li></ul><p>The Diagnostics packages are separate from</p> <p>That's...a long title. I've published release packages for System.Diagnostics support for NServiceBus and MongoDB:</p><ul><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics/">NServiceBus.Extensions.Diagnostics</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.DiagnosticSources/">MongoDB.Driver.Core.Extensions.DiagnosticSources</a></li></ul><p>And published new beta packages targeting the <a href="https://medium.com/opentelemetry/opentelemetry-net-beta-released-e1b070f0a5bc">beta release of OpenTelemetry</a>:</p><ul><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics.OpenTelemetry/">NServiceBus.Extensions.Diagnostics.OpenTelemetry</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.OpenTelemetry/">MongoDB.Driver.Core.Extensions.OpenTelemetry</a></li></ul><p>The Diagnostics packages are separate from the OpenTelemetry ones and only reference System.Diagnostics package. The OpenTelemetry ones now use OpenTelemetry beta, which switches to solely using Activity for recording OpenTelemetry spans, instead of mapping to a Span.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=OcRbBK7alDA:FUvbVoP1VBQ:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=OcRbBK7alDA:FUvbVoP1VBQ:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=OcRbBK7alDA:FUvbVoP1VBQ:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=OcRbBK7alDA:FUvbVoP1VBQ:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=OcRbBK7alDA:FUvbVoP1VBQ:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/OcRbBK7alDA" height="1" width="1" alt=""/> End-to-End Integration Testing with NServiceBus: How It Works https://jimmybogard.com/end-to-end-integration-testing-with-nservicebus-how-it-works/ Jimmy Bogard urn:uuid:ec9b8e3c-6ef7-4676-6dd6-f3b9f7d3959f Tue, 14 Jul 2020 20:16:56 +0000 <p>In my last post, I walked through <a href="https://jimmybogard.com/end-to-end-integration-testing-with-nservicebus/">setting up end-to-end integration testing with NServiceBus</a>, and how we can use it to black box test <a href="https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageEndpoint.html">message endpoints</a> similar to how the <a href="https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1">ASP.NET Core integration testing works</a>. In this post, I want to walk through how it all works underneath the</p> <p>In my last post, I walked through <a href="https://jimmybogard.com/end-to-end-integration-testing-with-nservicebus/">setting up end-to-end integration testing with NServiceBus</a>, and how we can use it to black box test <a href="https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageEndpoint.html">message endpoints</a> similar to how the <a href="https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1">ASP.NET Core integration testing works</a>. In this post, I want to walk through how it all works underneath the covers.</p><p>The major challenge in testing an async messaging flow is observability - how do we know when the work is started or completed? Our tests are written in an imperative, top-down, "Arrange-Act-Assert" style but the system under test (SUT) is intentionally asynchronous.</p><p>In my <a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">end-to-end diagnostics series</a>, I walked through adding observability to our message endpoints, and it turns out, we can leverage this same mechanism to observe our SUT in our integration tests.</p><h3 id="what-is-done">What is done?</h3><p>In order to know what we need to test, we need to understand what it means for the "Act" step in a test to complete successfully. In a normal synchronous interaction, the "Act" is guaranteed to complete when the method returns - whether this call is "awaited" or not, it's still blocking until the response is returned.</p><p>With async messaging, our calling code only blocks while until the message is "ack"d:</p><pre><code class="language-csharp">var firstMessage = new FirstMessage {Message = "Hello World"}; var session = _factory.Services.GetService&lt;IMessageSession&gt;(); await _session.SendLocal(firstMessage); // OK now what?</code></pre><p>We could do something silly like an <code>await Task.Delay(something??)</code> but that's not terribly efficient. Those task delays easily add up, and we could have finished far earlier but just not known it.</p><p>In an end-to-end message endpoint scenario, there are some top-level indications my work is complete:</p><ul><li>When a specific message is processed</li><li>When a specific message is sent</li></ul><p>Depending on my scenario, when one of these conditions is met, it's now safe to make my assertions of the messages or the handler's expected side effects.</p><p>When we have an external source of signals we're waiting to receive, we don't want to wait forever. We'll still want a timeout - a maximum amount of time we wait for those observable conditions to be met, otherwise we'll just timeout.</p><p>For those that have used various BDD or QA testing tools, this is pretty common for UI tests. We wait for the browser to make its calls and update things, and wait for some expected UI state to move on, but not wait forever. It's not much different, except instead of waiting for some DOM element to show up, we're waiting for messages.</p><h3 id="observing-messages">Observing Messages</h3><p>To be notified for messages sent or processed, we can first look at our diagnostic events we created in the end-to-end tracing posts. Our "Act" code needs to look for specific messages sent/received. Unfortunately, the same diagnostic event we used for telemetry is a bit lower level than we want - for example, the messages are transport-oriented, with <code>byte[]</code> messages bodies and the like.</p><p>Luckily, NServiceBus has an <a href="https://docs.particular.net/nservicebus/pipeline/steps-stages-connectors">additional pipeline hook for logical messages sent/received</a>, and this context object has more relevant information we'd want in our tests. We can then expose more diagnostic events at the logical message level, instead of physical. First, a behavior for processing messages:</p><pre><code class="language-csharp">public class IncomingLogicalMessageDiagnostics : Behavior&lt;IIncomingLogicalMessageContext&gt; { private readonly DiagnosticListener _diagnosticListener; private const string EventName = ActivityNames.IncomingLogicalMessage + ".Processed"; public IncomingLogicalMessageDiagnostics(DiagnosticListener diagnosticListener) =&gt; _diagnosticListener = diagnosticListener; public IncomingLogicalMessageDiagnostics() : this(new DiagnosticListener(ActivityNames.IncomingLogicalMessage)) { } public override async Task Invoke(IIncomingLogicalMessageContext context, Func&lt;Task&gt; next) { await next().ConfigureAwait(false); if (_diagnosticListener.IsEnabled(EventName)) { _diagnosticListener.Write(EventName, context); } } } </code></pre><p>In this example, I'm not using Activity information, I'm simply raising a single diagnostic event since the physical message behavior already does all of the telemetry work. We await the next step in the pipeline, and raise the event. This ensures our diagnostic event only raises when the message is successfully handled.</p><p>The parameter we include in the diagnostic event, the <code>IIncomingLogicalMessageContext</code>, includes the deserialized message and its type information. That way if we want to observe a specific message type with specific data, we can.</p><p>Next, we create a similar pipeline behavior for outgoing logical messages:</p><pre><code class="language-csharp">public class OutgoingLogicalMessageDiagnostics : Behavior&lt;IOutgoingLogicalMessageContext&gt; { private readonly DiagnosticListener _diagnosticListener; private const string EventName = ActivityNames.OutgoingLogicalMessage + ".Sent"; public OutgoingLogicalMessageDiagnostics(DiagnosticListener diagnosticListener) =&gt; _diagnosticListener = diagnosticListener; public OutgoingLogicalMessageDiagnostics() : this(new DiagnosticListener(ActivityNames.OutgoingLogicalMessage)) { } public override async Task Invoke(IOutgoingLogicalMessageContext context, Func&lt;Task&gt; next) { await next().ConfigureAwait(false); if (_diagnosticListener.IsEnabled(EventName)) { _diagnosticListener.Write(EventName, context); } } } </code></pre><p>Almost exactly the same! With our diagnostic events defined and set up in behaviors (this is in the <a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics/">NServiceBus.Extensions.Diagnostics</a> package), we can listen in on these events in our integration test.</p><h3 id="observing-diagnostics-events-in-a-test">Observing Diagnostics Events in a Test</h3><p>I boiled the "wait for an event" method to a single method in a fixture:</p><pre><code class="language-csharp">private static async Task&lt;ObservedMessageContexts&gt; ExecuteAndWait&lt;TMessageContext&gt;( Func&lt;Task&gt; testAction, Func&lt;TMessageContext, bool&gt; predicate, TimeSpan? timeout = null) where TMessageContext : IPipelineContext {</code></pre><p>This method takes a generic parameter - an <code>IPipelineContext</code>, which is either the incoming or outgoing logical message contexts. I need the test action to perform as a <code>Func&lt;Task&gt;</code> as I will need to call this after setting up the event observation. Finally, I need to know when to "stop" observing, with a predicate around the pipeline context information and a timeout.</p><p>The return value is a collection of all the observed message contexts - so the "Assert" step can look at everything sent/received.</p><p>First, I want to default a timeout and initialize some collectors:</p><pre><code class="language-csharp">timeout = Debugger.IsAttached ? (TimeSpan?)null : timeout ?? TimeSpan.FromSeconds(10); var incomingMessageContexts = new List&lt;IIncomingLogicalMessageContext&gt;(); var outgoingMessageContexts = new List&lt;IOutgoingLogicalMessageContext&gt;(); var obs = Observable.Empty&lt;object&gt;();</code></pre><p>The Activity API is a two-level set of observables. The first is when the initial diagnostic listener starts up, and the second is when diagnostic events are published. It's a little confusing to be sure, but not terrible. We want to first subscribe to all listeners when the two listener names we care about are published:</p><pre><code class="language-csharp">using var allListenerSubscription = DiagnosticListener.AllListeners .Subscribe(listener =&gt; { switch (listener.Name) { case ActivityNames.IncomingLogicalMessage: // Subscribe to the incoming listener break; case ActivityNames.OutgoingLogicalMessage: // Subscribe to the outgoing listener break; } }; </code></pre><p>When we run our application, our <code>Subscribe</code> method will be called for all diagnostic listeners starting up - the ASP.NET Core ones, HttpClient ones, SqlClient ones etc. but we only care about the NServiceBus logical message listeners.</p><p>Once we have this set up, we need to do 2 things inside those each switch block:</p><ul><li>Capture the observable so that we can await it based on our predicate later</li><li>Capture all contexts in those lists</li></ul><p>For each, we can rely on Reactive Extensions via the <code>System.Reactive.Core</code> library. Here's the incoming logical message block:</p><pre><code class="language-csharp">var incomingObs = listener .Select(e =&gt; e.Value) .Cast&lt;IIncomingLogicalMessageContext&gt;(); incomingObs.Subscribe(incomingMessageContexts.Add); if (typeof(TMessageContext) == typeof(IIncomingLogicalMessageContext)) { obs = obs.Merge(incomingObs); } break; </code></pre><p>The <code>listener</code> passed in is an <code>IObservable&lt;KeyValuePair&lt;string, object&gt;&gt;</code> where the <code>object</code> is that message context object. The first thing we can do is use LINQ to treat this observable as <code>IObservable&lt;*message context type&gt;</code>. Next, we add a single subscription action to simply capture the <code>IIncomingLogicalMessageContext</code> values on the list we created earlier.</p><p>Next, because I can test <em>multiple</em> hosts and endpoints, we <code>Merge</code> the different listener observables into our single <code>obs</code> variable we declared earlier, as our diagnostic listener object is created per endpoint.</p><p>The outgoing message switch block looks almost identical:</p><pre><code class="language-csharp">var outgoingObs = listener .Select(e =&gt; e.Value) .Cast&lt;IOutgoingLogicalMessageContext&gt;(); outgoingObs.Subscribe(outgoingMessageContexts.Add); if (typeof(TMessageContext) == typeof(IOutgoingLogicalMessageContext)) { obs = obs.Merge(outgoingObs); } break; </code></pre><p>The idea is we're only listening to incoming <em>or</em> outgoing messages, so I use that generic parameter to understand which one we care about.</p><p>Now that we've got our captured and merged observable, we can construct or final observable that includes the predicate and timeout, if exists (it won't if we're debugging):</p><pre><code class="language-csharp">var finalObs = obs.Cast&lt;TMessageContext&gt;().TakeUntil(predicate); if (timeout != null) { finalObs = finalObs.Timeout(timeout.Value); } </code></pre><p>With our observable in place, we can execute our test action and await for our observable to complete (either by satisfying our predicate or timing out):</p><pre><code class="language-csharp">await testAction(); // Force the observable to complete await finalObs; return new ObservedMessageContexts( incomingMessageContexts, outgoingMessageContexts); </code></pre><p>Once the final observable sequence completes, we can return back to our test the list of observed incoming/outgoing message contexts we capture earlier.</p><p>With this in place, our tests have a simple filter they can pass in based on their logical message type, with all the sent/received messages to assert against:</p><pre><code class="language-csharp">[Fact] public async Task Should_execute_orchestration_saga() { var client = _fixture.WebAppHost.CreateClient(); var message = Guid.NewGuid().ToString(); var response = await ExecuteAndWaitForHandled&lt;SaySomethingResponse&gt;(() =&gt; client.GetAsync($"saysomething?message={message}"), TimeSpan.FromSeconds(30)); var saySomethingResponses = response.ReceivedMessages.OfType&lt;SaySomethingResponse&gt;().ToList(); saySomethingResponses.Count.ShouldBe(1); saySomethingResponses[0].Message.ShouldContain(message); } </code></pre><p>It took a little bit to get the observables to work correctly, but once in, we can easily add end-to-end integration testing for complex messaging scenarios. Observability has a long tail of payoffs, as I found with integration testing.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Avn6mG1q9C4:NXdBes-VE9w:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Avn6mG1q9C4:NXdBes-VE9w:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Avn6mG1q9C4:NXdBes-VE9w:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Avn6mG1q9C4:NXdBes-VE9w:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Avn6mG1q9C4:NXdBes-VE9w:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/Avn6mG1q9C4" height="1" width="1" alt=""/> End-to-End Integration Testing with NServiceBus https://jimmybogard.com/end-to-end-integration-testing-with-nservicebus/ Jimmy Bogard urn:uuid:0bad8a81-6f57-0741-4556-94624cdf7ae2 Wed, 08 Jul 2020 16:43:12 +0000 <p>One of the interesting side effects of adding diagnostic events to infrastructure is that you can now "listen in" to what's going on in your applications for black box testing. This can be especially useful in scenarios where you're building on top of a framework that includes a lot of</p> <p>One of the interesting side effects of adding diagnostic events to infrastructure is that you can now "listen in" to what's going on in your applications for black box testing. This can be especially useful in scenarios where you're building on top of a framework that includes a lot of built-in behavior, such as ASP.NET Core and NServiceBus.</p><p>In ASP.NET Core, you can write tests directly against your controllers/handlers/pages, but that won't execute the entire request pipeline, only a little snippet. Similarly, you can write a test against an NServiceBus handler, but there's a lot more behavior going on around it.</p><p>To get an idea that the entire application can execute successfully, you need to actually <em>run</em> the application, which can be a huge challenge for message-oriented architectures that are by nature asynchronous.</p><p>To help with this, I created the <a href="https://github.com/jbogard/NServiceBus.Extensions.IntegrationTesting">NServiceBus.Extensions.IntegrationTesting</a> project that leverages the <a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">diagnostics events</a> I blogged about to listen in to messages sent/received.</p><h3 id="usage">Usage</h3><p>I created a rather complex scenario in that series, in which an API call would result in either an orchestration or choreography-based workflow, depending on the API call I used. Eventually, this worklow terminates, but the only way I could test this entire flow was to run the entire system.</p><p>If I want to test the entire end-to-end system, I can start with the <a href="https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1">integration testing capabilities with ASP.NET Core</a>. This package creates a special test host and test HTTP client that executes the entire pipeline, but all in-memory.</p><p>I want to do similar with NServiceBus, except using the <a href="https://docs.particular.net/transports/learning/">Learning Transport</a> instead of a "real" transport. That way, I don't have to worry about provisioning queues in test environments.</p><p>The NServiceBus integration test project includes two extensions:</p><ul><li>A helper method to configure your endpoint for integration testing</li><li>A fixture class that observes the diagnostic events around some supplied action</li></ul><p>In our Xunit tests, we create a derived <code>WebApplicationFactory</code> that includes the test setup for our NServiceBus host:</p><pre><code class="language-csharp">public class TestFactory : WebApplicationFactory&lt;HostApplicationFactoryTests&gt; { public EndpointFixture EndpointFixture { get; } public TestFactory() =&gt; EndpointFixture = new EndpointFixture(); protected override IHostBuilder CreateHostBuilder() =&gt; Host.CreateDefaultBuilder() .UseNServiceBus(ctxt =&gt; { var endpoint = new EndpointConfiguration("HostApplicationFactoryTests"); endpoint.ConfigureTestEndpoint(); return endpoint; }) .ConfigureWebHostDefaults(b =&gt; b.Configure(app =&gt; {})); </code></pre><p>In the above example, we're using a Worker Service that doesn't run a web host. However, the <code>WebApplicationFactory</code> still expects one, so we create an empty web application. The <code>ConfigureTestEndpoint</code> method configures our endpoint to run in a test mode (turn off auditing, retries, etc.)</p><p>Finally, in our test class, we add our fixture:</p><pre><code class="language-csharp">public class HostApplicationFactoryTests : IClassFixture&lt;HostApplicationFactoryTests.TestFactory&gt; { private readonly TestFactory _factory; public HostApplicationFactoryTests(TestFactory factory) =&gt; _factory = factory; </code></pre><p>Now in our test, we execute our "Send" and wait for a message to be handled that we expect:</p><pre><code class="language-csharp">[Fact] public async Task Can_send_and_wait() { var firstMessage = new FirstMessage {Message = "Hello World"}; var session = _factory.Services.GetService&lt;IMessageSession&gt;(); var result = await _factory .EndpointFixture .ExecuteAndWaitForHandled&lt;FinalMessage&gt;(() =&gt; session.SendLocal(firstMessage)); result.IncomingMessageContexts.Count.ShouldBe(3); result.OutgoingMessageContexts.Count.ShouldBe(3); result.ReceivedMessages.ShouldNotBeEmpty(); var message = result.ReceivedMessages.OfType&lt;FinalMessage&gt;().Single(); message.Message.ShouldBe(firstMessage.Message); } </code></pre><p>This simple workflow is 3 different message handlers in a row, and kick things off with an initial message. However, we can get more complicated and create a fixture that includes multiple different test hosts:</p><pre><code class="language-csharp">public class SystemFixture : IDisposable { public WebAppFactory WebAppHost { get; } public WebApplicationFactory&lt;Program&gt; WorkerHost { get; } public ChildWorkerServiceFactory ChildWorkerHost { get; } public EndpointFixture EndpointFixture { get; } </code></pre><p>Our integration test can now send an API call via the web host, and it kick off the messages to execute the entire saga until it's complete:</p><pre><code class="language-csharp">[Fact] public async Task Should_execute_orchestration_saga() { var client = _fixture.WebAppHost.CreateClient(); var message = Guid.NewGuid().ToString(); var response = await _fixture.EndpointFixture.ExecuteAndWaitForHandled&lt;SaySomethingResponse&gt;(() =&gt; client.GetAsync($"saysomething?message={message}"), TimeSpan.FromSeconds(30)); var saySomethingResponses = response.ReceivedMessages.OfType&lt;SaySomethingResponse&gt;().ToList(); saySomethingResponses.Count.ShouldBe(1); saySomethingResponses[0].Message.ShouldContain(message); } </code></pre><p>This snippet executes the entire workflow:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/7/81636_Picture0078.png" class="kg-image"><figcaption>Orchestration workflow</figcaption></figure><p>It kicks off the test with the initial POST, then waits until the final REPLY to complete the test. Each application runs in a test host, listening to messages, sending messages, making API calls etc. I just need to know what "Done" should be, so I wait for the <code>SaySomethingResponse</code> message to be handled.</p><p>The <code>EndpointFixture</code> class also lets me listen for messages sent, in case "done" is some message that gets sent but not handled by these systems.</p><p>I find these kinds of integration tests especially useful for process manager/saga scenarios, where I'm pushing over the first domino, and I want to test that the last domino falls successfully. I still expect to have tests all along the way, but I want to have something that tests the entire scenario end-to-end in an environment that runs the entire stack.</p><p>In the next post, I'll walk through the code in the integration tests extensions project to show how I used diagnostics events and Reactive Extensions to "know" when to stop.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=FPhpJGs7Tg4:1VdSatFwEYI:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=FPhpJGs7Tg4:1VdSatFwEYI:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=FPhpJGs7Tg4:1VdSatFwEYI:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=FPhpJGs7Tg4:1VdSatFwEYI:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=FPhpJGs7Tg4:1VdSatFwEYI:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/FPhpJGs7Tg4" height="1" width="1" alt=""/> Becoming a Technical Fellow at Headspring, and New Beginnings https://jimmybogard.com/becoming-technical-fellow-at-headspring-and-new-beginnings/ Jimmy Bogard urn:uuid:351f284b-ba2a-7eba-c7aa-4804af85c417 Tue, 23 Jun 2020 19:46:43 +0000 <p>My journey at <a href="https://headspring.com/">Headspring</a> started for me at a local developer meetup, the "Austin Agile Lunch" group, way back in 2007. I had started blogging and getting involved in the community, attending the local .NET user group, code camp, and most influentially, the first ALT.NET open space conference. At</p> <p>My journey at <a href="https://headspring.com/">Headspring</a> started for me at a local developer meetup, the "Austin Agile Lunch" group, way back in 2007. I had started blogging and getting involved in the community, attending the local .NET user group, code camp, and most influentially, the first ALT.NET open space conference. At this lunch, <a href="https://jeffreypalermo.com/">Jeffrey Palermo</a>, who had started at Headspring that year, asked the innocuous question "Do you know anyone looking for a job?" And the naive me answered "No, but if I know of anyone, I'll be glad to point them your way!" Jeffrey, always direct, tried again, "Do you know anyone named JIMMY looking for a job?"</p><p>I had been quite unhappy at my current gig (as anyone who's seen my talks about it can attest), but wasn't quite sure what was next. I had done a cash-flow challenged startup, a product company, and Big Corp IT, all in the span of the first 4 years of my career. Although Headspring was only 5 people at the time, it offered something I couldn't get anywhere else. An opportunity to drive for excellence in an environment where everyone else had a similar passion, and an ability to affect positive change at a level I couldn't dream of before.</p><p>Within a year or so after I started, thanks to support and encouragement from coworkers and the opportunities I now had, I received my first Microsoft MVP award, co-authored my first book, published my first OSS project used in production, and wrote dozens of blog posts of everything I was working on, all in trying to live up to a promise I made to myself to pass on what I had learned as I had indebted myself to so many others who shared their own learnings.</p><p>To pay it forward.</p><p>This culminated in Headspring recognizing me as the <a href="https://headspring.com/2020/06/09/jimmy-bogard-is-named-headsprings-first-ever-technical-fellow/">first-ever Technical Fellow</a>, an honor I never set out to receive, and I'm even more indebted to others to repay.</p><p>And now, 13 years later, my Headspring journey shifts (again).</p><p>For some years, I looked at going independent as the next logical step for my career. However, I love working at Headspring, the people, the clients, and the work, and could not imaging giving that up, for almost any opportunity. I recognized that although I <em>could</em> ask for the flexibility of an independent consultant, I <em>would</em> not ask for, nor accept, that sort of special treatment. As it became clear that I could go independent, I approached the Headspring leadership to look at possibilities of being independent while also at Headspring.</p><p>Thanks to the great support of Dustin and rest of the Headspring leadership, we were able to come to a mutually beneficial agreement. While I'm continuing my role as the Chief Architect at Headspring, I've stepped back as a full-time employee and staying on as an independent contractor (Jimmy Bogard Consulting LLC, I'm not going to win any awards for creativity).</p><p>My role as Chief Architect I hope (and plan) to continue for years to come, as I believe in Headspring's vision and future, and we've signed a partnership agreement to that effect. What changes for me is my day-to-day projects and clients may or may not be strictly Headspring projects and clients, or any projects at all (maybe become a true brisket master?). My hope is to be able to have the flexibility to consult on the kinds of opportunities that are only feasible as an independent consultant, while still helping to fulfill the vision and mission of Headspring that I've helped to define and shape for over a decade.</p><p>As my grandfather was fond of saying, "Onward and upward".</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=T7JGqgq9LDk:RY1f4kvepQw:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=T7JGqgq9LDk:RY1f4kvepQw:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=T7JGqgq9LDk:RY1f4kvepQw:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=T7JGqgq9LDk:RY1f4kvepQw:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=T7JGqgq9LDk:RY1f4kvepQw:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/T7JGqgq9LDk" height="1" width="1" alt=""/> Building End-to-End Diagnostics: User-Defined Context with Correlation Context https://jimmybogard.com/building-end-to-end-diagnostics-user-defined-context-with-correlation-context/ Jimmy Bogard urn:uuid:907a2f36-e2f4-df10-3b2d-87493402be8a Mon, 22 Jun 2020 13:06:04 +0000 <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/">Activity and Span Correlation</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-visualizations-with-exporters/">Visualization with Exporters</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-user-defined-context-with-correlation-context/">User-Defined Context with Correlation Context</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>With a brief detour to <a href="https://jimmybogard.com/diagnostics-and-instrumentation-for-mongodb-and-nservicebus/">push out some NuGet packages</a>, I wanted to pick up with a common issue folks run into once they</p> <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/">Activity and Span Correlation</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-visualizations-with-exporters/">Visualization with Exporters</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-user-defined-context-with-correlation-context/">User-Defined Context with Correlation Context</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>With a brief detour to <a href="https://jimmybogard.com/diagnostics-and-instrumentation-for-mongodb-and-nservicebus/">push out some NuGet packages</a>, I wanted to pick up with a common issue folks run into once they start implementing distributed tracing. And that problem is one of locality - trying to find <em>one</em> trace among many, the needle in the haystack. In the past, I've resorted to combing through logs, using timestamps as a very poor means of trying to diagnose an issue.</p><p>We can decorate our logs with context information to make searching easier, but when it comes to our traces, how can we triangulate something a user did with a workflow and the associated traces?</p><p>This is where the emerging <a href="https://w3c.github.io/correlation-context/">Correlation Context standard</a> comes in to play - the ability for application code to add arbitrary information to traces - and critically - have that information flow through subsequent spans.</p><p>With distributed trace spans, you can add information to a span so that it eventually shows up in a collector/exporter. This is possible today with the System.Diagnostics.Activity API through tags:</p><pre><code class="language-csharp">Activity.Current.AddTag("user.id", user.Id);</code></pre><p>But that information does not flow through to any subsequent spans within a process, or to subsequent processes. It exists for a single Activity/span, then it's gone.</p><p>This is where Correlation Context comes in. Trace context spec defines a <a href="https://www.w3.org/TR/trace-context/#tracestate-header">"tracestate" header</a> for <em>vendor-specific</em> trace information to propagate, and Correlation Context allows application code to add <em>application-specific</em> trace information to propagate.</p><h3 id="propagating-correlation-context-in-nservicebus">Propagating Correlation Context in NServiceBus</h3><p>ASP.NET Core will <a href="https://github.com/dotnet/aspnetcore/blob/b6698757a85f9b6d6c63311e64dd6ac9e734ef56/src/Hosting/Hosting/src/Internal/HostingApplicationDiagnostics.cs#L269">automatically parse correlation context header information</a>, and places this in <code>Activity.Baggage</code>:</p><pre><code class="language-csharp">string[] baggage = headers.GetCommaSeparatedValues(HeaderNames.CorrelationContext); if (baggage.Length &gt; 0) { foreach (var item in baggage) { if (NameValueHeaderValue.TryParse(item, out var baggageItem)) { activity.AddBaggage(baggageItem.Name.ToString(), HttpUtility.UrlDecode(baggageItem.Value.ToString())); } } } </code></pre><p>Baggage, unlike Tags, will flow through to child Activities. When an Activity starts, the parent <code>Activity.Current</code> will flow its ParentId through to the new Activity. When you access an Activity's <code>Baggage</code>, the implementation pulls the current baggage, its Parent's baggage, and every parent up the chain.</p><p>With NServiceBus, we want to also parse incoming baggage, and propagate outgoing baggage through the <code>Correlation-Context</code> header. And although this header is still in draft mode with the W3C, it already has implementation support in ASP.NET Core 3.0.</p><p>To parse incoming headers, we can do the same operation that ASP.NET Core does, in our code that parses the incoming <code>traceparent</code>, we can look for the <code>Correlation-Context</code> header and place the values in <code>Activity.Baggage</code>:</p><pre><code class="language-csharp">if (context.MessageHeaders.TryGetValue(Headers.CorrelationContextHeaderName, out var correlationContext)) { var baggage = correlationContext.Split(','); if (baggage.Length &gt; 0) { foreach (var item in baggage) { if (NameValueHeaderValue.TryParse(item, out var baggageItem)) { activity.AddBaggage(baggageItem.Name, HttpUtility.UrlDecode(baggageItem.Value)); } } } } </code></pre><p>Now that we have correlation context in our Baggage, any other child activities will have this baggage, too.</p><p>The last piece is propagation, in our original outgoing NServiceBus behavior that propagates <code>traceparent</code>:</p><pre><code class="language-csharp">if (!context.Headers.ContainsKey(Headers.CorrelationContextHeaderName)) { var baggageItems = activity.Baggage.Select(item =&gt; $"{item.Key}={item.Value}"); var headerValue = string.Join(",", baggageItems); if (!string.IsNullOrEmpty(headerValue)) { context.Headers[Headers.CorrelationContextHeaderName] = headerValue; } }</code></pre><p>Will now propagate the baggage out through the <code>Correlation-Context</code> header. With the incoming and outgoing header behaviors in place, any service can drop some data into baggage and have it propagate to all downstream services.</p><p>So all done, right? Well, not quite, as even though we added information to the Activity.Baggage, it doesn't necessarily mean that those values get exported to our tracing tools. Unfortunately today, OpenTelemetry exporters only consider the <code>Tags</code> portion of an <code>Activity</code> for exporting. This will be opt-in in the future, but for now, we'll need to manually copy our Baggage to Tags during the export process (or in child activities).</p><p>In the next post, we'll walk through exactly that - creating a "breadcrumb" Activity that piggybacks on events of other activities.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=4O8vAXNvmiY:8fUsmSDPBNs:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=4O8vAXNvmiY:8fUsmSDPBNs:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=4O8vAXNvmiY:8fUsmSDPBNs:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=4O8vAXNvmiY:8fUsmSDPBNs:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=4O8vAXNvmiY:8fUsmSDPBNs:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/4O8vAXNvmiY" height="1" width="1" alt=""/> Diagnostics and Instrumentation Packages for MongoDB and NServiceBus Published https://jimmybogard.com/diagnostics-and-instrumentation-for-mongodb-and-nservicebus/ Jimmy Bogard urn:uuid:8b9bcf54-74f8-1bc7-8dcf-8f643eac206a Wed, 17 Jun 2020 14:25:00 +0000 <p>As part of the <a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">end-to-end diagnostics and tracing blog series</a>, I had an end goal of eventually publishing out NuGet packages for diagnostics (Activity and DiagnosticSource). I'm happy to announce that I've released 4 packages to NuGet:</p><ul><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics/">NServiceBus.Extensions.Diagnostics</a></li><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics.OpenTelemetry/">NServiceBus.Extensions.Diagnostics.OpenTelemetry</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.DiagnosticSources/">MongoDB.Driver.Core.Extensions.DiagnosticSources</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.OpenTelemetry/">MongoDB.</a></li></ul> <p>As part of the <a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">end-to-end diagnostics and tracing blog series</a>, I had an end goal of eventually publishing out NuGet packages for diagnostics (Activity and DiagnosticSource). I'm happy to announce that I've released 4 packages to NuGet:</p><ul><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics/">NServiceBus.Extensions.Diagnostics</a></li><li><a href="https://www.nuget.org/packages/NServiceBus.Extensions.Diagnostics.OpenTelemetry/">NServiceBus.Extensions.Diagnostics.OpenTelemetry</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.DiagnosticSources/">MongoDB.Driver.Core.Extensions.DiagnosticSources</a></li><li><a href="https://www.nuget.org/packages/MongoDB.Driver.Core.Extensions.OpenTelemetry/">MongoDB.Driver.Core.Extensions.OpenTelemetry</a></li></ul><p>These packages add:</p><ul><li><a href="https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/ActivityUserGuide.md">Activity</a> and <a href="https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md">DiagnosticsSource</a> support</li><li><a href="https://opentelemetry.io/">OpenTelemetry</a> adapters</li><li><a href="https://www.w3.org/TR/trace-context/">W3C Trace Context</a> and <a href="https://w3c.github.io/correlation-context/">Correlation Context</a> support for NServiceBus</li></ul><p>With the diagnostics packages, you can write your own <code>DiagnosticListener</code>s to subscribe to diagnostics Activity events. The OpenTelemetry packages listen to these Activity events and adapt them to OpenTelemetry spans. The various existing exporters can output these spans to Zipkin, Jaeger, Application Insights, and many other observability tools.</p><p>I've updated the ReadMe's for each repo to include instructions for how to use:</p><ul><li><a href="https://github.com/jbogard/MongoDB.Driver.Core.Extensions.DiagnosticSources">Mongo.Driver.Core.Extensions.DiagnosticSources</a></li><li><a href="https://github.com/jbogard/NServiceBus.Extensions.Diagnostics">NServiceBus.Extensions.Diagnostics</a></li></ul><p>With these packages, you can easily configure OpenTelemetry to include MongoDB and NServiceBus support:</p><pre><code class="language-csharp">services.AddOpenTelemetry(builder =&gt; builder .UseZipkin(o =&gt; { o.Endpoint = new Uri("http://localhost:9411/api/v2/spans"); o.ServiceName = EndpointName; }) .AddNServiceBusAdapter(opt =&gt; opt.CaptureMessageBody = true) .AddMongoDBAdapter(opt =&gt; opt.CaptureCommandText = true) .AddRequestAdapter() .AddDependencyAdapter(configureSqlAdapterOptions: opt =&gt; opt.CaptureTextCommandContent = true); );</code></pre><p>Enjoy!</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=kjvtnvKICWo:AMdNMhLvCn0:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=kjvtnvKICWo:AMdNMhLvCn0:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=kjvtnvKICWo:AMdNMhLvCn0:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=kjvtnvKICWo:AMdNMhLvCn0:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=kjvtnvKICWo:AMdNMhLvCn0:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/kjvtnvKICWo" height="1" width="1" alt=""/> Picking a Web Microframework https://lostechies.com/ryansvihla/2020/05/27/picking-a-microframework/ Los Techies urn:uuid:b75786e5-449a-a555-4277-5555fd14eb08 Wed, 27 May 2020 00:23:00 +0000 I’ve had to use this at work the last couple of weeks. We had a “home grown” framework for a new application we’re working and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization anyway to do routes, etc). <p>I’ve had to use this at work the last couple of weeks. We had a “home grown” framework for a new application we’re working and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization anyway to do routes, etc).</p> <p>However, being that the group I was working with is pretty “anti framework” I had to settle on something that was light weight, integrated with jetty and allowed us to work the way that was comfortable for us as team (also it had to work with Scala).</p> <h2 id="microframeworks">Microframeworks</h2> <p>The team had shown a lot of disdain for Play (which I had actually quite a lot when I last was leading a JVM based tech stack) and Spring Boot as being too heavy weight, so these were definitely out.</p> <p>Fortunately, in the JVM world there is a big push back now on heavy web frameworks so meant I had lots of choices for “non frameworks” but could still do some basic security, routing, authentication but not hurt the existing team’s productivity.</p> <p>There are probably 3 dozen microframeworks to choose from with varying degrees of value but the two that seemed to easiest to start with today were:</p> <ul> <li><a href="https://scalatra.org">Scalatra</a></li> <li><a href="https://javalin.io">Javalin</a></li> <li><a href="https://quarkus.io">Quarkus</a></li> </ul> <h3 id="my-attempt-with-quarkus">My Attempt with Quarkus</h3> <p><a href="https://quarkus.io/">Quarkus</a> has a really great getting started story but it’s harder to get started on an existing project with it, it was super trivial to add, and after a couple of days of figuring out the magic incantation I just decided to punt on it. I think because of it’s popularity in the Cloud Native space (which we’re trying to target), the backing of <a href="https://developers.redhat.com/blog/2019/03/07/quarkus-next-generation-kubernetes-native-java-framework/">Red Hat</a>, and the pluggable nature of the stack there are a lot of reasons to want this to work. In the end because of the timeline it didn’t make the cut. But it may come back.</p> <h3 id="my-attempt-with-javalin">My Attempt with Javalin</h3> <p>Javalin despite being a less popular project than Quarkus it is getting some buzz. It also looks like it just slides into the team’s existing Servlet code base. I wanted this to work very badly but stopped before I even started because of <a href="https://github.com/tipsy/javalin/issues/931">this issue</a> so this was out despite being on paper a really execellent framework.</p> <h3 id="my-attempt-with-scalatra">My Attempt with Scalatra</h3> <p><a href="https://scalatra.org/">Scalatra</a> has been around for a number of years and is inspired by <a href="http://sinatrarb.com/">Sinatra</a> which I used quite a bit in my Ruby years. This took a few minutes to get going just following their <a href="https://scalatra.org/guides/2.7/deployment/standalone.html">standalone directions</a> and then some more to successful convert the routes and account for learning curves with routes.</p> <p>Some notes:</p> <ul> <li>The routing API and parameters etc are very nice to work with IMO.</li> <li>It was <a href="https://scalatra.org/guides/2.7/formats/json.html">very easy</a> to get json by default support setup.</li> <li>Metrics were <a href="https://scalatra.org/guides/2.7/monitoring/metrics.html">very easy</a> to wire up.</li> <li>Swagger integration was pretty rough, while it looks good on paper I could not get an example to show up, and it is unable to <a href="https://github.com/scalatra/scalatra/issues/343">handle case classes or enums</a> which we use.</li> <li>Benchmark performance when I’ve <a href="https://johnykov.github.io/bootzooka-akka-http-vs-scalatra.html">looked</a> around the web was pretty bad, I’ve not done enough to figure out if this is real or not. I’ve seen first hand a lot of benchmarking are just wrong.</li> <li>Integration with JUnit has been rough and I cannot seem to get the correct port to fire, I suspect I have to stop using the @Test annotation is all (which I’m not enjoying).</li> <li>Http/2 support is still lacking despite being available in the version of Jetty they’re on, I’ve read a few places that an issue is keeping <a href="https://github.com/eclipse/jetty.project/issues/1364">web sockets working</a> but either way there is <a href="https://github.com/scalatra/scalatra/issues/757">no official support in the project yet</a>.</li> </ul> <h2 id="conclusion">Conclusion</h2> <p>I think we’re going to stick with Scalatra for the time being as it is a muture framework that works well for our current goals. However, the lack of http/2 support maybe a deal breaker in the medium term.</p> Building End-to-End Diagnostics: Visualization with Exporters https://jimmybogard.com/building-end-to-end-diagnostics-visualizations-with-exporters/ Jimmy Bogard urn:uuid:8d7d0ca1-a2a2-e1a7-97b5-6ad550dd68e5 Tue, 19 May 2020 14:46:42 +0000 <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/">Activity and Span Correlation</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-visualizations-with-exporters/">Visualization with Exporters</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>In the last post, we looked at surfacing our diagnostics events from the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.activity?view=netcore-3.1">Activity</a> and <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.diagnosticsource?view=netcore-3.1">DiagnosticSource</a> APIs on through to <a href="https://opentelemetry.io/">OpenTelemetry</a> Spans.</p><blockquote>Side note - the .NET team is</blockquote> <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/">Activity and Span Correlation</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-visualizations-with-exporters/">Visualization with Exporters</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>In the last post, we looked at surfacing our diagnostics events from the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.activity?view=netcore-3.1">Activity</a> and <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.diagnosticsource?view=netcore-3.1">DiagnosticSource</a> APIs on through to <a href="https://opentelemetry.io/">OpenTelemetry</a> Spans.</p><blockquote>Side note - the .NET team is working to close the gap between the OTel "Span" concept and the Activity API. <a href="https://medium.com/opentelemetry/opentelemetry-net-sdk-progress-3a63dcdc6cb0">The end goal is that the Activity class represents an OTel "Span".</a> </blockquote><p>With all of these pieces in place, we can now export these traces to...well, something! The benefit of this common OTel API means that it should be much easier to export your traces to some collector that can then store, index, and visualize your traces. But before we look at visualization, let's build a couple of complex but contrived scenarios for a more real-world scenario distributed system.</p><h3 id="orchestration-sample">Orchestration Sample</h3><p>I wanted to try to use as many different distributed components as possible that I typically see in my client work, in an arrangement of different <a href="https://www.enterpriseintegrationpatterns.com/patterns/conversation/">conversation patterns</a>, so I wanted an example system that did:</p><ul><li>Request/Response with HTTP</li><li>Request/Response with AMQP</li><li>Pub/Sub with AMQP</li><li>Orchestration and Choreography with AMQP</li><li>Database interaction with SQL and MongoDB</li></ul><p>There's probably someone winning a distributed systems bingo at this point. My sample application will have 3 different endpoints:</p><ul><li>Web Application (receives the first request)</li><li>Worker Application (<a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&amp;tabs=visual-studio#worker-service-template">Worker SDK</a>)</li><li>Child Worker Application (another Worker SDK)</li></ul><p>Because I'm lazy, I'll kick things off with the Swagger UI, and initiate 2 kinds of interactions:</p><ul><li>Process manager using <a href="https://microservices.io/patterns/data/saga.html">orchestration</a></li><li>Process manager using <a href="https://microservices.io/patterns/data/saga.html">choreography</a></li></ul><p>First, let's look at an orchestration conversation:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/5/191222_Picture0078.png" class="kg-image"><figcaption>Conversation using orchestration with HTTP and AMQP</figcaption></figure><p>The code is a bit much, but let's walk through the interactions:</p><ol><li>HTTP POST to Web Application</li><li>AMQP Send to Worker Application</li><li>Worker Application receives message and starts process manager; AMQP Send to itself</li><li>Worker Application receives message, makes HTTP POST to Web Application</li><li>Web Application makes SQL call</li><li>Worker Application receives HTTP reply, makes AMQP Reply</li><li>Worker Application process manager receives reply, makes AMQP Send to Child Application</li><li>Child Application receives message, makes MongoDB call and makes AMQP Reply to Worker Application</li><li>Worker Application process manager receives reply, marks process as complete and makes AMQP Reply to process manager initiator</li><li>Web Application receives message and conversation completes</li></ol><p>WHEW. Now this conversation does resemble ones I've built, but it is a bit contrived as I don't usually have these back-and-forth communications. But it does show the different kinds of communications in a complex conversation.</p><p>If something were to go awry in this conversation, we wouldn't have much recourse to fix anything without distributed tracing.</p><h3 id="exporting-traces-with-opentelemetry">Exporting Traces with OpenTelemetry</h3><p>Configuring our applications to export their traces and spans is easy, first we add the appropriate exporter packages:</p><pre><code class="language-xml">&lt;PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="0.2.0-alpha.283" /&gt; &lt;PackageReference Include="OpenTelemetry.Exporter.Zipkin" Version="0.2.0-alpha.283" /&gt; </code></pre><p>Once we have our packages referenced, we can configure them with the rest of our OpenTelemetry setup:</p><pre><code class="language-csharp">services.AddOpenTelemetry(builder =&gt; builder .UseZipkin(o =&gt; { o.Endpoint = new Uri("http://localhost:9411/api/v2/spans"); o.ServiceName = Program.EndpointName; }) .UseJaeger(c =&gt; { c.AgentHost = "localhost"; c.AgentPort = 6831; c.ServiceName = Program.EndpointName; }) .AddNServiceBusAdapter() .AddRequestAdapter() .AddDependencyAdapter(configureSqlAdapterOptions: opt =&gt; opt.CaptureTextCommandContent = true)); </code></pre><p>Each call to an exporter needs some configuration on how to connect to it. Zipkin and Jaeger will need a URI, while Application Insights would need our instrumentation key.</p><p>That's it! Once we've configured our exporters, our application will export them seamlessly as it handles requests. Let's look at traces for our orchestration-style communication (and we should see all the steps above), first in <a href="https://zipkin.io/">Zipkin</a>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/5/191238_Picture0079.png" class="kg-image"><figcaption>Zipkin trace of orchestration conversation</figcaption></figure><p>And next in <a href="https://www.jaegertracing.io/">Jaeger</a>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/5/19131_Picture0080.png" class="kg-image"><figcaption>Jaeger trace of orchestration conversation</figcaption></figure><p>In each of these traces, we can follow the conversation all the way from the initial HTTP POST all the way down to the final AMQP "reply" back to the process manager originator. And because this is an orchestration-style process, the execution times are completely linear - no process hands off to another until it completes.</p><p>Without our integration into NServiceBus, we wouldn't have this connected conversation - only disconnected individual traces for each HTTP request. With OpenTelemetry, we've got a complete view of our entire conversation, with each step recorded along the way.</p><p>Let's contrast this with a choreography-style conversation to see how that might look.</p><h3 id="choreography-sample">Choreography Sample</h3><p>In our orchestration sample, we used commands and replies to dictate a flow for our conversation. With choreography, our communication is more event-oriented, and multiple steps can execute in parallel. Our choreography conversation looks like:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/5/191317_Picture0081.png" class="kg-image"><figcaption>Conversation using choreography with HTTP and AMQP</figcaption></figure><p>This conversation is a bit different than the orchestration conversation because we're using events, rather than commands, to coordinate an activity. This allows some processes to execute in parallel. The worker application and child application now handle the published event simultaneously, and publish an event when their step completes.</p><p>Finally, our process manager subscribes to the "complete" event from both processes, and when both events are received, the process manager publishes its final event back to the Web Application.</p><p>We can see that there are fewer messages in this approach, as each application uses an event to fire, as opposed to the two messages involved in request/response. With our choreography application set up, let's look to see how this conversation manifests in Zipkin:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/5/191419_Picture0082.png" class="kg-image"><figcaption>Zipkin trace of choreography conversation</figcaption></figure><p>And in Jaeger:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/5/191421_Picture0083.png" class="kg-image"><figcaption>Jaeger trace for choreography conversation</figcaption></figure><p>In each of these traces, we see the interaction kick off with the Web Application publishing an event. Next, both the Child Worker Service and Worker Service execute in parallel, making requests and doing work. We also see the Child Worker Service publishing an event to the Worker Service process manager, but it is not until the <em>second</em> event that the process manager receives that the final publish and receive from the Web Application happens.</p><p>Looking at each of these traces, we can see how the conversation patterns differ in their communication style and processing.</p><p>With OpenTelemetry, we've made it simple to add end-to-end distributed tracing to our system. None of our application code is aware of OpenTelemetry even existing, and with the upcoming changes to the Activity API, adding distributed tracing will be made even easier.</p><p>In the next (and last) post, I'll share some future direction of OpenTelemetry, as well as links to the sample extensions made into released NuGet packages.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=JZcpXs5EC_A:2z-ZhMLxCB0:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=JZcpXs5EC_A:2z-ZhMLxCB0:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=JZcpXs5EC_A:2z-ZhMLxCB0:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=JZcpXs5EC_A:2z-ZhMLxCB0:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=JZcpXs5EC_A:2z-ZhMLxCB0:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/JZcpXs5EC_A" height="1" width="1" alt=""/> Building End-to-End Diagnostics: Activity and Span Correlation https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/ Jimmy Bogard urn:uuid:e5f80961-f7e4-0ba5-61c0-7defc1972747 Wed, 13 May 2020 14:23:28 +0000 <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/">Activity and Span Correlation</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>In the last post, we looked at hooking up our diagnostics events (and Activities) to <a href="https://opentelemetry.io/">OpenTelemetry</a>, where our main task was creating the appropriate span attributes based on the <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/README.md">OpenTelemetry Span conventions</a></p> <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-activity-and-span-correlation/">Activity and Span Correlation</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>In the last post, we looked at hooking up our diagnostics events (and Activities) to <a href="https://opentelemetry.io/">OpenTelemetry</a>, where our main task was creating the appropriate span attributes based on the <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/README.md">OpenTelemetry Span conventions</a>. When everything runs, we get lovely correlated events based on our W3C tracecontext headers flowing through, as our Activity maintains <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.activity.parentid?view=netcore-3.1">a link to its parent</a>, regardless of where that parent activity came from.</p><p>The Activity to Span connection works great, as long as we can maintain a link between the OTel "Span" and the Activity Start/Stop events themselves, and we're effectively able to start and stop the <em>right</em> activity when something interesting happens in our observed code.</p><p>Sometimes, it's not so easy to correlate the "start" and "stop" events for a given observed event. Consider our NServiceBus consumer activity logic:</p><pre><code class="language-csharp">public override async Task Invoke( IIncomingPhysicalMessageContext context, Func&lt;Task&gt; next) { var activity = StartActivity(context); try { await next().ConfigureAwait(false); } finally { StopActivity(activity, context); } } </code></pre><p>For the Start and Stop activities, we have the reference to the activity we started in that <code>activity</code> variable. We can do this because the infrastructure hook for consuming a message wraps the entire operation, and we're passed a continuation (that <code>next</code> parameter). But what happens if the infrastructure hooks we have for "start" and "stop" are physically separate? How do we know which <code>Activity</code> to stop once the observed event completes?</p><p>I ran into this exact situation with the <a href="https://mongodb.github.io/mongo-csharp-driver/">MongoDB .NET Driver</a>, which does have a mechanism to raise internal diagnostic events, but your handling of these events is physically separate in your code.</p><h3 id="correlating-events">Correlating Events</h3><p>The main interface to listen to internal diagnostic events in the MongoDB .NET Driver is the <code><a href="https://mongodb.github.io/mongo-csharp-driver/2.11/apidocs/html/T_MongoDB_Driver_Core_Events_IEventSubscriber.htm">IEventSubscriber</a></code> interface:</p><pre><code class="language-csharp">public interface IEventSubscriber { bool TryGetEventHandler&lt;TEvent&gt;(out Action&lt;TEvent&gt; handler); } </code></pre><p>We can register subscribers, but we only get this one method that gets called based on different <code>TEvent</code> types. That means that we don't get a single method for opening a connection or issuing a command. Those are separate calls with separate callbacks.</p><p>First, let's get some boilerplate to handle these events. There's a helper class, <code><a href="https://mongodb.github.io/mongo-csharp-driver/2.11/apidocs/html/T_MongoDB_Driver_Core_Events_ReflectionEventSubscriber.htm">ReflectionEventSubscriber</a></code>, that lets us define methods with a signature that accepts the event objects and the subscriber uses reflection to find these events. The events we're looking to handle are <code><a href="https://mongodb.github.io/mongo-csharp-driver/2.11/apidocs/html/T_MongoDB_Driver_Core_Events_CommandStartedEvent.htm">CommandStartedEvent</a></code>, <code><a href="https://mongodb.github.io/mongo-csharp-driver/2.11/apidocs/html/T_MongoDB_Driver_Core_Events_CommandSucceededEvent.htm">CommandSucceededEvent</a></code>, and <a href="https://mongodb.github.io/mongo-csharp-driver/2.11/apidocs/html/T_MongoDB_Driver_Core_Events_ConnectionFailedEvent.htm"><code>CommandFailedEvent</code></a>. Our event subscriber implementation can wrap this reflection-based event subscriber:</p><pre><code class="language-csharp">public class DiagnosticsActivityEventSubscriber : IEventSubscriber { private readonly ReflectionEventSubscriber _subscriber; public DiagnosticsActivityEventSubscriber() =&gt; _subscriber = new ReflectionEventSubscriber(this, bindingFlags: BindingFlags.Instance | BindingFlags.NonPublic); public bool TryGetEventHandler&lt;TEvent&gt;(out Action&lt;TEvent&gt; handler) =&gt; _subscriber.TryGetEventHandler(out handler); private void Handle(CommandStartedEvent @event) { // Start activity } private void Handle(CommandSucceededEvent @event) { // Stop activity } private void Handle(CommandFailedEvent @event) { // Stop activity } } </code></pre><p>As we can see above, the methods to handle the start/stop activities are physically separate, but we need to share our started <code>Activity</code> between these! We can store the started activity in a field, but when the stop events come in, we need to make sure we stop the <em>right</em> activity.</p><p>To find the "right" activity, we need some way to correlate the information in the <code>CommandStartedEvent</code> object and the <code>CommandSucceededEvent</code>/ <code>CommandFailedEvent</code> objects. Luckily, the Mongo drivers for other languages are quite similar to .NET, and some of those other drivers have OpenTelemetry implementations. From that code, we can see that these events have a <code>RequestId</code> that uniquely identifies this request against the MongoDB server. This could work!</p><h3 id="implementing-our-event-subscriber">Implementing our Event Subscriber</h3><p>First, we need the plumbing of our diagnostic source in our <code>DiagnosticsActivityEventSubscriber</code> class to raise the diagnostic events:</p><pre><code class="language-csharp">public class DiagnosticsActivityEventSubscriber : IEventSubscriber { public const string ActivityName = "MongoDB.Driver.Core.Events.Command"; private static readonly DiagnosticSource _diagnosticListener = new DiagnosticListener(ActivityName); </code></pre><p>Next, we somewhere to place our started <code>Activity</code> objects, correlated by the <code>RequestId</code> property. The most straightforward solution would be a <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2?view=netcore-3.1">ConcurrentDictionary</a></code> for this value:</p><pre><code class="language-csharp">private readonly ConcurrentDictionary&lt;int, Activity&gt; _activityMap = new ConcurrentDictionary&lt;int, Activity&gt;(); </code></pre><p>Now, when we start our activity, we follow the normal steps we did in our NServiceBus example, with the added step of adding the started activity to our dictionary:</p><pre><code class="language-csharp">private void Handle(CommandStartedEvent @event) { var activity = new Activity(ActivityName); if (_diagnosticListener.IsEnabled(CommandStarted.EventName, @event)) { _diagnosticListener.StartActivity(activity, new CommandStarted(@event)); } else { activity.Start(); } _activityMap.TryAdd(@event.RequestId, activity); } </code></pre><p>With the event started and stored locally, we can stop our activity based on pulling the activity back out of the local dictionary:</p><pre><code class="language-csharp">private void Handle(CommandSucceededEvent @event) { if (_activityMap.TryRemove(@event.RequestId, out var activity)) { _diagnosticListener.StopActivity(activity, new CommandSucceeded(@event)); } } private void Handle(CommandFailedEvent @event) { if (_activityMap.TryRemove(@event.RequestId, out var activity)) { _diagnosticListener.StopActivity(activity, new CommandFailed(@event)); } } </code></pre><p>Initially, I tried to use <code>Activity.Current</code> to pull out the activity, but that won't always be correct if the async context hasn't flowed through to my event handler (it did not). Instead, with this correlated dictionary, I can ensure I stop the <em>correct</em> activities and my eventual OTel spans will correlate as well.</p><h3 id="opentelemetry-integration">OpenTelemetry Integration</h3><p>On the OTel side, we have a similar issue where the "Current Span" assumes a common execution context. But since context isn't shared, we have to implement our <code>ListenerHandler</code> in a similar fashion, but this time, correlate a <code>TelemetrySpan</code>:</p><pre><code class="language-csharp">internal class CommandListener : ListenerHandler { public CommandListener(string sourceName, Tracer tracer) : base(sourceName, tracer) { } private readonly ConcurrentDictionary&lt;int, TelemetrySpan&gt; _spanMap = new ConcurrentDictionary&lt;int, TelemetrySpan&gt;(); </code></pre><p>Now when we receive the <code>OnStartActivity</code> call, we have to perform a similar operation to store our <code>TelemetrySpan</code> based on that <code>OperationId</code>:</p><pre><code class="language-csharp">public override void OnStartActivity(Activity activity, object payload) { if (!(payload is CommandStarted message)) { AdapterEventSource.Log.NullPayload("CommandListener.OnStartActivity"); return; } Tracer.StartActiveSpanFromActivity($"mongodb.{message.Event.CommandName}", activity, SpanKind.Client, out var span); SetSpanAttributes(span, message); _spanMap.TryAdd(message.Event.RequestId, span); } </code></pre><p>From here, stopping the activity will mean, instead of us accessing the <code>CurrentSpan</code>, pulling it out of our dictionary:</p><pre><code class="language-csharp">public override void OnStopActivity(Activity activity, object payload) { if (!(payload is CommandSucceeded message)) { AdapterEventSource.Log.NullPayload("CommandListener.OnStopActivity"); return; } if (_spanMap.TryRemove(message.Event.RequestId, out var span)) { span.End(); } } public override void OnException(Activity activity, object payload) { if (!(payload is CommandFailed message)) { AdapterEventSource.Log.NullPayload("CommandListener.OnExceptionActivity"); return; } if (_spanMap.TryRemove(message.Event.RequestId, out var span)) { span.Status = Status.Unknown.WithDescription(message.Event.Failure.Message); SetSpanAttributes(span, message); span.End(); } } </code></pre><p>Although it took a bit more work to store the <code>Activity</code> and <code>TelemetrySpan</code> locally, doing so ensured that we correlated the correct instances for eventual publishing to our tracing adapters. If we only went with the <code>Current</code> properties, we'd be stopping the <em>wrong</em> activities and spans, resulting in a very wonky looking request graph.</p><p>The last piece is registering my subscriber with MongoDB, but it's quite specific to that driver so you can check out the source code to see how it's all registered and hooked up.</p><p>In the next (and last) post, I'll walk through my sample application and see how these adapters will be hooked up and used to visualize complicated traces in Zipkin, Jaeger, and Application Insights.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Q6uj2no8RxQ:1eMwU5s4ykY:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Q6uj2no8RxQ:1eMwU5s4ykY:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Q6uj2no8RxQ:1eMwU5s4ykY:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=Q6uj2no8RxQ:1eMwU5s4ykY:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=Q6uj2no8RxQ:1eMwU5s4ykY:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/Q6uj2no8RxQ" height="1" width="1" alt=""/> Building End-to-End Diagnostics: OpenTelemetry Integration https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/ Jimmy Bogard urn:uuid:e130e56c-0c17-9826-87c9-ef9f28a6f0b5 Thu, 07 May 2020 13:06:39 +0000 <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>In the last post, we looked at providing diagnostic event hooks into our code at specific points so that "something" could listen in. For our purposes, we want to listen in to surface telemetry data, but</p> <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-opentelemetry-integration/">OpenTelemetry Integration</a></li></ul><p><a href="https://github.com/jbogard/nsb-diagnostics-poc">Source Code</a></p><p>In the last post, we looked at providing diagnostic event hooks into our code at specific points so that "something" could listen in. For our purposes, we want to listen in to surface telemetry data, but the uses are far wider for diagnostic events, such as logging, testing, and metrics. We want to focus on telemetry through the <a href="https://opentelemetry.io/\">OpenTelemetry project</a>.</p><p>So why OpenTelemetry? Just another standard we need to follow? As someone that uses NServiceBus on a wide variety of clients, I'm exposed to a number of different observability tools, to name a few:</p><ul><li>Zipkin</li><li>Jaeger</li><li>Dynatrace</li><li>Application Insights</li></ul><p>If you're a library author, like NServiceBus, and you want to enable observability through these tools, you'd have to create and maintain packages for <em>each one</em> of those products. Instead, if you could target some common API, like you can today for:</p><ul><li>Logging</li><li>Dependency Injection</li><li>Configuration</li></ul><p>Then you don't have to have some extensive matrix of support. You target one common API, and the implementation providers can plug in to that common model. OpenTelemetry is just this - providing a standardized API for tracing primitives (and eventually, much more). The .NET SDK for OpenTelemetry (in alpha at the moment, in beta for most other runtimes/platforms) provides that SDK, as well as convenient bridges for the diagnostics event API</p><h3 id="plugging-into-opentelemetry">Plugging into OpenTelemetry</h3><p>The diagnostic events by themselves provide quite a bit of value, but we need to <em>listen</em> to them in order to do something. That's where OpenTelemetry comes in - the "something" listening to diagnostic events. Since <a href="https://github.com/open-telemetry/opentelemetry-dotnet">OpenTelemetry .NET SDK</a> is currently in alpha, I'm going to reference the OpenTelemetry MyGet repository and reference the OpenTelemetry alpha package:</p><pre><code class="language-xml">&lt;Project Sdk="Microsoft.NET.Sdk"&gt; &lt;PropertyGroup&gt; &lt;TargetFramework&gt;netstandard2.0&lt;/TargetFramework&gt; &lt;/PropertyGroup&gt; &lt;ItemGroup&gt; &lt;PackageReference Include="OpenTelemetry" Version="0.2.0-alpha.231" /&gt; &lt;/ItemGroup&gt; &lt;ItemGroup&gt; &lt;ProjectReference Include="..\NServiceBus.Diagnostics\NServiceBus.Diagnostics.csproj" /&gt; &lt;/ItemGroup&gt; &lt;/Project&gt; </code></pre><p>This library I created serves as the listener to diagnostic events, as well as provide an extension to OpenTelemetry to register those listeners. But first, what do we need to create? With the current OpenTelemetry SDK, we need 3 things:</p><ul><li>A <code>ListenerHandler</code> that receives <code>Activity</code>-based diagnostic events</li><li>An <code>Adapter</code> that subscribes a <code>ListenerHandler</code> to diagnostic events</li><li>Extension method to <code>TracerBuilder</code> that registers our <code>Adapter</code> with OpenTelemetry</li></ul><h3 id="listenerhandlers">ListenerHandlers</h3><p>For NServiceBus, we have two kinds of activities - sending and processing messages. That means we'll have two telemetry events, two spans, and two listeners. The OpenTelemetry SDK includes a helper for dealing with the Activity/OpenTelemetry bridge, and that's a base <code><a href="https://github.com/open-telemetry/opentelemetry-dotnet/blob/master/src/OpenTelemetry/Adapter/ListenerHandler.cs">ListenerHandler</a></code> class. Note: this is all subject to change as it's all alpha, but the primitives defined in the <a href="https://github.com/open-telemetry/opentelemetry-specification">OpenTelemetry Specification</a> are at least in beta.</p><p>That base <code>ListenerHandler</code> class has methods to override - when an Activity starts, stops, raises an exception event, or any other custom event. For us, we only have start/stop events, so we can create a class that overrides those two methods:</p><pre><code class="language-csharp">internal class SendMessageListener : ListenerHandler { public SendMessageListener(string sourceName, Tracer tracer) : base(sourceName, tracer) { } public override void OnStartActivity(Activity activity, object payload) { ProcessEvent(activity, payload as BeforeSendMessage); } public override void OnStopActivity(Activity activity, object payload) { ProcessEvent(activity, payload as AfterSendMessage); } </code></pre><p>The <code>payload</code> is the argument fed into <code>StartActivity</code> in our diagnostic listener, allowing our OpenTelemetry listener to have an actual class to work with. The processing of those events needs to create OpenTelemetry spans from the activities, and add attributes to the span.</p><p>Our <code>ProcessEvent</code> method is a bit long, so let's break it up into parts:</p><pre><code class="language-csharp">private void ProcessEvent(Activity activity, BeforeSendMessage payload) { if (payload == null) { AdapterEventSource.Log.NullPayload("SendMessageListener.OnStartActivity"); return; } var span = StartSpanFromActivity(activity, payload); if (span.IsRecording) { SetSpanAttributes(activity, payload, span); } } </code></pre><p>If the payload wasn't recognized as what we expect, then we simply return. Next, we start an OpenTelemetry span from the activity. Finally, if we detect that the span is recording, then we apply the span attributes.</p><p>The <code>StartSpanFromActivity</code> is:</p><pre><code class="language-csharp">private TelemetrySpan StartSpanFromActivity(Activity activity, BeforeSendMessage payload) { payload.Context.Headers.TryGetValue(Headers.MessageIntent, out var intent); var routes = payload.Context.RoutingStrategies .Select(r =&gt; r.Apply(payload.Context.Headers)) .Select(t =&gt; { switch (t) { case UnicastAddressTag u: return u.Destination; case MulticastAddressTag m: return m.MessageType.Name; default: return null; } }) .ToList(); var operationName = $"{intent ?? activity.OperationName} {string.Join(", ", routes)}"; Tracer.StartActiveSpanFromActivity(operationName, activity, SpanKind.Producer, out var span); return span; } </code></pre><p>I wanted to have a more meaningful name here, so I had to do a bit of munging of the incoming data to detect what "kind" of message we're sending here from the intent of the message (Send/Publish/Reply etc.). I also wanted to record where this operation was being sent to, so I also record the routes. The reason this is a little wonky is that NServiceBus is an abstraction over messaging, so I don't have access for example to the inner workings of RabbitMQ or Azure Service Bus or MSMQ.</p><p>The final result however is a <code>Span</code> with a kind of <code>SpanKind.Producer</code> started from the <code>Activity</code> we pass in. The basic guidance of setting the span and its attributes comes from the <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/messaging.md">OpenTelemetry span conventions around messaging</a>.</p><p>With the span started, we now want to apply the span attributes. Following the span guidance, we set several well-known attributes:</p><pre><code class="language-csharp">private static void SetSpanAttributes(Activity activity, BeforeSendMessage payload, TelemetrySpan span) { span.SetAttribute("messaging.message_id", payload.Context.MessageId); span.SetAttribute("messaging.message_payload_size_bytes", payload.Context.Body.Length); span.ApplyContext(payload.Context.Builder.Build&lt;ReadOnlySettings&gt;(), payload.Context.Headers); foreach (var tag in activity.Tags) { span.SetAttribute($"messaging.nservicebus.{tag.Key.ToLowerInvariant()}", tag.Value); } } </code></pre><p>As well as fill in the span with any tags passed in through the Activity. Those extra tags translated into span attributes will allow the diagnostics hooks to add any extra details they want that can finally show up in telemetry events. There's a lot more attributes being set in the <code>ApplyContext</code> method, but it's very specific to NServiceBus internals.</p><p>Finally, to end the span, the <code>ProcessEvent</code> method takes in the <code>AfterSendMessage</code> payload but doesn't really do anything with it - the current <code>Span</code> already tracks the given <code>Activity</code> so we don't have anything additional to add in its data:</p><pre><code class="language-csharp">private void ProcessEvent(Activity activity, AfterSendMessage payload) { Tracer.CurrentSpan.End(); } </code></pre><p>The corresponding <code>ProcessMessageListener</code> is quite similar, and you can find that code over on GitHub. With our listeners in place, we now need the hooks into OpenTelemetry to register our listeners with its configuration.</p><h3 id="registering-our-listeners">Registering Our Listeners</h3><p>The last two steps in OpenTelemetry integration are to provide the bridge from "what diagnostic events should I be listening to" to the <code>ListenerHandler</code> classes we created above. Listening to diagnostic events isn't the most straightforward thing in the world - it uses a global registry and a series of nested observables to subscribe to (more on that in a future post), and you need to be mindful to clean up your subscriptions to avoid memory leaks.</p><p>Luckily, OpenTelemetry .NET SDK has this nailed down for us, we just need to create a disposable class that uses a <code>DiagnosticSourceSubscriber</code> to listen to specific named diagnostic events. Here's the one for the "Send" messages:</p><pre><code class="language-csharp">public class NServiceBusSendAdapter : IDisposable { private readonly DiagnosticSourceSubscriber _diagnosticSourceSubscriber; public NServiceBusSendAdapter(Tracer tracer) { _diagnosticSourceSubscriber = new DiagnosticSourceSubscriber( new SendMessageListener("NServiceBus.Diagnostics.Send", tracer), null); _diagnosticSourceSubscriber.Subscribe(); } public void Dispose() =&gt; _diagnosticSourceSubscriber?.Dispose(); } </code></pre><p>We create an instance of the <code>DiagnosticSourceSubscriber</code>, which is the main helper for bridging diagnostic events and activities, and pass in an instance of our <code>SendMessageListener</code> class we saw earlier. The name of the source, <code>NServiceBus.Diagnostics.Send</code>, is the root name of our diagnostic event.</p><p>When the diagnostic events get raised, they are then suffixed with "Start" and "Stop" in their name, and the base listener class uses that convention to call the <code>OnStart</code> and <code>OnStop</code> methods. If you compare this code with say, the Azure SDK code for listening to diagnostic events, ours is MUCH easier to make sense of.</p><p>Finally, we need to create an extension to OpenTelemetry configuration for our adapters to register them with OpenTelemetry:</p><pre><code class="language-csharp">public static class TraceBuilderExtensions { public static TracerBuilder AddNServiceBusAdapter(this TracerBuilder builder) =&gt; builder .AddAdapter(t =&gt; new NServiceBusReceiveAdapter(t)) .AddAdapter(t =&gt; new NServiceBusSendAdapter(t)); } </code></pre><p>With that in place, we can now register our adapters in our application startup. But before we get to that, what happens when our underlying infrastructure does <em>not</em> play nicely with this model? In the next post, I'll walk through adding telemetry to the <a href="https://mongodb.github.io/mongo-csharp-driver/">Mongo .NET driver</a>, where async context is not preserved and we have to do some more work to correlate diagnostic events.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=eKTXGePAetk:Q2jvcS91acE:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=eKTXGePAetk:Q2jvcS91acE:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=eKTXGePAetk:Q2jvcS91acE:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=eKTXGePAetk:Q2jvcS91acE:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=eKTXGePAetk:Q2jvcS91acE:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/eKTXGePAetk" height="1" width="1" alt=""/> Building End-to-End Diagnostics and Tracing: Diagnostic Events https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/ Jimmy Bogard urn:uuid:32169056-86d9-2ecb-885b-4dd999fc67b8 Mon, 20 Apr 2020 15:28:47 +0000 <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li></ul><p>In the last post, we looked at the <a href="https://www.w3.org/TR/trace-context/">W3C Trace Context</a> standard, and how we can extend NServiceBus to propagate these new headers using middleware. With many monitoring and observability tools, this can be enough for tracing tooling to stitch</p> <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li><li><a href="https://jimmybogard.com/building-end-to-end-tracing-diagnostic-events/">Diagnostic Events</a></li></ul><p>In the last post, we looked at the <a href="https://www.w3.org/TR/trace-context/">W3C Trace Context</a> standard, and how we can extend NServiceBus to propagate these new headers using middleware. With many monitoring and observability tools, this can be enough for tracing tooling to stitch together the entire request timeline.</p><p>These tools rely on agents installed in your environments, either as background services or container sidecars to "listen in" on your traffic and report back to some centralized reporting store that we can then use to view traces. Although this approach can seem a bit intrusive, it does not require any changes to existing code.</p><p>One disadvantage to this approach is that it can be difficult to report more fine grained metrics - things like:</p><ul><li>How long did this operation take?</li><li>What was the nature of this communication?</li><li>What was the cycle time/processing time?</li><li>Did this operation complete successfully? Was it retried? How many times?</li></ul><p>For more complex observability scenarios, merely attaching headers to requests isn't going to suffice.</p><p>Instead, we need to surface diagnostics out of our system into "something". Eventually, that "something" will be <a href="https://opentelemetry.io/">OpenTelemetry</a>, but first, we need to surface diagnostics.</p><h3 id="diagnostics-events">Diagnostics Events</h3><p>With the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.activity?view=netcore-3.1">Activity API</a>, we have a central place where we can store diagnostic information across requests, but the Activity API can do quite a bit more. We want to surface diagnostic information out to OpenTelemetry, but I don't want to tie my middleware <em>directly</em> to one kind of telemetry listener. Even though OpenTelemetry is an emerging standard, I'd like some indirection from my events to OpenTelemetry so that <em>any</em> tool that cares about diagnostic events can listen in to what's going on.</p><p>Similar to Activity, we need a fundamental component to expose diagnostic events. With .NET Core, that component is a <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.diagnosticlistener?view=netcore-3.1">DiagnosticListener</a>. A DiagnosticListener exposes an observable (rather than pure .NET events) so that interested subscribers can listen in to diagnostic events, completely decoupled from our code raising the events.</p><p>Of course, we won't raise events in a vacuum - we actually need to design what these events look like.</p><p>In the .NET Core codebase, one can find many different examples of raising diagnostic events:</p><ul><li><a href="https://github.com/dotnet/aspnetcore/blob/master/src/Hosting/Hosting/src/Internal/HostingApplicationDiagnostics.cs#L314">Incoming ASP.NET Core requests</a></li><li><a href="https://github.com/dotnet/runtime/blob/4f9ae42d861fcb4be2fcd5d3d55d5f227d30e723/src/libraries/System.Net.Http/src/System/Net/Http/DiagnosticsHandler.cs#L78">Outgoing HTTP requests</a></li></ul><p>You can find others in the EF Core codebase, and some Azure SDKs as well. The general idea is we create a listener with a well-known name, and write to the listener at appropriate times (typically, when our activity starts/stops).</p><p>The Activity API and diagnostic listener API <em>roughly</em> correspond to the OpenTelemetry, but there is an <a href="https://github.com/dotnet/runtime/issues/31373">open GitHub issue</a> for .NET 5 to strengthen this bond.</p><p>Raising a diagnostic event is fairly straightforward, we call the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.diagnosticlistener.write?view=netcore-3.1">Write</a> method:</p><pre><code class="language-csharp"> _listener.Write("EventName", &lt;event object&gt;);</code></pre><p>Now we need to figure out two things - what to call our event and what that event object should be!</p><h3 id="diagnostics-events-1">Diagnostics Events</h3><p>In some usages of raising the diagnostic events inside the .NET Core codebase, there wasn't any correlation between the activity name and event name, and worse - that event object was an anonymous type!</p><pre><code class="language-csharp">_listener.Write("My Cool Event", new { SomeValue = someValue, AnotherValue = anotherValue });</code></pre><p>This causes major headaches with consumers of the event who only receive something of type <code>object</code> but have to use reflection instead of casting to get the information out.</p><p>With our events, I don't want to fall into that trap, we want our consumers to have</p><ul><li>Some type that they can cast the <code>object</code> to</li><li>That type should include all available information</li></ul><p>We can find some more guidelines in the <a href="https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md">Diagnostic Source Users Guide</a>, which recommends how we design and use diagnostic listeners and events.</p><p>We also need to decide what interesting points we want to raise the diagnostic events, but that's somewhat straightforward for us - when the activities we create start and stop. We can always add other diagnostic events in the future if we like. With that in mind, we can create two consumer and two producer events, representing the start/stop of the activity:</p><pre><code class="language-csharp">public class BeforeSendMessage { public const string EventName = Constants.ProducerActivityName + ".Start"; public BeforeSendMessage(IOutgoingPhysicalMessageContext context) =&gt; Context = context; public IOutgoingPhysicalMessageContext Context { get; } } public class AfterSendMessage { public const string EventName = Constants.ProducerActivityName + ".Stop"; public AfterSendMessage(IOutgoingPhysicalMessageContext context) =&gt; Context = context; public IOutgoingPhysicalMessageContext Context { get; } } public class BeforeProcessMessage { public const string EventName = Constants.ConsumerActivityName + ".Start"; public BeforeProcessMessage(IIncomingPhysicalMessageContext context) =&gt; Context = context; public IIncomingPhysicalMessageContext Context { get; } } public class AfterProcessMessage { public const string EventName = Constants.ConsumerActivityName + ".Stop"; public AfterProcessMessage(IIncomingPhysicalMessageContext context) =&gt; Context = context; public IIncomingPhysicalMessageContext Context { get; } } </code></pre><p>Rather than trying to parse and pluck individual pieces of information out of the currently executing context, I just include the entire <code>Context</code> of whatever behavior is executing. I don't have any additional information while processing, but if I did, I'd likely leverage the <code>Activity</code> instead of customizing the event message.</p><p>With the events defined, now I can move on to declaring and writing diagnostic events in my middleware.</p><h3 id="producing-diagnostic-events">Producing diagnostic events</h3><p>Following the diagnostic source user guidelines, each of our middleware behaviors needs to declare a <code>DiagnosticSource</code>:</p><pre><code class="language-csharp">public class ConsumerDiagnostics : Behavior&lt;IIncomingPhysicalMessageContext&gt; { private static readonly DiagnosticSource _diagnosticListener = new DiagnosticListener(Constants.ConsumerActivityName); </code></pre><p>I use the same name for my activity as I do for the listener name. Individual diagnostic events written by that <code>DiagnosticSource</code> will then have that name prepended to the event name, so I'll have <code>NServiceBus.Diagnostics.Receive</code> as my activity name and <code>NServiceBus.Diagnostics.Receive.Start</code> and <code>NServiceBus.Diagnostics.Receive.Stop</code> as the diagnostic event names.</p><p>With my listener created, I want to write the start and stop events where I used to start my <code>Activity</code>. However, the usage guidelines recommend only raising a diagnostic event when there are listeners to avoid the overhead. I'll use the <code>StartActivity</code> method as a convenience to start my activity if there is a listener:</p><pre><code class="language-csharp">if (_diagnosticListener.IsEnabled(BeforeProcessMessage.EventName, context)) { _diagnosticListener.StartActivity(activity, new BeforeProcessMessage(context)); } else { activity.Start(); } </code></pre><p>The <code>StartActivity</code> and <code>StopActivity</code> methods on <code>DiagnosticSource</code> will take care of the work of starting/stopping the activity and raising the diagnostic event, named based on the <code>Activity</code> operation name and adding ".Start" or ".Stop" to the end.</p><p>Ending the activity now delegates to the diagnostics listener:</p><pre><code class="language-csharp">private static void StopActivity(Activity activity, IOutgoingPhysicalMessageContext context) { _diagnosticListener.StopActivity(activity, new AfterSendMessage(context)); } </code></pre><p>With this in place, I've now got diagnostic events firing as part of my middleware, and any interested listeners can pick those events up and do whatever they want!</p><p>Right now, I'm only interested in listening to diagnostic events for telemetry purposes, but diagnostic events are also useful for logging, debugging, testing, and anything else where I want to peak "under the hood" of whatever is going on.</p><p>In the next post, we'll look at leveraging the OpenTelemetry SDK (in alpha) to create listeners and collectors for these new events.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=J4-3KtdaTqo:1qR9TYHMm-I:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=J4-3KtdaTqo:1qR9TYHMm-I:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=J4-3KtdaTqo:1qR9TYHMm-I:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=J4-3KtdaTqo:1qR9TYHMm-I:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=J4-3KtdaTqo:1qR9TYHMm-I:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/J4-3KtdaTqo" height="1" width="1" alt=""/> Building End-to-End Diagnostics and Tracing: Trace Context https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/ Jimmy Bogard urn:uuid:e50af674-e895-d6ea-eea3-23d7b9895c26 Tue, 07 Apr 2020 17:11:23 +0000 <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li></ul><p>In the last post, I walked through the overall problem we run into with diagnosing issues in distributed systems - mainly that it can be difficult to determine causality because we don't have that "stack trace" with a single in-process application.</p><p>To</p> <p>Posts in this series:</p><ul><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/">An Intro</a></li><li><a href="https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer-trace-context/">Trace Context</a></li></ul><p>In the last post, I walked through the overall problem we run into with diagnosing issues in distributed systems - mainly that it can be difficult to determine causality because we don't have that "stack trace" with a single in-process application.</p><p>To create a sort of "trace" in a distributed system, we need some way to build breadcrumbs into our communications. When one system communicates with another, and that system calls another, we need some way to link those requests together:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/71246_Picture0072.png" class="kg-image"><figcaption>Distributed systems communicating</figcaption></figure><p>In a downstream system that experiences a failure, how do we link that failure to the original request, and all between? This is where distributed tracing comes in.</p><p>Product companies and OSS filled the void, but there became a problem - each product, OSS or not, had its own way of providing additional context to each request to be able to link them together. The solution to causality is rather simple - we just need some context of the parent system/process that initiated the current request. That context is as simple as providing some unique identifier for the current request to all subsequent requests.</p><p>Very recently (February 2020), a new W3C standard exited "draft" status entered the "recommendation", <a href="https://www.w3.org/TR/trace-context/">Trace Context.</a>  This standard describes mainly:</p><ul><li>What is the ID of the current request?</li><li>What is the ID of the parent request?</li></ul><p>It also allows requests to include some state information, but most important are those identifications. With an ID and parent ID, we can now create a <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">directed acyclic graph</a>, very similar to what we would see in a Git commit history.</p><h3 id="trace-context-in-net-core">Trace Context in .NET Core</h3><p>In order to flow tracing identifiers through a request pipeline, regardless of the technology of the "in" and "out" request, we need some means of capturing the incoming tracing identifiers (on headers), storing them, and flowing them to outgoing headers. The basic pieces for this flow are:</p><ol><li>Incoming requests pull trace identifiers and store in an <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.activity?view=netcore-3.1">Activity</a></code></li><li><code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.activity.current?view=netcore-3.1#System_Diagnostics_Activity_Current">Activity.Current</a></code> includes any additional information for the current activity</li><li>Outgoing requests read information from <code>Activity.Current</code> and place on outgoing trace headers.</li></ol><p>One of the big pushes for observability in .NET Core 3.0 was to <a href="https://devblogs.microsoft.com/aspnet/improvements-in-net-core-3-0-for-troubleshooting-and-monitoring-distributed-apps/">enable this W3C standard</a>. Although it's not turned on by default for backwards compatibility reasons, if you turn it on:</p><pre><code class="language-csharp">public class Program { public static void Main(string[] args) { Activity.DefaultIdFormat = ActivityIdFormat.W3C; CreateHostBuilder(args).Build().Run(); } </code></pre><p>That will use the W3C standards for identifiers, but we still need to consume and propagate these trace identifiers. Luckily, we can see how ASP.NET Core and HttpClient did this:</p><ul><li><a href="https://github.com/dotnet/aspnetcore/blob/master/src/Hosting/Hosting/src/Internal/HostingApplicationDiagnostics.cs#L248">ASP.NET Core consuming incoming Trace Context headers</a></li><li><a href="https://github.com/dotnet/runtime/blob/4f9ae42d861fcb4be2fcd5d3d55d5f227d30e723/src/libraries/System.Net.Http/src/System/Net/Http/DiagnosticsHandler.cs#L254">HttpClient enriching outgoing requests from Activity.Current with Trace Context headers</a></li></ul><p>There's a <em>lot</em> more going on in that code that we'll get to soon, but first things first, we need middleware for NServiceBus to:</p><ul><li>Start an Activity for incoming requests and set its parent ID from the <code>traceparent</code> header</li><li>Set the <code>traceparent</code> header for outgoing requests</li></ul><p>Luckily for us, NServiceBus has a robust middleware API that makes it easy for us to add these pieces, <a href="https://docs.particular.net/nservicebus/pipeline/manipulate-with-behaviors">Behaviors</a>.</p><h3 id="incoming-requests-to-activity">Incoming Requests to Activity</h3><p>The first step in the process for diagnostics is to start an <code>Activity</code> at the beginning of processing a message and stop it at the end. We need to go one step further to add the appropriate parent ID. We can do this with a behavior defined for incoming messages:</p><pre><code class="language-csharp">public class ConsumerDiagnostics : Behavior&lt;IIncomingPhysicalMessageContext&gt; { public override async Task Invoke( IIncomingPhysicalMessageContext context Func&lt;Task&gt; next) { var activity = StartActivity(context); try { await next().ConfigureAwait(false); } finally { StopActivity(activity, context); } } </code></pre><p>Behaviors in NServiceBus are similar to ASP.NET Core middleware. You get two parameters, the first being the context of the operation performed, and the second delegate to perform the next action in the chain.</p><p>The <code>StartActivity</code> method needs to do two things - start an <code>Activity</code>, and set pull the <code>traceparent</code> header off the incoming message:</p><pre><code class="language-csharp">private static Activity StartActivity(IIncomingPhysicalMessageContext context) { var activity = new Activity(Constants.ConsumerActivityName); if (!context.MessageHeaders.TryGetValue( Constants.TraceParentHeaderName, out var requestId)) { context.MessageHeaders.TryGetValue( Constants.RequestIdHeaderName, out requestId); } if (!string.IsNullOrEmpty(requestId)) { // This is the magic activity.SetParentId(requestId); if (context.MessageHeaders.TryGetValue( Constants.TraceStateHeaderName, out var traceState)) { activity.TraceStateString = traceState; } } // The current activity gets an ID with the W3C format activity.Start(); return activity; } </code></pre><p>We first create an activity with a good name, in my case I chose <code>NServiceBus.Diagnostics.Receive</code>. There's not a <em>ton</em> of recommendations about naming activities, but it should be something meaningful to the overall operation that's being performed. Activity names are hierarchical for future purposes, so we want to adhere to some sort of namespacing. The ASP.NET Core name is <code>Microsoft.AspNetCore.Hosting.HttpRequestIn</code> and HttpClient is <code>System.Net.Http.HttpRequestOut</code>.</p><p>After creating the <code>Activity</code>, I try to pull the <code>traceparent</code> header value out. I'm also trying to be a good citizen and pull the older, previous <code>request-id</code> header value out. Once i have this, I can set the <code>ParentId</code> on the <code>Activity</code>. Finally, if it exists, I'll pull the <code>tracestate</code> value and stuff it into the <code>Activity</code>. There's some more things in store for distributed tracing related to additional correlation context items, but for now, I'll leave that alone.</p><p>Finally, I start the activity, and <code>Activity.Current</code> represents this new activity. Stopping the activity is straightforward - the only thing I really need to care about is setting an end time of the Activity:</p><pre><code class="language-csharp">private static void StopActivity(Activity activity, IIncomingPhysicalMessageContext context) { if (activity.Duration == TimeSpan.Zero) { activity.SetEndTime(DateTime.UtcNow); } activity.Stop(); } </code></pre><p>Setting an appropriate end time for the activity will mean more later on when we start raising diagnostic events, but we want to make sure the duration of the event is just around calling the <code>next</code> item in the pipeline.</p><p>That's incoming requests, what about outgoing ones?</p><h3 id="propagating-trace-context-in-outgoing-messages">Propagating trace context in outgoing messages</h3><p>Just like we have incoming behaviors for messages, NServiceBus has outgoing behaviors as well. We just need to reverse the flow from above - set the <code>traceparent</code> header on outgoing messages from the current <code>Actvity</code>:</p><pre><code class="language-csharp">public class ProducerDiagnostics : Behavior&lt;IOutgoingPhysicalMessageContext&gt; { public override async Task Invoke( IOutgoingPhysicalMessageContext context, Func&lt;Task&gt; next) { var activity = StartActivity(context); InjectHeaders(activity, context); try { await next().ConfigureAwait(false); } finally { StopActivity(activity, context); } } </code></pre><p>Starting the activity is much simpler now:</p><pre><code class="language-csharp">private static Activity StartActivity(IOutgoingPhysicalMessageContext context) { var activity = new Activity(Constants.ProducerActivityName); activity.Start(); return activity; } </code></pre><p>But wait, we're not setting the parent ID! For outgoing messages, we don't need to. If there's a current started activity, our activity will automatically have its <code>ParentId</code> set to <code>Activity.Current.Id</code>, so we're good to go without managing all that ourselves.</p><p>Next, we need to inject the headers of the current activity into the outgoing request:</p><pre><code class="language-csharp">private static void InjectHeaders( Activity activity, IOutgoingPhysicalMessageContext context) { if (activity.IdFormat == ActivityIdFormat.W3C) { if (!context.Headers.ContainsKey(Constants.TraceParentHeaderName)) { context.Headers[Constants.TraceParentHeaderName] = activity.Id; if (activity.TraceStateString != null) { context.Headers[Constants.TraceStateHeaderName] = activity.TraceStateString; } } } else { if (!context.Headers.ContainsKey(Constants.RequestIdHeaderName)) { context.Headers[Constants.RequestIdHeaderName] = activity.Id; } } } </code></pre><p>The new request's parent ID will be <em>this</em> activity's ID, and that new parent ID will be consumed by downstream systems as well.</p><p>The magic here is <code>Activity.Current</code>, an async local static property that means that anything sharing the same async context will get the same <code>Activity.Current</code> value.</p><p>Stopping the activity looks exactly the same as the incoming requests:</p><pre><code class="language-csharp">private static void StopActivity( Activity activity, IOutgoingPhysicalMessageContext context) { if (activity.Duration == TimeSpan.Zero) { activity.SetEndTime(DateTime.UtcNow); } activity.Stop(); } </code></pre><p>To enable these behaviors, you can use NServiceBus <a href="https://docs.particular.net/nservicebus/pipeline/features">Features</a> to add these behaviors to the processing pipeline automatically:</p><pre><code class="language-csharp">public class DiagnosticsFeature : Feature { public DiagnosticsFeature() { EnableByDefault(); } protected override void Setup(FeatureConfigurationContext context) { context.Pipeline.Register(new ConsumerDiagnostics(), "Parses incoming W3C trace information from incoming messages."); context.Pipeline.Register(new ProducerDiagnostics(), "Appends W3C trace information to outgoing messages."); } } </code></pre><p>I enable this feature by default, with the future idea that anyone that references this package/assembly will get this behavior opted in. With all of this in place, how does this look in practice?</p><h3 id="a-dummy-distributed-system">A Dummy Distributed System</h3><p>I wanted to simulate all these different kinds of flows, which use a variety of hosts and communication:</p><ul><li>Incoming HTTP to Outgoing NServiceBus</li><li>Incoming NServiceBus to Outgoing HTTP</li><li>Incoming NServiceBus to Outgoing NServiceBus</li></ul><p>Incoming HTTP will be a regular ASP.NET Core application and host, and the incoming NServiceBus will be a worker service. I wanted to capture all manners of communication with these two applications:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/71626_Dummy%20Distributed%20System.png" class="kg-image"><figcaption>Dummy distributed system communicating via HTTP and AMQP</figcaption></figure><p>My Web Server is a web application with this diagnostics code added, plus using the <a href="https://docs.particular.net/nservicebus/hosting/extensions-hosting">NServiceBus extension to .NET Core generic hosting</a>. I created a simple API controller that uses the NServiceBus <code>IMessageSession</code> injected to send an AMQP message via RabbitMQ:</p><pre><code class="language-csharp">[HttpGet] public async Task&lt;ActionResult&gt; Get(string message) { var command = new SaySomething { Message = message }; _logger.LogInformation("Sending message {message}", command.Message); await _messageSession.Send(command); return Accepted(); }</code></pre><p>The handler of this message on the worker service makes the HTTP call and a Reply:</p><pre><code class="language-csharp">public class SaySomethingHandler : IHandleMessages&lt;SaySomething&gt; { private readonly ILogger&lt;SaySomethingHandler&gt; _logger; private static readonly HttpClient _httpClient = new HttpClient { BaseAddress = new Uri("https://localhost:5001") }; public SaySomethingHandler(ILogger&lt;SaySomethingHandler&gt; logger) =&gt; _logger = logger; public async Task Handle(SaySomething message, IMessageHandlerContext context) { var content = await _httpClient.GetStringAsync("/weatherforecast/today"); dynamic json = Deserialize&lt;ExpandoObject&gt;(content); var temp = (int)json.temperatureF.GetInt32(); _logger.LogInformation("Saying {message} and the weather today is {weather}F", message.Message, temp); await context.Reply(new SaySomethingResponse { Message = $"Back at ya {message.Message}" }); } }</code></pre><p>The API endpoint is rather dumb, it's the weather dummy data that I stuck in a database:</p><pre><code class="language-csharp">[HttpGet("today")] public async Task&lt;WeatherForecast&gt; GetToday() { var forecastCount = await _dbContext.Forecasts.CountAsync(); var rng = new Random(); return await _dbContext.Forecasts.Skip(rng.Next(forecastCount)).FirstAsync(); }</code></pre><p>And the Reply handler doesn't do anything fun, but it stops the distributed flow:</p><pre><code class="language-csharp">public class SaySomethingResponseHandler : IHandleMessages&lt;SaySomethingResponse&gt; { private readonly ILogger&lt;SaySomethingResponseHandler&gt; _logger; public SaySomethingResponseHandler(ILogger&lt;SaySomethingResponseHandler&gt; logger) =&gt; _logger = logger; public Task Handle(SaySomethingResponse message, IMessageHandlerContext context) { _logger.LogInformation("Received {message}", message.Message); return Task.CompletedTask; } } </code></pre><p>With all the pieces in place, let's trace the flow from the initial request all the way through each receiver and out again.</p><h3 id="tracing-the-flow">Tracing the flow</h3><p>It all starts with initiating the request with the Swagger UI:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/71636_Picture0073.png" class="kg-image"><figcaption>Initial request in Swagger UI</figcaption></figure><p>Logging the <code>Activity.Current.Id</code> and <code>Activity.Current.ParentId</code>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/71652_Picture0074.png" class="kg-image"><figcaption>Initial request activity ID and parent ID</figcaption></figure><p>We see that the current activity ID has a value, but the parent ID does not. This makes sense - the Swagger UI doesn't track activities and does not pass a <code>traceparent</code> header along.</p><p>With the message sent, let's look at the message in RabbitMQ to see if it has a <code>traceparent</code> value that matches the above:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/71656_Picture0075.png" class="kg-image"><figcaption>RabbitMQ message with matching traceparent header value</figcaption></figure><p>It does! Let's now run the whole system end-to-end and watch the activity IDs in our logs:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/7173_Picture0076.png" class="kg-image"><figcaption>Aggregate logs for both web and worker service with activity IDs</figcaption></figure><p>We can see that all of our activity IDs share the same <code><a href="https://www.w3.org/TR/trace-context/#trace-id">trace-id</a></code> fragment, while the <code>parent-id</code> values differ (technical detail, but these link to a span in tracing terms).</p><p>With our tracing identifiers correctly propagating, we've laid the groundwork to start putting humpty dumpty together again. In the next post, we'll look at how we can raise diagnostic events so that something can see these traces outside of directly instrumenting our traffic.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=LacO9TqKoTg:YqwOjwNmym0:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=LacO9TqKoTg:YqwOjwNmym0:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=LacO9TqKoTg:YqwOjwNmym0:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=LacO9TqKoTg:YqwOjwNmym0:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=LacO9TqKoTg:YqwOjwNmym0:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/LacO9TqKoTg" height="1" width="1" alt=""/> Building End-to-End Diagnostics and Tracing: An Intro https://jimmybogard.com/building-end-to-end-diagnostics-and-tracing-a-primer/ Jimmy Bogard urn:uuid:5f3a22e2-fa0d-73d8-14e5-50e3c1e10b73 Wed, 01 Apr 2020 15:23:15 +0000 <p>As microservices introduced (forced) highly complex distributed systems into organizations, the tools required to operate these architectures needed to evolve as well. What was a simple stack trace in a single in-process monolith became a web of network calls.</p><p>In my first large-scale distributed system, well before the term "microservice"</p> <p>As microservices introduced (forced) highly complex distributed systems into organizations, the tools required to operate these architectures needed to evolve as well. What was a simple stack trace in a single in-process monolith became a web of network calls.</p><p>In my first large-scale distributed system, well before the term "microservice" was coined, we hit the problem of operating complex distributed systems almost the second the entire system was turned on. When something went wrong, it could take hours to track down the issue. My old talk on <a href="https://www.youtube.com/watch?v=gfh-VCTwMw8">Avoiding Microservices Megadisasters</a> goes into one of these stories - that it took 2 weeks just to figure out how a request was stitched together.</p><p>Since then, diagnostics and tracing have come a long way. In this series, I want to walk through adding diagnostics and tracing to a library I've used quite a lot over the years - NServiceBus. Based on those techniques, we can add diagnostics and tracing to any network-communicating library or component.</p><h3 id="the-overall-problem">The Overall Problem</h3><p>In a single in-process application, when something goes wrong, you have a stack trace telling you exactly where in the system an exception occurred. But if you've got distributed systems communication with each other, it's not enough to have a stack trace of a single application. Often, we need to understand causality all the way back out to the original external trigger or event that led to a fault.</p><p>The solution to this problem is "<a href="https://microservices.io/patterns/observability/distributed-tracing.html">distributed tracing</a>". Instead of having a single call stack, we connect multiple call stacks together by introducing some additional tracing metadata between each node.</p><p>Over the years, many tools and products arose to fill this niche. I've used a few, and built a few, but with each new tool rose a new means to plug it in.</p><p>If I wanted to use Dynatrace, I needed to have Dynatrace plugins to everything I used. If I wanted to use Zipkin, the same. And if those plugins didn't exist for whatever library I was using, I needed to build that myself. Each tool had its own way of providing its tracing context. Zipkin has its own, and NServiceBus has its own, and some don't have anything.</p><p>This is where standards come in - to provide a common way of:</p><ul><li>Identifying and propagating tracing information</li><li>Raising diagnostic event notifications</li><li>Reporting diagnostic telemetry information</li></ul><p>NServiceBus has a very robust distributed tracing mechanism and reporting tool with <a href="https://particular.net/serviceinsight">ServiceInsight</a>, however, similar to Zipkin/Jaeger/Prometheus etc., it's using proprietary means of doing so and doesn't directly plug in to any other reporting tool or network communication.</p><h3 id="the-plan">The Plan</h3><p>In order to make any new network component "play nice" with distributed tracing, a few things need to happen:</p><ul><li>All incoming network traffic needs to capture tracing information</li><li>All outgoing network traffic needs to propagate tracing information</li><li>Any interesting diagnostic event needs to be emitted</li><li>Diagnostic events raised need to be captured and re-emitted as telemetry</li></ul><p>In this series, I'll walk through each of these steps, the standards applied, and middleware needed to connect all the pieces together. In the end, we'll have a complete picture of a distributed system that uses ASP.NET Core, HttpClient, and RabbitMQ, and SQL together in a single picture:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://jimmybogardsblog.blob.core.windows.net/jimmybogardsblog/images/2020/4/11519_image.png" class="kg-image"><figcaption>Distributed trace connecting ASP.NET Core, HttpClient, RabbitMQ, and SQL</figcaption></figure><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=lRV6NTp-6o4:cxzBiPbDu0s:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=lRV6NTp-6o4:cxzBiPbDu0s:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=lRV6NTp-6o4:cxzBiPbDu0s:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=lRV6NTp-6o4:cxzBiPbDu0s:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=lRV6NTp-6o4:cxzBiPbDu0s:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/lRV6NTp-6o4" height="1" width="1" alt=""/> NServiceBus and .NET Core Generic Host https://jimmybogard.com/nservicebus-and-net-core-generic-host/ Jimmy Bogard urn:uuid:347f9fe1-9126-5341-47f9-63aa1ecea54e Mon, 23 Mar 2020 13:19:53 +0000 <p>My current client is using .NET Core 2.x, with plans to upgrade to 3.x next month. As part of that system, we do quite a bit of messaging, with NServiceBus as the tool of choice to help make this easier. To get it working with our .NET Core</p> <p>My current client is using .NET Core 2.x, with plans to upgrade to 3.x next month. As part of that system, we do quite a bit of messaging, with NServiceBus as the tool of choice to help make this easier. To get it working with our .NET Core 2.x applications, we did quite a bit of what I laid out in my <a href="https://jimmybogard.com/building-messaging-endpoints-in-azure-a-generic-host/">Messaging Endpoints in Azure</a> series.</p><p>Since then, NServiceBus released first-class support for the .NET Core Generic Host, which underwent a fairly large refactoring in the 2.x to 3.0 timeframe. <a href="https://andrewlock.net/ihostingenvironment-vs-ihost-environment-obsolete-types-in-net-core-3/">Andrew Lock's post</a> goes into more detail, but the gist of it is, NServiceBus has first-class support for .NET Core 3.x and later.</p><p>What that means for us is hosting NServiceBus inside a .NET Core application couldn't be easier. The <a href="https://docs.particular.net/nservicebus/hosting/extensions-hosting">NServiceBus.Extensions.Hosting</a> package provides all the integration we need to add a hosted NServiceBus service <em>and</em> integrate with the built-in DI container.</p><h3 id="configuring-nservicebus">Configuring NServiceBus</h3><p>With any kind of hosted .NET Core application (Console, ASP.NET Core, Worker), we just need to add the extensions package:</p><pre><code class="language-xml">&lt;Project Sdk="Microsoft.NET.Sdk.Web"&gt; &lt;PropertyGroup&gt; &lt;TargetFramework&gt;netcoreapp3.1&lt;/TargetFramework&gt; &lt;/PropertyGroup&gt; &lt;ItemGroup&gt; &lt;PackageReference Include="NServiceBus.Extensions.Hosting" Version="1.0.0" /&gt; </code></pre><p>And add the configuration directly off of the host builder:</p><pre><code class="language-csharp">Host.CreateDefaultBuilder(args) .UseNServiceBus(hostBuilderContext =&gt; { var endpointConfiguration = new EndpointConfiguration("WebApplication"); // configure endpoint here return endpointConfiguration; }) .ConfigureWebHostDefaults(webBuilder =&gt; { webBuilder.UseStartup&lt;Startup&gt;(); }); </code></pre><p>Or with a <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&amp;tabs=visual-studio">Worker SDK</a>:</p><pre><code class="language-xml">&lt;Project Sdk="Microsoft.NET.Sdk.Worker"&gt; &lt;PropertyGroup&gt; &lt;TargetFramework&gt;netcoreapp3.1&lt;/TargetFramework&gt; &lt;/PropertyGroup&gt; &lt;ItemGroup&gt; &lt;PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.2" /&gt; </code></pre><p>It's really not much different:</p><pre><code class="language-csharp">public static IHostBuilder CreateHostBuilder(string[] args) =&gt; Host.CreateDefaultBuilder(args) .UseNServiceBus(hostBuilderContext =&gt; { var endpointConfiguration = new EndpointConfiguration("WorkerService"); // configure endpoint here return endpointConfiguration; }); </code></pre><p>And our endpoint is up and running.</p><h3 id="logging-and-serialization">Logging and Serialization</h3><p>We're not quite there yet, though. The out-of-the-box serialization is XML (which is fine by me), but many folks prefer JSON. Additionally, the logging support inside of NServiceBus is <em>not</em> currently integrated with this package. For serialization, we can use the new System.Text.Json support instead of Newtonsoft.Json.</p><p>We'll pull in the community packages from <a href="https://github.com/SimonCropp">Simon Cropp</a>:</p><ul><li><a href="https://github.com/NServiceBusExtensions/NServiceBus.Json">NServiceBus.Json</a></li><li><a href="https://github.com/NServiceBusExtensions/NServiceBus.MicrosoftLogging">NServiceBus.MicrosoftLogging.Hosting</a></li></ul><p>With those two packages in place, we can configure our host's serializer and logging:</p><pre><code class="language-csharp">Host.CreateDefaultBuilder(args) .UseMicrosoftLogFactoryLogging() .UseNServiceBus(hostBuilderContext =&gt; { var endpointConfiguration = new EndpointConfiguration("WorkerService"); endpointConfiguration.UseSerialization&lt;SystemJsonSerializer&gt;(); </code></pre><p>We now have integrated logging, hosting, and dependency injection with anything that uses the generic host.</p><h3 id="using-the-logger">Using the logger</h3><p>Now in our handlers, we can add dependencies directly on the Microsoft logger,  <code>ILogger&lt;T&gt;</code>:</p><pre><code class="language-csharp">public class SaySomethingHandler : IHandleMessages&lt;SaySomething&gt; { private readonly ILogger&lt;SaySomethingHandler&gt; _logger; public SaySomethingHandler(ILogger&lt;SaySomethingHandler&gt; logger) =&gt; _logger = logger; public Task Handle(SaySomething message, IMessageHandlerContext context) { _logger.LogInformation("Saying {message}", message.Message); return context.Reply(new SaySomethingResponse { Message = $"Back at ya {message.Message}" }); } }</code></pre><p>And we get an integrated logging experience:</p><pre><code class="language-text">info: NServiceBus.LicenseManager[0] Selected active license from C:\Users\jbogard\AppData\Local\ParticularSoftware\license.xml License Expiration: 2020-06-16 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development info: Microsoft.Hosting.Lifetime[0] Content root path: C:\Users\jbogard\source\repos\NsbActivities\WorkerService info: WorkerService.SaySomethingHandler[0] Saying Hello World! </code></pre><p>Now with this logging and dependency injection integration, we can use <em>any</em> logger or container that extends the built-in abstractions. My current client (and most) use Serilog, which makes it very easy to plug in to the generic host as well.</p><p>With these packages, we'll be able to <em>delete</em> a lot of infrastructure code that wasn't adding any value, which is always a good thing.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=sqwnUZbrx-c:IOu8kmb1FrI:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=sqwnUZbrx-c:IOu8kmb1FrI:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=sqwnUZbrx-c:IOu8kmb1FrI:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=sqwnUZbrx-c:IOu8kmb1FrI:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=sqwnUZbrx-c:IOu8kmb1FrI:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/sqwnUZbrx-c" height="1" width="1" alt=""/> Avoid In-Memory Databases for Tests https://jimmybogard.com/avoid-in-memory-databases-for-tests/ Jimmy Bogard urn:uuid:3462f902-ee8d-042f-096f-ba77a99f4f8f Wed, 18 Mar 2020 18:25:12 +0000 <p>A <a href="https://github.com/dotnet/efcore/issues/18457">controversial</a> GitHub issue came to my attention a couple of weeks ago around ditching the <a href="https://docs.microsoft.com/en-us/ef/core/providers/in-memory/?tabs=dotnet-core-cli">in-memory provider</a> for Entity Framework Core. This seemed like a no-brainer to me - these database providers are far from trivial to maintain, even for in-memory strategies. It's something our teams learned nearly a</p> <p>A <a href="https://github.com/dotnet/efcore/issues/18457">controversial</a> GitHub issue came to my attention a couple of weeks ago around ditching the <a href="https://docs.microsoft.com/en-us/ef/core/providers/in-memory/?tabs=dotnet-core-cli">in-memory provider</a> for Entity Framework Core. This seemed like a no-brainer to me - these database providers are far from trivial to maintain, even for in-memory strategies. It's something our teams learned nearly a decade ago, that trying to swap out an in-memory strategy for unit testing simply doesn't provide the value folks may hope for.</p><p>It seems rather simple at first - especially in the .NET world and EF Core. EF Core's primary read API is <a href="https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/">LINQ</a>. LINQ has two flavors - <code>IEnumerable</code> and <code>IQueryable</code>. With <code>IQueryable</code>, an <code><a href="https://docs.microsoft.com/en-us/dotnet/api/system.linq.iqueryprovider?view=netcore-3.1">IQueryProvider</a></code> translates expression trees to...well whatever makes sense for the underlying store. There's a neat trick that you can do, however, as <code>IEnumerable</code> has a method, <code>AsQueryable</code>, to allow complex expression trees to evaluate directly against an in-memory <code>IEnumerable</code>.</p><p>Thus, in-memory queryables were born. So why not take advantage of this possibility for unit tests? Why not allow us to swap the implementation of some queryable to the in-memory equivalent, and allow us to write unit tests against in-memory stores?</p><p>It all seems so simple, but unfortunately, the devil is in the details.</p><h3 id="simple-ain-t-easy">Simple Ain't Easy</h3><p>LINQ providers aren't easy. They're not even merely difficult, they're some of the <strong>most complex pieces of code you'll see</strong>. Why is that?</p><p>A LINQ provider is a compiler, a lexer, and a parser, but doesn't own any of those pieces. It's a transpiler, but instead of the output being text, it's API calls. You have to do similar operations as an actual compiler, dealing with ASTs, building your own AST (often), and figuring out how to make a very wide and complex API that is all the <code>IQueryable</code> surface area.</p><p>Any ORM maintainer can tell you - the code in the query provider can <em>dwarf</em> the code in the rest of the codebase. This is unlike other ORMs that provide a specialized API or language, such as NHibernate or the MongoDB C# driver.</p><p>LINQ surfaces a great amount of flexibility, but that flexibility makes it quite difficult to translate into SQL or any other database API that's already been specifically designed for <em>that database</em>. We're trying to wrap a fit-for-purpose API with a generic-purpose API, and one that can run either in-memory or translated into a database.</p><p>On top of that, a good deal of the LINQ surface area can't be translated into SQL, and there are Very Important things that don't translate into LINQ. So you have to extend <code>IQueryable</code> to do fun things like:</p><pre><code class="language-csharp">using (var context = new BloggingContext()) { var blogs = context.Blogs .Include(blog =&gt; blog.Posts) .ThenInclude(post =&gt; post.Author) .ThenInclude(author =&gt; author.Photo) .ToList(); }</code></pre><p>Yikes! We also have <code>async</code> in the mix, so now we're at the point where the <code>IQueryable</code> isn't remotely the same for in-memory.</p><p>But that won't stop us from trying!</p><h3 id="pitfalls-of-in-memory">Pitfalls of In-Memory</h3><p>Our teams tried a number of years ago to speed up integration tests by swapping in-memory providers, but we found a <em>number</em> of problems with this approach that led us to abandoning it altogether.</p><h4 id="you-must-write-a-real-integration-test-anyway">You MUST write a real integration test anyway</h4><p>First and foremost, an in-memory provider is a pale imitation for the real thing. Even with writing in-memory tests, we still absolutely wrote integration tests against a real database. Those unit tests with in-memory looked exactly like the integration tests, just with a different provider.</p><p>Which led us to wonder - if we were writing the tests twice, what was the value of having two tests?</p><p>You <em>could</em> write a single test codebase, and run it twice - one with in-memory and one with the real thing, but that has other problems.</p><h4 id="you-must-allow-raw-access-to-the-underlying-data-api">You MUST allow raw access to the underlying data API</h4><p>ORMs allow you to encapsulate your data access, which is good, it allows us to be more productive by focusing on the business problem at hand. But it's also bad because it abstracts your data access, leaving developers to assume that they don't actually need to understand what is going on behind the scenes.</p><p>In our projects, we take a pragmatic approach - use the ORM's API when it works, and drop down to the database API when it becomes painful. ORMs these days make it quite easy, such as <a href="https://docs.microsoft.com/en-us/ef/core/querying/raw-sql">EF Core's raw SQL</a> capabilities:</p><pre><code class="language-csharp">var blogs = context.Blogs .FromSqlRaw("SELECT * FROM dbo.Blogs") .ToList();</code></pre><p>There are numerous limitations, many more than with EF6, which is why we often bring in a tool like Dapper to do complex SQL:</p><pre><code class="language-csharp">var employeeHierarchy = connection.Execute&lt;EmployeeDto&gt;(@"WITH cte_org AS ( SELECT staff_id, first_name, manager_id FROM sales.staffs WHERE manager_id IS NULL UNION ALL SELECT e.staff_id, e.first_name, e.manager_id FROM sales.staffs e INNER JOIN cte_org o ON o.staff_id = e.manager_id ) SELECT * FROM cte_org;");</code></pre><p>So how do we handle this scenario in our tests? Don't write the unit test (this assumes we're actually writing tests twice)? Somehow exclude it?</p><p>What I tend to find is that instead of dropping down to SQL, developers <em>avoid</em> SQL just so that it satisfies the tool. This is unacceptable.</p><h4 id="the-apis-don-t-match">The APIs don't match</h4><p>The in-memory API of vanilla <code>IQueryProvider</code> doesn't match the LINQ query provider. This means you'll have methods that don't make sense, are no-ops, or even nonsensical for in-memory.</p><p>The most obvious example is <code>Include</code>, which instructs the LINQ provider to basically do a join to eagerly fetch some child records. This is to avoid multiple round trips. However, this means nothing to in-memory. You can keep it, remove it, add more, remove more, doesn't matter.</p><p>It gets worse on the flip side - when LINQ provides some APIs that aren't supported by the query provider. Since LINQ can run in-memory, it can execute <em>anything</em> on the client side. But when you try to run <em>anything</em> on the server, that won't work:</p><pre><code class="language-csharp">var blogs = context.Blogs .Where(blog =&gt; StandardizeUrl(blog.Url).Contains("dotnet")) .ToList();</code></pre><p>Instead, LINQ providers allow a narrow subset of methods, and even beyond that, a limited set of core .NET methods to translate on the server. But not all obvious methods, and not even all overloads are supported. You don't know this until you actually run the LINQ query against the enormous LINQ provider.</p><h4 id="databases-means-transactions">Databases Means Transactions</h4><p>If I look at a typical integration test we write, we're using both the public and non-public API in a series of transactions to interact with the system under test. Here's a typical example:</p><pre><code class="language-csharp">[Fact] public async Task Should_edit_department() { var adminId = await SendAsync(new CreateEdit.Command { FirstMidName = "George", LastName = "Costanza", HireDate = DateTime.Today }); var admin2Id = await SendAsync(new CreateEdit.Command { FirstMidName = "George", LastName = "Costanza", HireDate = DateTime.Today }); var dept = new Department { Name = "History", InstructorId = adminId, Budget = 123m, StartDate = DateTime.Today }; await InsertAsync(dept); Edit.Command command = null; await ExecuteDbContextAsync(async (ctxt, mediator) =&gt; { var admin2 = await FindAsync&lt;Instructor&gt;(admin2Id); command = new Edit.Command { Id = dept.Id, Name = "English", Administrator = admin2, StartDate = DateTime.Today.AddDays(-1), Budget = 456m }; await mediator.Send(command); }); var result = await ExecuteDbContextAsync(db =&gt; db.Departments.Where(d =&gt; d.Id == dept.Id).Include(d =&gt; d.Administrator).SingleOrDefaultAsync()); result.Name.ShouldBe(command.Name); result.Administrator.Id.ShouldBe(command.Administrator.Id); result.StartDate.ShouldBe(command.StartDate.GetValueOrDefault()); result.Budget.ShouldBe(command.Budget.GetValueOrDefault()); }</code></pre><p>It's long, but it combines both public APIs (sending commands to create items) and non-public APIs (interacting directly with the <code>DbContext</code> to insert rows), executing an individual command for the test, then finally querying to pull an item out.</p><p>In integration tests of long ago, we'd put this entire set of operations in a transaction/unit of work. That's not at all how the application behaves, however, and we'd see many false positives that would only break when each operation was distinct. This is because ORMs use patterns like Unit of Work and Identity Map to determine what to persist and when.</p><p>With in-memory providers, there is no "ACID", everything is immediately durable. Each operation is immediately performed, and <a href="https://github.com/dotnet/efcore/blob/master/src/EFCore.InMemory/Storage/Internal/InMemoryTransaction.cs">transactions do nothing</a>! It might seem like a trivial thing, who cares if everything is always immediately durable? The problem is, just like an integration test that uses a single transaction, is that real-life behavior is much different and more complex, and will break in ways you can't predict. Enough false positives, and you wind up distrusting these unit tests.</p><p>The database enforces constraints and visibility and isolation levels that these attempts can't, and inevitably, you'll hit problems</p><h3 id="but-it-s-working-for-me-">But it's working for me!</h3><p>Great! You're one of the lucky few. Your usage is trivial enough that can fit into the constraints of an in-memory provider. We've tried this (and other in-memory DBs, like SQLite), and it's always failed.</p><p>Unfortunately for the EF team, maintaining this provider <em>for public consumption</em> is a cost for them, and a tradeoff. They're coding that instead of coding something else. The question becomes - is the value of a (always flawed) in-memory provider worth the effort for the team?</p><p>For me, no, it's not worth negative effects for our team.</p><div class="feedflare"> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=EZE_YP3T9Ac:w-EBMYGJ0m8:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?d=yIl2AUoC8zA" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=EZE_YP3T9Ac:w-EBMYGJ0m8:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=EZE_YP3T9Ac:w-EBMYGJ0m8:V_sGLiPBpWU" border="0"></img></a> <a href="http://feeds.feedburner.com/~ff/GrabBagOfT?a=EZE_YP3T9Ac:w-EBMYGJ0m8:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/GrabBagOfT?i=EZE_YP3T9Ac:w-EBMYGJ0m8:gIN9vFwOqvQ" border="0"></img></a> </div><img src="http://feeds.feedburner.com/~r/GrabBagOfT/~4/EZE_YP3T9Ac" height="1" width="1" alt=""/> Getting started with Cassandra: Data modeling in the brief https://lostechies.com/ryansvihla/2020/02/05/getting-started-cassandra-part-3/ Los Techies urn:uuid:faeba5a6-db95-bc14-4f6f-333e146885f1 Wed, 05 Feb 2020 20:23:00 +0000 Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point. <p>Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.</p> <h2 id="introduction">Introduction</h2> <p>Cassandra distributes data around the cluster via the <em>partition</em> <em>key</em>.</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="p">(</span><span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">balance</span> <span class="nb">float</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">postal_code</span><span class="p">,</span> <span class="n">id</span><span class="p">));</span> </code></pre></div></div> <p>In the above table the <em>partition</em> <em>key</em> is <code class="language-plaintext highlighter-rouge">postal_code</code> and the <em>clustering</em> <em>column</em> is<code class="language-plaintext highlighter-rouge">id</code>. The <em>partition</em> <em>key</em> will locate the data on the cluster for us. The clustering column allows us multiple rows per <em>partition</em> <em>key</em> so that we can filter how much data we read per partition. The ‘optimal’ query is one that retrieves data from only one node and not so much data that GC pressure or latency issues result. The following query is breaking that rule and retrieving 2 partitions at once via the IN parameter.</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="k">WHERE</span> <span class="n">postal_code</span> <span class="k">IN</span> <span class="p">(</span><span class="s1">'77002'</span><span class="p">,</span> <span class="s1">'77043'</span><span class="p">);</span> </code></pre></div></div> <p>This <em>can</em> <em>be</em> slower than doing two separate queries asynchronously, especially if those partitions are on two different nodes (imagine if there are 1000+ partitions in the IN statement). In summary, the simple rule to stick to is “1 partition per query”.</p> <h3 id="partition-sizes">Partition sizes</h3> <p>A common mistake when data modeling is to jam as much data as possible into a single partition.</p> <ul> <li>This doesn’t distribute the data well and therefore misses the point of a distributed database.</li> <li>There are practical limits on the <a href="https://issues.apache.org/jira/browse/CASSANDRA-9754">performance of partition sizes</a></li> </ul> <h3 id="table-per-query-pattern">Table per query pattern</h3> <p>A common approach to optimize around partition lookup is to create a table per query, and write to all of them on update. The following example has two related tables both to solve two different queries</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">--query by postal_code</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="p">(</span><span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">balance</span> <span class="nb">float</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">postal_code</span><span class="p">,</span> <span class="n">id</span><span class="p">));</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="k">WHERE</span> <span class="n">postal_code</span> <span class="o">=</span> <span class="s1">'77002'</span><span class="p">;</span> <span class="c1">--query by id</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="p">(</span><span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">name</span> <span class="nb">text</span><span class="p">,</span> <span class="n">address</span> <span class="nb">text</span><span class="p">,</span> <span class="n">city</span> <span class="nb">text</span><span class="p">,</span> <span class="k">state</span> <span class="nb">text</span><span class="p">,</span> <span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">country</span> <span class="nb">text</span><span class="p">,</span> <span class="n">balance</span> <span class="nb">float</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">id</span><span class="p">));</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">7895</span><span class="n">c6ff</span><span class="o">-</span><span class="mi">008</span><span class="n">b</span><span class="o">-</span><span class="mi">4</span><span class="n">e4c</span><span class="o">-</span><span class="n">b0ff</span><span class="o">-</span><span class="n">ba4e4e099326</span><span class="p">;</span> </code></pre></div></div> <p>You can update both tables at once with a logged batch:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">BEGIN</span> <span class="n">BATCH</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span><span class="p">,</span> <span class="n">city</span><span class="p">,</span> <span class="k">state</span><span class="p">,</span> <span class="n">postal_code</span><span class="p">,</span> <span class="n">country</span><span class="p">,</span> <span class="n">balance</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">7895</span><span class="n">c6ff</span><span class="o">-</span><span class="mi">008</span><span class="n">b</span><span class="o">-</span><span class="mi">4</span><span class="n">e4c</span><span class="o">-</span><span class="n">b0ff</span><span class="o">-</span><span class="n">ba4e4e099326</span><span class="p">,</span> <span class="s1">'Bordeaux'</span><span class="p">,</span> <span class="s1">'Gironde'</span><span class="p">,</span> <span class="s1">'33000'</span><span class="p">,</span> <span class="s1">'France'</span><span class="p">,</span> <span class="mi">56</span><span class="p">.</span><span class="mi">20</span><span class="p">);</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="p">(</span><span class="n">postal_code</span><span class="p">,</span> <span class="n">id</span><span class="p">,</span> <span class="n">balance</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="s1">'33000'</span><span class="p">,</span> <span class="mi">7895</span><span class="n">c6ff</span><span class="o">-</span><span class="mi">008</span><span class="n">b</span><span class="o">-</span><span class="mi">4</span><span class="n">e4c</span><span class="o">-</span><span class="n">b0ff</span><span class="o">-</span><span class="n">ba4e4e099326</span><span class="p">,</span> <span class="mi">56</span><span class="p">.</span><span class="mi">20</span><span class="p">)</span> <span class="p">;</span> <span class="n">APPLY</span> <span class="n">BATCH</span><span class="p">;</span> </code></pre></div></div> <h3 id="source-of-truth">Source of truth</h3> <p>A common design pattern is to have one table act as the authoritative one over data, and if for some reason there is a mismatch or conflict in other tables as long as there is one considered “the source of truth” it makes it easy to fix any conflicts later. This is typically the table that would match what we see in typical relational databases and has all the data needed to generate all related views or indexes for different query methods. Taking the prior example, <code class="language-plaintext highlighter-rouge">my_table</code> is the source of truth:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">--source of truth table</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="p">(</span><span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">name</span> <span class="nb">text</span><span class="p">,</span> <span class="n">address</span> <span class="nb">text</span><span class="p">,</span> <span class="n">city</span> <span class="nb">text</span><span class="p">,</span> <span class="k">state</span> <span class="nb">text</span><span class="p">,</span> <span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">country</span> <span class="nb">text</span><span class="p">,</span> <span class="n">balance</span> <span class="nb">float</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">id</span><span class="p">));</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">7895</span><span class="n">c6ff</span><span class="o">-</span><span class="mi">008</span><span class="n">b</span><span class="o">-</span><span class="mi">4</span><span class="n">e4c</span><span class="o">-</span><span class="n">b0ff</span><span class="o">-</span><span class="n">ba4e4e099326</span><span class="p">;</span> <span class="c1">--based on my_key.my_table and so we can query by postal_code</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="p">(</span><span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">balance</span> <span class="nb">float</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">postal_code</span><span class="p">,</span> <span class="n">id</span><span class="p">));</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="k">WHERE</span> <span class="n">postal_code</span> <span class="o">=</span> <span class="s1">'77002'</span><span class="p">;</span> </code></pre></div></div> <p>Next we discuss strategies for keeping tables of related in sync.</p> <h3 id="materialized-views">Materialized views</h3> <p>Materialized views are a feature that ships with Cassandra but is currently considered rather experimental. If you want to use them anyway:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="n">MATERIALIZED</span> <span class="k">VIEW</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code</span> <span class="k">AS</span> <span class="k">SELECT</span> <span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">balance</span> <span class="nb">float</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="k">WHERE</span> <span class="n">postal_code</span> <span class="k">IS</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">AND</span> <span class="n">id</span> <span class="k">IS</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">postal_code</span><span class="p">,</span> <span class="n">id</span><span class="p">));</span> </code></pre></div></div> <p>Materialized views at least run faster than the comparable BATCH insert pattern, but they have a number of bugs and known issues that are still pending fixes.</p> <h3 id="secondary-indexes">Secondary indexes</h3> <p>This are the original server side approach to handling different query patterns but it has a large number of downsides:</p> <ul> <li>rows are read serially one node at time until limit is reached.</li> <li>a suboptimal storage layout leading to very large partitions if the data distribution of the secondary index is not ideal.</li> </ul> <p>For just those two reasons I think it’s rare that one can use secondary indexes and expect reasonable performance. However, you can make one by hand and just query that data asynchronously to avoid some of the downsides.</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code_2i</span> <span class="p">(</span><span class="n">postal_code</span> <span class="nb">text</span><span class="p">,</span> <span class="n">id</span> <span class="n">uuid</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">postal_code</span><span class="p">,</span> <span class="n">id</span><span class="p">));</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table_by_postal_code_2i</span> <span class="k">WHERE</span> <span class="n">postal_code</span> <span class="o">=</span> <span class="s1">'77002'</span><span class="p">;</span> <span class="c1">--retrieve all rows then asynchronously query the resulting ids</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="n">ad004ff2</span><span class="o">-</span><span class="n">e5cb</span><span class="o">-</span><span class="mi">4245</span><span class="o">-</span><span class="mi">94</span><span class="n">b8</span><span class="o">-</span><span class="n">d6acbc22920a</span><span class="p">;</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="n">d30e9c65</span><span class="o">-</span><span class="mi">17</span><span class="n">a1</span><span class="o">-</span><span class="mi">44</span><span class="n">da</span><span class="o">-</span><span class="n">bae0</span><span class="o">-</span><span class="n">b7bb742eefd6</span><span class="p">;</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">my_key</span><span class="p">.</span><span class="n">my_table</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="n">e016ae43</span><span class="o">-</span><span class="mi">3</span><span class="n">d4e</span><span class="o">-</span><span class="mi">4093</span><span class="o">-</span><span class="n">b745</span><span class="o">-</span><span class="mi">8583627</span><span class="n">eb1fe</span><span class="p">;</span> </code></pre></div></div> <h2 id="exercises">Exercises</h2> <h3 id="contact-list">Contact List</h3> <p>This is a good basic first use case as one needs to use multiple tables for the same data, but there should not be too many.</p> <h4 id="requirements">requirements</h4> <ul> <li>contacts should have first name, last name, address, state/region, country, postal code</li> <li>lookup by contacts id</li> <li>retrieve all contacts by a given last name</li> <li>retrieve counts by zip code</li> </ul> <h3 id="music-service">Music Service</h3> <p>Takes the basics from the previous exercise and requires a more involved understanding of the concepts. It will require many tables and some difficult trade-offs on partition sizing. There is no one correct way to do this.</p> <h4 id="requirements-1">requirements</h4> <ul> <li>songs should have album, artist, name, and total likes</li> <li>The contact list exercise, can be used as a basis for the “users”, users will have no login because we’re trusting people</li> <li>retrieve all songs by artist</li> <li>retrieve all songs in an album</li> <li>retrieve individual song and how many times it’s been liked</li> <li>retrieve all liked songs for a given user</li> <li>“like” a song</li> <li>keep a count of how many times a song has been listened to by all users</li> </ul> <h3 id="iot-analytics">IoT Analytics</h3> <p>This will require some extensive time series modeling and takes some of the lessons from the Music Service further. The table(s) used will be informed by the query.</p> <h4 id="requirements-2">requirements</h4> <ul> <li>use the music service data model as a basis, we will be tracking each “registered device” that uses the music service</li> <li>a given user will have 1-5 devices</li> <li>log all songs listened to by a given device</li> <li>retrieve songs listened for a device by day</li> <li>retrieve songs listened for a device by month</li> <li>retrieve total listen time for a device by day</li> <li>retrieve total listen time for a device by month</li> <li>retrieve artists listened for a device by day</li> <li>retrieve artists listened for a device by month</li> </ul> Getting started with Cassandra: Load testing Cassandra in brief https://lostechies.com/ryansvihla/2020/02/04/getting-started-cassandra-part-2/ Los Techies urn:uuid:c943ad17-c8c9-9027-730e-494f4fdb5d29 Tue, 04 Feb 2020 20:23:00 +0000 An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra. <p>An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a <em>lot</em> of the nuance that one would normally get into when talking about load testing cassandra.</p> <h2 id="if-you-have-no-data-model-in-mind">If you have no data model in mind</h2> <p>Use cassandra stress since it’s around:</p> <ul> <li>first initialize the keyspace with RF3 <code class="language-plaintext highlighter-rouge">cassandra-stress "write cl=ONE no-warmup -col size=FIXED(15000) -schema replication(strategy=SimpleStrategy,factor=3)"</code></li> <li>second run stress <code class="language-plaintext highlighter-rouge">cassandra-stress "mixed n=1000k cl=ONE -col size=FIXED(15000)</code></li> <li>repeat as often as you’d like with as many clients as you want.</li> </ul> <h2 id="if-you-have-a-specific-data-model-in-mind">If you have a specific data model in mind</h2> <p>You can use cassandra-stress, but I suspect you’re going to find your data model isn’t supported (collections for example) or that you don’t have the required PHD to make it work the way you want. There are probably 2 dozen options from here you can use to build your load test, some of the more popular ones are gatling, jmeter, and tlp-stress. My personal favorite for this though, write a small simple python or java program that replicates your use case accurately in your own code, using a faker library to generate your data. This takes more time but you tend to have less surprises in production as it will accurately model your code.</p> <h3 id="small-python-script-with-python-driver">Small python script with python driver</h3> <ul> <li>use python3 and virtualenv</li> <li><code class="language-plaintext highlighter-rouge">python -m venv venv</code></li> <li>source venv/bin/activate</li> <li>read and follow install <a href="https://docs.datastax.com/en/developer/python-driver/3.21/getting_started/">docs</a></li> <li>if you want to skip the docs you can get away with <code class="language-plaintext highlighter-rouge">pip install cassandra-driver</code></li> <li>install a faker library <code class="language-plaintext highlighter-rouge">pip install Faker</code></li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">argparse</span> <span class="kn">import</span> <span class="nn">uuid</span> <span class="kn">import</span> <span class="nn">time</span> <span class="kn">import</span> <span class="nn">random</span> <span class="kn">from</span> <span class="nn">cassandra.cluster</span> <span class="kn">import</span> <span class="n">Cluster</span> <span class="kn">from</span> <span class="nn">cassandra.query</span> <span class="kn">import</span> <span class="n">BatchStatement</span> <span class="kn">from</span> <span class="nn">faker</span> <span class="kn">import</span> <span class="n">Faker</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="p">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s">'simple load generator for cassandra'</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">'--hosts'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s">'127.0.0.1'</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'comma separated list of hosts to use for contact points'</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">'--port'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">9042</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'port to connect to'</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">'--trans'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">1000000</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'number of transactions'</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">'--inflight'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">25</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'number of operations in flight'</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">'--errors'</span><span class="p">,</span> <span class="n">default</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'number of errors before stopping. default is unlimited'</span><span class="p">)</span> <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="p">.</span><span class="n">parse_args</span><span class="p">()</span> <span class="n">fake</span> <span class="o">=</span> <span class="n">Faker</span><span class="p">([</span><span class="s">'en-US'</span><span class="p">])</span> <span class="n">hosts</span> <span class="o">=</span> <span class="n">args</span><span class="p">.</span><span class="n">hosts</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">","</span><span class="p">)</span> <span class="n">cluster</span> <span class="o">=</span> <span class="n">Cluster</span><span class="p">(</span><span class="n">hosts</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="n">args</span><span class="p">.</span><span class="n">port</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="n">session</span> <span class="o">=</span> <span class="n">cluster</span><span class="p">.</span><span class="n">connect</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s">"setup schema"</span><span class="p">);</span> <span class="n">session</span><span class="p">.</span><span class="n">execute</span><span class="p">(</span><span class="s">"CREATE KEYSPACE IF NOT EXISTS my_key WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1}"</span><span class="p">)</span> <span class="n">session</span><span class="p">.</span><span class="n">execute</span><span class="p">(</span><span class="s">"CREATE TABLE IF NOT EXISTS my_key.my_table (id uuid, name text, address text, state text, zip text, balance int, PRIMARY KEY(id))"</span><span class="p">)</span> <span class="n">session</span><span class="p">.</span><span class="n">execute</span><span class="p">(</span><span class="s">"CREATE TABLE IF NOT EXISTS my_key.my_table_by_zip (zip text, id uuid, balance bigint, PRIMARY KEY(zip, id))"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">"allow schema to replicate throughout the cluster for 30 seconds"</span><span class="p">)</span> <span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">"prepare queries"</span><span class="p">)</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">session</span><span class="p">.</span><span class="n">prepare</span><span class="p">(</span><span class="s">"INSERT INTO my_key.my_table (id, name, address, state, zip, balance) VALUES (?, ?, ?, ?, ?, ?)"</span><span class="p">)</span> <span class="n">insert_rollup</span> <span class="o">=</span> <span class="n">session</span><span class="p">.</span><span class="n">prepare</span><span class="p">(</span><span class="s">"INSERT INTO my_key.my_table_by_zip (zip, id, balance) VALUES (?, ?, ?)"</span><span class="p">)</span> <span class="n">row_lookup</span> <span class="o">=</span> <span class="n">session</span><span class="p">.</span><span class="n">prepare</span><span class="p">(</span><span class="s">"SELECT * FROM my_key.my_table WHERE id = ?"</span><span class="p">)</span> <span class="n">rollup</span> <span class="o">=</span> <span class="n">session</span><span class="p">.</span><span class="n">prepare</span><span class="p">(</span><span class="s">"SELECT sum(balance) FROM my_key.my_table_by_zip WHERE zip = ?"</span><span class="p">)</span> <span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">ids</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">error_counter</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">query</span> <span class="o">=</span> <span class="bp">None</span> <span class="n">params</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">ids</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">get_id</span><span class="p">():</span> <span class="n">items</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">ids</span><span class="p">)</span> <span class="k">if</span> <span class="n">items</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="c1">## nothing present so return something random </span> <span class="k">return</span> <span class="n">uuid</span><span class="p">.</span><span class="n">uuid4</span><span class="p">()</span> <span class="k">if</span> <span class="n">items</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span> <span class="k">return</span> <span class="n">ids</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">return</span> <span class="n">ids</span><span class="p">[</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">items</span> <span class="o">-</span><span class="mi">1</span><span class="p">)]</span> <span class="k">print</span><span class="p">(</span><span class="s">"starting transactions"</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">trans</span><span class="p">):</span> <span class="n">chance</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="k">if</span> <span class="n">chance</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">chance</span> <span class="o">&lt;</span> <span class="mi">50</span><span class="p">:</span> <span class="n">new_id</span> <span class="o">=</span> <span class="n">uuid</span><span class="p">.</span><span class="n">uuid4</span><span class="p">()</span> <span class="n">ids</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_id</span><span class="p">)</span> <span class="n">state</span> <span class="o">=</span> <span class="n">fake</span><span class="p">.</span><span class="n">state_abbr</span><span class="p">()</span> <span class="n">zip_code</span> <span class="o">=</span> <span class="n">fake</span><span class="p">.</span><span class="n">zipcode_in_state</span><span class="p">(</span><span class="n">state</span><span class="p">)</span> <span class="n">balance</span> <span class="o">=</span> <span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">50000</span><span class="p">)</span> <span class="n">query</span> <span class="o">=</span> <span class="n">BatchStatement</span><span class="p">()</span> <span class="n">name</span> <span class="o">=</span> <span class="n">fake</span><span class="p">.</span><span class="n">name</span><span class="p">()</span> <span class="n">address</span> <span class="o">=</span> <span class="n">fake</span><span class="p">.</span><span class="n">address</span><span class="p">()</span> <span class="n">bound_insert</span> <span class="o">=</span> <span class="n">insert</span><span class="p">.</span><span class="n">bind</span><span class="p">([</span><span class="n">new_id</span><span class="p">,</span> <span class="n">fake</span><span class="p">.</span><span class="n">name</span><span class="p">(),</span> <span class="n">fake</span><span class="p">.</span><span class="n">address</span><span class="p">(),</span> <span class="n">state</span><span class="p">,</span> <span class="n">zip_code</span><span class="p">,</span> <span class="n">balance</span><span class="p">])</span> <span class="n">query</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">bound_insert</span><span class="p">)</span> <span class="n">bound_insert_rollup</span> <span class="o">=</span> <span class="n">insert_rollup</span><span class="p">.</span><span class="n">bind</span><span class="p">([</span><span class="n">zip_code</span><span class="p">,</span> <span class="n">new_id</span><span class="p">,</span> <span class="n">balance</span><span class="p">])</span> <span class="n">query</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">bound_insert_rollup</span><span class="p">)</span> <span class="k">elif</span> <span class="n">chance</span> <span class="o">&gt;</span> <span class="mi">50</span> <span class="ow">and</span> <span class="n">chance</span> <span class="o">&lt;</span> <span class="mi">75</span><span class="p">:</span> <span class="n">query</span> <span class="o">=</span> <span class="n">row_lookup</span><span class="p">.</span><span class="n">bind</span><span class="p">([</span><span class="n">get_id</span><span class="p">()])</span> <span class="k">elif</span> <span class="n">chance</span> <span class="o">&gt;</span> <span class="mi">75</span><span class="p">:</span> <span class="n">zip_code</span> <span class="o">=</span> <span class="n">fake</span><span class="p">.</span><span class="n">zipcode</span><span class="p">()</span> <span class="n">query</span> <span class="o">=</span> <span class="n">rollup</span><span class="p">.</span><span class="n">bind</span><span class="p">([</span><span class="n">zip_code</span><span class="p">])</span> <span class="n">threads</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">session</span><span class="p">.</span><span class="n">execute_async</span><span class="p">(</span><span class="n">query</span><span class="p">))</span> <span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="n">args</span><span class="p">.</span><span class="n">inflight</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="n">t</span><span class="p">.</span><span class="n">result</span><span class="p">()</span> <span class="c1">#we don't care about result so toss it </span> <span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s">"unexpected exception %s"</span> <span class="o">%</span> <span class="n">e</span><span class="p">)</span> <span class="k">if</span> <span class="n">args</span><span class="p">.</span><span class="n">errors</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span> <span class="n">error_counter</span> <span class="o">=</span> <span class="n">error_counter</span> <span class="o">+</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">error_counter</span> <span class="o">&gt;</span> <span class="n">args</span><span class="p">.</span><span class="n">errors</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s">"too many errors stopping. Consider raising --errors flag if this happens more quickly than you'd like"</span><span class="p">)</span> <span class="k">break</span> <span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">print</span><span class="p">(</span><span class="s">"submitted %i of %i transactions"</span> <span class="o">%</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">args</span><span class="p">.</span><span class="n">trans</span><span class="p">))</span> <span class="k">finally</span><span class="p">:</span> <span class="n">cluster</span><span class="p">.</span><span class="n">shutdown</span><span class="p">()</span> </code></pre></div></div> <h3 id="small-java-program-with-latest-java-driver">Small java program with latest java driver</h3> <ul> <li>download java 8</li> <li>create a command line application in your project technology of choice (I used maven in this example for no particularly good reason)</li> <li>download a faker lib like <a href="https://github.com/DiUS/java-faker">this one</a> and the <a href="https://github.com/datastax/java-driver">Cassandra java driver from DataStax</a> again using your preferred technology to do so.</li> <li>run the following code sample somewhere (set your RF and your desired queries and data model)</li> <li>use different numbers of clients at your cluster until you get enough “saturation” or the server stops responding.</li> </ul> <p><a href="https://github.com/rssvihla/simple_cassandra_load_test/tree/master/java/simple-cassandra-stress">See complete example</a></p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">pro.foundev</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.lang.RuntimeException</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.lang.Thread</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.Locale</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.ArrayList</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.function.*</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.Random</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.UUID</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.CompletionStage</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.net.InetSocketAddress</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.datastax.oss.driver.api.core.CqlSession</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.datastax.oss.driver.api.core.CqlSessionBuilder</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.datastax.oss.driver.api.core.cql.*</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.github.javafaker.Faker</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">App</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">args</span> <span class="o">)</span> < Getting started with Cassandra: Setting up a Multi-DC environment https://lostechies.com/ryansvihla/2020/02/03/getting-started-cassandra-part-1/ Los Techies urn:uuid:f11f8200-727e-e8cb-60b2-06dc45a16751 Mon, 03 Feb 2020 20:23:00 +0000 This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers. <p>This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.</p> <h2 id="a-new-cluster">A new cluster</h2> <ul> <li>In cassandra.yaml set <code class="language-plaintext highlighter-rouge">endpoint_snitch: GossipingPropertyFileSnitch</code>, some prefer PropertyFileSnitch for the ease of pushing out one file. GossipingPropertyFileSnitch is harder to get wrong in my experience.</li> <li>set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.</li> <li>finish adding all of your nodes.</li> <li>if using authentication, set <code class="language-plaintext highlighter-rouge">system_auth</code> keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created <code class="language-plaintext highlighter-rouge">ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};</code>, run repair after changing RF</li> <li><code class="language-plaintext highlighter-rouge">nodetool repair -pr system_auth</code> on each node in the cluster on the new keyspace.</li> <li>create your new keyspaces for your app with RF 3 in each dc (much like you did for the <code class="language-plaintext highlighter-rouge">system_auth</code> step above).</li> <li><code class="language-plaintext highlighter-rouge">nodetool repair -pr whatever_new_keyspace</code> on each node in the cluster on the new keyspace.</li> </ul> <h2 id="an-existing-cluster">An existing cluster</h2> <p>This is harder and involves more work and more options, but I’m going to discuss the way that gets you into the least amount of trouble operationally.</p> <ul> <li>make sure <em>none</em> of the drivers you use to connect to cassnadra are using DowngradingConsistencyRetryPolicy, or using the maligned withUsedHostsPerRemoteDc, especially allowRemoteDCsForLocalConsistencyLevel, as this may cause your driver to send requests to the remote data center before it’s populated with data.</li> <li>switch <code class="language-plaintext highlighter-rouge">endpoint_snitch</code> on each node to GossipingPropertyFileSnitch</li> <li>set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.</li> <li>bootstrap each node in the new data center.</li> <li>if using authentication, set <code class="language-plaintext highlighter-rouge">system_auth</code> keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created <code class="language-plaintext highlighter-rouge">ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};</code>, run repair after changing RF</li> <li><code class="language-plaintext highlighter-rouge">nodetool repair -pr system_auth</code> on each node in the cluster on the new keyspace.</li> <li>alter your app keyspaces for your app with RF 3 in each dc (much like you did for the <code class="language-plaintext highlighter-rouge">system_auth</code> step above),</li> <li><code class="language-plaintext highlighter-rouge">nodetool repair -pr whatever_keyspace</code> on each node in the cluster on the new keyspace.</li> </ul> <p>enjoy new data center</p> <h3 id="how-to-get-data-to-new-dc">how to get data to new dc</h3> <h4 id="repair-approach">Repair approach</h4> <p>Best done with if your repair jobs can’t be missed or stopped, either because you have a process like opscenter or repear running repairs. It also has the advantage of being very easy, and if you’ve already automated repair you’re basically done.</p> <ul> <li>let repair jobs continue..that’s it!</li> </ul> <h4 id="rebuild-approach">Rebuild approach</h4> <p>Faster less resource intensive, and if you have enough time to complete it while repair is stopped. Rebuild is easier to ‘resume’ than repair in many ways, so this has a number of advantages.</p> <ul> <li>run <code class="language-plaintext highlighter-rouge">nodetool rebuild</code> on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.</li> <li>run <code class="language-plaintext highlighter-rouge">nodetool cleanup</code></li> </ul> <h4 id="yolo-rebuild-with-repair">YOLO rebuild with repair</h4> <p>This will probably overstream it’s share of data and honestly a lot of folks do this for some reason in practice:</p> <ul> <li>leave repair jobs running</li> <li>run <code class="language-plaintext highlighter-rouge">nodetool rebuild</code> on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.</li> <li>run <code class="language-plaintext highlighter-rouge">nodetool cleanup</code> on each node</li> </ul> <h2 id="cloud-strategies">Cloud strategies</h2> <p>There are a few valid approaches to this and none of them are wrong IMO.</p> <h3 id="region--dc-rack--az">region == DC, rack == AZ</h3> <p>Will need to get into racks and a lot of people get this wrong and imbalance the racks, but you get the advantage of more intelligent failure modes, with racks mapping to AZs.</p> <h3 id="azregardless-of-region--dc">AZ..regardless of region == DC</h3> <p>This allows things to be balanced easily, but you have no good option for racks then. However, some people think racks are overated, and I’d say a majority of clusters run with one rack.</p> Gocode Vim Plugin and Go Modules https://blog.jasonmeridth.com/posts/gocode-vim-plugin-and-go-modules/ Jason Meridth urn:uuid:c9be1149-395b-e365-707e-8fa2f475093c Sat, 05 Jan 2019 17:09:26 +0000 <p>I recently purchased <a href="https://lets-go.alexedwards.net/">Let’s Go</a> from Alex Edwards. I wanted an end-to-end Golang website tutorial. It has been great. Lots learned.</p> <p>Unfortunately, he is using Go’s modules and the version of the gocode Vim plugin I was using did not support Go modules.</p> <h3 id="solution">Solution:</h3> <p>Use <a href="https://github.com/stamblerre/gocode">this fork</a> of the gocode Vim plugin and you’ll get support for Go modules.</p> <p>I use <a href="https://github.com/junegunn/vim-plug">Vim Plug</a> for my Vim plugins. Huge fan of Vundle but I like the post-actions feature of Plug. I just had to change one line of my vimrc and re-run updates.</p> <div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh">diff --git a/vimrc b/vimrc index 3e8edf1..8395705 100644 </span><span class="gd">--- a/vimrc </span><span class="gi">+++ b/vimrc </span><span class="gu">@@ -73,7 +73,7 @@ endif </span> let editor_name='nvim' Plug 'zchee/deoplete-go', { 'do': 'make'} endif <span class="gd">- Plug 'nsf/gocode', { 'rtp': 'vim', 'do': '~/.config/nvim/plugged/gocode/vim/symlink.sh' } </span><span class="gi">+ Plug 'stamblerre/gocode', { 'rtp': 'vim', 'do': '~/.vim/plugged/gocode/vim/symlink.sh' } </span> Plug 'godoctor/godoctor.vim', {'for': 'go'} " Gocode refactoring tool " } </code></pre></div></div> <p>That is the line I had to change then run <code class="highlighter-rouge">:PlugUpdate!</code> and the new plugin was installed.</p> <p>I figured all of this out due to <a href="https://github.com/zchee/deoplete-go/issues/134#issuecomment-435436305">this comment</a> by <a href="https://github.com/cippaciong">Tommaso Sardelli</a> on Github. Thank you Tommaso.</p> Raspberry Pi Kubernetes Cluster - Part 4 https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-4/ Jason Meridth urn:uuid:56f4fdcb-5310-bbaa-c7cf-d34ef7af7682 Fri, 28 Dec 2018 16:35:23 +0000 <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-1">Raspberry Pi Kubenetes Cluster - Part 1</a></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-2">Raspberry Pi Kubenetes Cluster - Part 2</a></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-3">Raspberry Pi Kubenetes Cluster - Part 3</a></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-4">Raspberry Pi Kubenetes Cluster - Part 4</a></p> <p>Howdy again.</p> <p>In this post I’m going to show you how to create a docker image to run on ARM architecture and also how to deploy it and view it.</p> <p>To start please view my basic flask application called fl8 <a href="https://github.com/meridth/fl8">here</a></p> <p>If you’d like to clone and use it:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@github.com:meridth/fl8.git <span class="o">&amp;&amp;</span> <span class="nb">cd </span>fl8 </code></pre></div></div> <h1 id="arm-docker-image">ARM docker image</h1> <p>First we need to learn about QEMU</p> <h3 id="what-is-qemu-and-qemu-installation">What is QEMU and QEMU installation</h3> <p>QEMU (Quick EMUlator) is an Open-Source hosted hypervisor, i.e. an hypervisor running on a OS just as other computer programs, which performs hardware virtualization. QEMU emulates CPUs of several architectures, e.g. x86, PPC, ARM and SPARC. It allows the execution of non-native target executables emulating the native execution and, as we require in this case, the cross-building process.</p> <h3 id="base-docker-image-that-includes-qemu">Base Docker image that includes QEMU</h3> <p>Please open the <code class="highlighter-rouge">Dockerfile.arm</code> and notice the first line: <code class="highlighter-rouge">FROM hypriot/rpi-alpine</code>. This is a base image that includes the target qemu statically linked executable, <em>qemu-arm-static</em> in this case. I chose <code class="highlighter-rouge">hypriot/rpi-alpine</code> because the alpine base images are much smaller than other base images.</p> <h3 id="register-qemu-in-the-build-agent">Register QEMU in the build agent</h3> <p>To add QEMU in the build agent there is a specific Docker Image performing what we need, so just run in your command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">--rm</span> <span class="nt">--privileged</span> multiarch/qemu-user-static:register <span class="nt">--reset</span> </code></pre></div></div> <h3 id="build-image">Build image</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build <span class="nt">-f</span> ./Dockerfile.arm <span class="nt">-t</span> meridth/rpi-fl8 <span class="nb">.</span> </code></pre></div></div> <p>And voila! You now have an image that will run on Raspberry Pis.</p> <h1 id="deployment-and-service">Deployment and Service</h1> <p><code class="highlighter-rouge">/.run-rpi.sh</code> is my script where I run a Kubernetes deployment with 3 replicas and a Kubernetes service. Please read <code class="highlighter-rouge">fl8-rpi-deployment.yml</code> and <code class="highlighter-rouge">fl8-rpi-service.yml</code>. They are only different from the other deployment and service files by labels. Labels are key/vaule pairs that can be used by selectors later.</p> <p>The deployment will pull my image from <code class="highlighter-rouge">meridth/rpi-fl8</code> on dockerhub. If you have uploaded your docker image somewhere you can change the deployment file to pull that image instead.</p> <h1 id="viewing-application">Viewing application</h1> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl get pods </code></pre></div></div> <p>Choose a pod to create the port forwarding ssh tunnel.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl port-forward <span class="o">[</span>pod-name] <span class="o">[</span>app-port]:[app-port] </code></pre></div></div> <p>Example: <code class="highlighter-rouge">kubectl port-forward rpi-fl8-5d84dd8ff6-d9tgz 5010:5010</code></p> <p>The final result when you go to <code class="highlighter-rouge">http://localhost:5010</code> in a browser.</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/port_forward.png" alt="port forward result" /></p> <p>Hope this helps someone else. Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-4/">Raspberry Pi Kubernetes Cluster - Part 4</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on December 28, 2018.</p> Raspberry Pi Kubernetes Cluster - Part 3 https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-3/ Jason Meridth urn:uuid:c12fa6c5-8e7a-6c5d-af84-3c0452cf4ae4 Mon, 24 Dec 2018 21:59:23 +0000 <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-1">Raspberry Pi Kubenetes Cluster - Part 1</a></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-2">Raspberry Pi Kubenetes Cluster - Part 2</a></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-3">Raspberry Pi Kubenetes Cluster - Part 3</a></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-4">Raspberry Pi Kubenetes Cluster - Part 4</a></p> <p>Well, it took me long enough to follow up on my previous posts. There are reasons.</p> <ol> <li>The day job has been fun and busy</li> <li>Family life has been fun and busy</li> <li>I kept hitting annoying errors when trying to get my cluster up and running</li> </ol> <p>The first two reasons are the usual reasons a person doesn’t blog. :)</p> <p>The last one is what prevented me from blogging sooner. I had mutliple issues when trying to use <a href="https://rak8s.io">rak8s</a> to setup my cluster. I’m a big fan of <a href="https://ansible.com">Ansible</a> and I do not like running scripts over and over. I did read <a href="https://gist.github.com/alexellis/fdbc90de7691a1b9edb545c17da2d975">K8S on Raspbian Lite</a> from top to bottom and realized automation would make this much better.</p> <!--more--> <h3 id="the-issues-i-experienced">The issues I experienced:</h3> <h4 id="apt-get-update-would-not-work">apt-get update would not work</h4> <p>I started with the vanilla Raspbian lite image to run on my nodes and had MANY MANY issues with running <code class="highlighter-rouge">apt-get update</code> and <code class="highlighter-rouge">apt-get upgrade</code>. The mirrors would disconnect often and just stall. This doesn’t help my attempted usage of rak8s which does both on the <code class="highlighter-rouge">cluster.yml</code> run (which I’ll talk about later).</p> <h4 id="rak8s-changes-needed-to-run-hypriotos-and-kubernetes-1131">rak8s changes needed to run HypriotOS and kubernetes 1.13.1</h4> <p>Clone the repo locally and I’ll walk you through what I changed to get <a href="https://rak8s.io">rak8s</a> working for me and HypriotOS.</p> <p>Change the following files:</p> <ul> <li><code class="highlighter-rouge">ansible.cfg</code> <ul> <li>change user from <code class="highlighter-rouge">pi</code> to <code class="highlighter-rouge">pirate</code></li> </ul> </li> <li><code class="highlighter-rouge">roles/kubeadm/tasks/main.yml</code> <ul> <li>add <code class="highlighter-rouge">ignore_errors: True</code> to <code class="highlighter-rouge">Disable Swap</code> task</li> <li>I have an open PR for this <a href="https://github.com/rak8s/rak8s/pull/46">here</a></li> </ul> </li> <li><code class="highlighter-rouge">group_vars/all.yml</code> <ul> <li>Change <code class="highlighter-rouge">kubernetes_package_version</code> to <code class="highlighter-rouge">"1.13.1-00"</code></li> <li>Change <code class="highlighter-rouge">kubernetes_version</code> to <code class="highlighter-rouge">"v1.13.1"</code></li> </ul> </li> </ul> <p>After you make those changes you can run <code class="highlighter-rouge">ansible-playbook cluster.yml</code> as the rak8s documentation suggests. Please note this is after you edit <code class="highlighter-rouge">inventory</code> and copy <code class="highlighter-rouge">ssh</code> keys to raspberry pis.</p> <h4 id="flannel-networking-issue-once-nodes-are-up">Flannel networking issue once nodes are up</h4> <p>After I get all of the nodes up I noticed the master node was marked ast <code class="highlighter-rouge">NotReady</code> and when I ran <code class="highlighter-rouge">kubectl describe node raks8000</code> I saw the following error:</p> <blockquote> <p>KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized</p> </blockquote> <p>This error is known in kubernetes &gt; 1.12 and flannel v0.10.0. It is mentioned in <a href="https://github.com/coreos/flannel/issues/1044">this issue</a>. The fix is specifically mentioned <a href="https://github.com/coreos/flannel/issues/1044#issuecomment-427247749">here</a>. It is to run the following command</p> <p><code class="highlighter-rouge">kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml</code></p> <p>After readin the issue it seems the fix will be in the next version of flannel and will be backported to v0.10.0.</p> <h1 id="a-running-cluster">A running cluster</h1> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/running_cluster.png" alt="Running Cluster" /></p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-3/">Raspberry Pi Kubernetes Cluster - Part 3</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on December 24, 2018.</p> MVP how minimal https://lostechies.com/ryansvihla/2018/12/20/mvp-how-minimal/ Los Techies urn:uuid:3afadd9e-98a7-8d37-b797-5403312a2999 Thu, 20 Dec 2018 20:00:00 +0000 MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas. In recent history I’ve seen up close two extreme constrasting examples of MVP: <p>MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas. In recent history I’ve seen up close two extreme constrasting examples of MVP:</p> <ul> <li>Mega Minimal: website and db, mostly manual on the backend</li> <li>Mega Mega: provisioning system, dynamic tuning of systems via ML, automated operations, monitoring a few others I’m leaving out.</li> </ul> <h2 id="feedback">Feedback</h2> <p>If we’re evaluating which approach gives us more feedback, Mega Minimal MVP is gonna win hands down here. Some will counter they don’t want to give people a bad impression with a limited product and that’s fair, but it’s better than no impression (the dreaded never shipped MVP). The Mega Mega MVP I referenced took months to demo. only had one of those checkboxes setup and wasn’t ever demod again. So we can categorical say that failed at getting any feedback.</p> <p>Whereas the Mega Minimal MVP, got enough feedback and users for the founders to realize that wasn’t a business for them. Better than after hiring a huge team and sinking a million plus into dev efforts for sure. Not the happy ending I’m sure you all were expecting, but I view that as mission accomplished.</p> <h2 id="core-value">Core Value</h2> <ul> <li>Mega Minimal, they only focused on a single feature, executed well enough that people gave them some positive feedback, but not enough to justify automating everything.</li> <li>Mega Mega. I’m not sure anyone who talked about the product saw the same core value, and there were several rewrites and shifts along the way.</li> </ul> <p>Advantage Mega Minimal again</p> <h2 id="what-about-entrants-into-a-crowded-field">What about entrants into a crowded field</h2> <p>Well that is harder and the MVP tends to be less minimal, because the baseline expectations are just much higher. I still lean towards Mega Minimal having a better chance at getting users, since there is a non zero chance the Mega Mega MVP will never get finished. I still think the exercise in focusing on core value that makes your product <em>not</em> a me too, and even considering how you can find a niche in a crowded field instead of just being “better”, and your MVP can be that niche differentiator.</p> <h2 id="internal-users">Internal users</h2> <p>Sometimes a good middle ground is considering getting lots of internal users if you’re really worried about bad experiences. This has it’s it’s definite downsides however, and you may not get diverse enough opinions. But it does give you some feedback while saving some face or bad experiences. I often think of the example of EC2 that was heavily used by Amazon, before being released to the world. That was a luxury Amazon had, where their customer base and their user base happened to be very similar, and they had bigger scale needs than any of their early customers, so the early internal feedback loop was a very strong signal.</p> <h2 id="summary">Summary</h2> <p>In the end however you want to approach MVPs is up to you, and if you find success with a meatier MVP than I have please don’t let me push you away from what works. But if you are having trouble shipping and are getting pushed all the time to add one more feature to that MVP before releasing it, consider stepping back and asking is this really core value for the product? Do you already have your core value? if so, consider just releasing it.</p> Surprise Go is ok for me now https://lostechies.com/ryansvihla/2018/12/13/surprise-go-is-ok/ Los Techies urn:uuid:53abf2a3-23f2-5855-0e2d-81148fb908bf Thu, 13 Dec 2018 20:23:00 +0000 I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve. <p>I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve.</p> <p>About 7 months ago I wrote about all the things I didn’t really care for in Go and now I either no longer am so bothered by it or things have improved.</p> <p>Go Modules so far is a huge improvement over Dep and Glide for dependency management. It’s easy to setup, performant and eliminates the GOPATH silliness. I haven’t tried it yet with some of the goofier libraries that gave me problems in the past (k8s api for example) so the jury is out on that, but again pretty impressed. I now longer have to check in vendor to speed up builds. Lesson use Go Modules.</p> <p>I pretty much stopped using channels for everything but shutdown signals and that fits my preferences pretty well, I use mutex and semaphores for my multithreaded code and feel no guilt about it. This cut out a lot of pain for me, and with the excellent race detector I feel really comfortable writing multi-threaded in Go now. Lesson, don’t use channels much.</p> <p>Lack of generics still sometimes sucks but I usually implement some crappy casting with dynamic types if I need that. I’ve sorta made my piece with just writing more code, and am no longer so hung up. Lesson relax.</p> <p>Error handling I’m still struggling with, I thought about using one of the error Wrap() libraries but an official one is in draft spec now, so I’ll wait on that. I now tend to have less nesting of functions as a result, this probably means longer functions than I like, but my code looks more “normal” now. This is a trade off I’m ok with. Lesson relax more.</p> <p>I see the main virtue of Go now that it is very popular in the infrastructure space where I am and so it’s becoming the common tongue (largely replacing Python for those sorts of tasks). For this, honestly it’s about right. It’s easy to rip out command line tools and deploy binaries for every platform with no runtime install.</p> <p>The community’s conservative attitude I sort of view as a feature now, in that there isn’t a bunch of different options that are popular and there is no arguing over what file format is used. This drove me up the wall initially, but I appreciate how much less time I spend on these things now.</p> <p>So now I suspect Go will be my “last” programming language. It’s not the one I would have chosen, but where I am at in my career, where most of my dev work is automation and tooling it fits the bill pretty well.</p> <p>Also equally important most of the people working with me didn’t have full time careers as developers or spend their time reading “Domain Driven Design” (amazing book) so adding in a bunched of nuanced stuff that maybe technically optimal but assumes the reader grasps all that nuance isn’t a good tradeoff for me.</p> <p>So I think I sorta get it now. I’ll never be a cheerleader for the language but it definitely solves my problems well enough.</p> Collaboration vs. Critique https://lostechies.com/derekgreer/2018/05/18/collaboration-vs-critique/ Los Techies urn:uuid:8a2d0bfb-9efe-2fd2-1e9b-6ba6d06055da Fri, 18 May 2018 17:00:00 +0000 While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams. The people aspect of software development, more often than not, tends to be the most difficult part of software engineering. Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs. The nature of software development tends to follow an empirical process model rather than a defined process model. That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset. Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation. This means that team members are often required to work together to determine how to proceed. Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation. One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique. <p>While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams. The people aspect of software development, more often than not, tends to be the most difficult part of software engineering. Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs. The nature of software development tends to follow an empirical process model rather than a defined process model. That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset. Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation. This means that team members are often required to work together to determine how to proceed. Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation. One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique.</p> <p>To help illustrate what I’ve seen happen countless times both in catch-up design sessions and code reviews, consider the following two scenarios:</p> <h3 id="scenario-1">Scenario 1</h3> <p>Tom and Sally are both developers on a team maintaining a large-scale application. Tom takes the next task in the development queue which happens to have some complex processes that will need to be addressed. Being the good development team that they are, both Tom and Sally are aware of the requirements of the application (i.e. how the app needs to work from the user’s perspective), but they have deferred design-level discussions until the time of implementation. After Tom gets into the process a little, seeing that the problem is non-trivial, he pings Sally to help him brainstorm different approaches to solving the problem. Tom and Sally have been working together for over a year and have become accustomed to these sort of ad-hoc design sessions. As they begin discussing the problem, they each start tossing ideas out on the proverbial table resulting in multiple approaches to compare and contrast. The nature of the discussion is such that neither Tom nor Sally are embarrassed or offended when the other points out flaws in the design because there’s a sense of safety in their mutual understanding that this is a brainstorming session and that neither have thought in depth about the solutions being set froth yet. Tom throws out a couple of ideas, but ends up shooting them down himself as he uses Sally as a sounding board for the ideas. Sally does the same, but toward the end of the conversation suggests a slight alteration to one of Tom’s initial suggestions that they think may make it work after all. They end the session both with a sense that they’ve worked together to arrive at the best solution.</p> <h3 id="scenario-2">Scenario 2</h3> <p>Bill and Jake are developers on another team. They tend to work in a more siloed fashion, but they do rely upon one another for help from time to time and they are required to do code reviews prior to their code being merged into the main branch of development. Bill takes the next task in the development queue and spends the better part of an afternoon working out a solution with a basic working skeleton of the direction he’s going. The next day he decides that it might be good to have Jake take a look at the design to make him aware of the direction. Seeing where Bill’s design misses a few opportunities to make the implementation more adaptable to changes in the future, Jake points out where he would have done things differently. Bill acknowledges that Jake’s suggestions would be better and would have probably been just as easy to implement from the beginning, but inwardly he’s a bit disappointed that Jake didn’t like his design as-is and that he has to do some rework. In the end, Bill is left with a feeling of critique rather than collaboration.</p> <p>Whether it’s a high-level UML diagram or working code, how one person tends to perceive feedback on the ideas comprising a potential solution has everything to do with timing. It can be the exact same feedback they would have received either way, but when the feedback occurs often makes a difference between whether it’s perceived as collaboration or critique. It’s all about when the conversation happens.</p> Testing Button Click in React with Jest https://derikwhittaker.blog/2018/05/07/testing-button-click-in-react-with-jest/ Maintainer of Code, pusher of bits… urn:uuid:a8e7d9fd-d718-a072-55aa-0736ac21bec4 Mon, 07 May 2018 17:01:59 +0000 When building React applications you will most likely find yourself using Jest as your testing framework.  Jest has some really, really cool features built in.  But when you use Enzyme you can take your testing to the nest level. One really cool feature is the ability to test click events via Enzyme to ensure your &#8230; <p><a href="https://derikwhittaker.blog/2018/05/07/testing-button-click-in-react-with-jest/" class="more-link">Continue reading <span class="screen-reader-text">Testing Button Click in React with&#160;Jest</span></a></p> <p>When building <a href="https://reactjs.org/" target="_blank" rel="noopener">React</a> applications you will most likely find yourself using <a href="https://facebook.github.io/jest" target="_blank" rel="noopener">Jest</a> as your testing framework.  Jest has some really, really cool features built in.  But when you use <a href="http://airbnb.io/enzyme/docs/guides/jest.html" target="_blank" rel="noopener">Enzyme</a> you can take your testing to the nest level.</p> <p>One really cool feature is the ability to test click events via Enzyme to ensure your code responds as expected.</p> <p>Before we get started you are going to want to make sure you have Jest and Enzyme installed in your application.</p> <ul> <li>Installing <a href="https://github.com/airbnb/enzyme/blob/master/docs/installation/README.md" target="_blank" rel="noopener">Enzyme</a></li> <li>Installing <a href="https://facebook.github.io/jest/docs/en/getting-started.html" target="_blank" rel="noopener">Jest</a></li> </ul> <p>Sample code under test</p> <p><img data-attachment-id="111" data-permalink="https://derikwhittaker.blog/2018/05/07/testing-button-click-in-react-with-jest/screen-shot-2018-05-07-at-12-52-56-pm/" data-orig-file="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640" data-orig-size="580,80" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="Screen Shot 2018-05-07 at 12.52.56 PM" data-image-description="" data-medium-file="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640?w=300" data-large-file="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640?w=580" class="alignnone size-full wp-image-111" src="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=640" alt="Screen Shot 2018-05-07 at 12.52.56 PM" srcset="https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png 580w, https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=150 150w, https://derikwhittaker.files.wordpress.com/2018/05/screen-shot-2018-05-07-at-12-52-56-pm.png?w=300 300w" sizes="(max-width: 580px) 100vw, 580px" /></p> <p>What I would like to be able to do is pull the button out of my component and test the <code>onClick</code> event handler.</p> <div class="code-snippet"> <pre class="code-content"> // Make sure you have your imports setup correctly import React from 'react'; import { shallow } from 'enzyme'; it('When active link clicked, will push correct filter message', () =&gt; { let passedFilterType = ''; const handleOnTotalsFilter = (filterType) =&gt; { passedFilterType = filterType; }; const accounts = {}; const wrapper = shallow(&lt;MyComponent accounts={accounts} filterHeader="" onTotalsFilter={handleOnTotalsFilter} /&gt;); const button = wrapper.find('#archived-button'); button.simulate('click'); expect(passedFilterType).toBe(TotalsFilterType.archived); }); </pre> </div> <p>Lets take a look at the test above</p> <ol> <li>First we are going to create a callback (click handler) to catch the bubbled up values.</li> <li>We use Enzyme to create our component <code>MyComponent</code></li> <li>We use the .find() on our wrapped component to find our &lt;Button /&gt; by id</li> <li>After we get our button we can call .simulate(&#8216;click&#8217;) which will act as a user clicking the button.</li> <li>We can assert that the expected value bubbles up.</li> </ol> <p>As you can see, simulating a click event of a rendered component is very straight forward, yet very powerful.</p> <p>Till next time,</p> Lessons from a year of Golang https://lostechies.com/ryansvihla/2018/05/07/lessons-from-a-year-of-go/ Los Techies urn:uuid:e37d6484-2864-cc2a-034c-cac3d89dede7 Mon, 07 May 2018 13:16:00 +0000 I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive. <p>I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive.</p> <p>This isn’t meant to evaluate if you should pick up Go or tell you what you should think of it, this is strictly meant to help people out that are new to the language but experienced in Java, Python, Ruby, C#, etc and have read some basic Go getting started guide.</p> <h2 id="dependency-management">Dependency management</h2> <p>This is probably the feature most frequently talked about by newcomers to Go and with some justification, as dependency management been a rapidly shifting area that’s nothing like what experienced Java, C#, Ruby or Python developers are used to.</p> <p>I’ll cut to the chase the default tool now is <a href="https://github.com/golang/dep">Dep</a> all other tools I’ve used such as <a href="https://github.com/Masterminds/glide">Glide</a> or <a href="https://github.com/tools/godep">Godep</a> they’re now deprecated in favor of Dep, and while Dep has advanced rapidly there are some problems you’ll eventually run into (or I did):</p> <ol> <li>Dep hangs randomly and is slow, which is supposedly network traffic <a href="https://github.com/golang/dep/blob/c8be449181dadcb01c9118a7c7b592693c82776f/docs/failure-modes.md#hangs">but it happens to everyone I know with tons of bandwidth</a>. Regardless, I’d like an option to supply a timeout and report an error.</li> <li>Versions and transitive depency conflicts can be a real breaking issue in Go still. So without shading or it’s equivalent two package depending on different versions of a given package can break your build, there are a number or proposals to fix this but we’re not there yet.</li> <li>Dep has some goofy ways it resolves transitive dependencies and you may have to add explicit references to them in your Gopkg.toml file. You can see an example <a href="https://kubernetes.io/blog/2018/01/introducing-client-go-version-6/">here</a> under <strong>Updating dependencies – golang/dep</strong>.</li> </ol> <h3 id="my-advice">My advice</h3> <ul> <li>Avoid hangs by checking in your dependencies directly into your source repository and just using the dependency tool (dep, godep, glide it doesn’t matter) for downloading dependencies.</li> <li>Minimize transitive dependencies by keeping stuff small and using patterns like microservices when your dependency tree conflicts.</li> </ul> <h2 id="gopath">GOPATH</h2> <p>Something that takes some adjustment is you check out all your source code in one directory with one path (by default ~/go/src ) and include the path to the source tree to where you check out. Example:</p> <ol> <li>I want to use a package I found on github called jim/awesomeness</li> <li>I have to go to ~/go/src and mkdir -p github.com/jim</li> <li>cd into that and clone the package.</li> <li>When I reference the package in my source file it’ll be literally importing github.com/jim/awesomeness</li> </ol> <p>A better guide to GOPATH and packages is <a href="https://thenewstack.io/understanding-golang-packages/">here</a>.</p> <h3 id="my-advice-1">My advice</h3> <p>Don’t fight it, it’s actually not so bad once you embrace it.</p> <h2 id="code-structure">Code structure</h2> <p>This is a hot topic and there are a few standards for the right way to structure you code from projects that do “file per class” to giant files with general concept names (think like types.go and net.go). Also if you’re used to using a lot of sub package you’re gonna to have issues with not being able to compile if for example you have two sub packages reference one another.</p> <h3 id="my-advice-2">My Advice</h3> <p>In the end I was reasonably ok with something like the following:</p> <ul> <li>myproject/bin for generated executables</li> <li>myproject/cmd for command line code</li> <li>myproject/pkg for code related to the package</li> </ul> <p>Now whatever you do is fine, this was just a common idiom I saw, but it wasn’t remotely all projects. I also had some luck with just jamming everything into the top level of the package and keeping packages small (and making new packages for common code that is used in several places in the code base). If I ever return to using Go for any reason I will probably just jam everything into the top level directory.</p> <h2 id="debugging">Debugging</h2> <p>No debugger! There are some projects attempting to add one but Rob Pike finds them a crutch.</p> <h3 id="my-advice-3">My Advice</h3> <p>Lots of unit tests and print statements.</p> <h2 id="no-generics">No generics</h2> <p>Sorta self explanatory and it causes you a lot of pain when you’re used to reaching for these.</p> <h3 id="my-advice-4">My advice</h3> <p>Look at the code generation support which uses pragmas, this is not exactly the same as having generics but if you have some code that has a lot of boiler plate without them this is a valid alternative. See this official <a href="https://blog.golang.org/generate">Go Blog post</a> for more details.</p> <p>If you don’t want to use generation you really only have reflection left as a valid tool, which comes with all of it’s lack of speed and type safety.</p> <h2 id="cross-compiling">Cross compiling</h2> <p>If you have certain features or dependencies you may find you cannot take advantage of one of Go’s better features cross compilation.</p> <p>I ran into this when using the confluent-go/kafka library which depends on the C librdkafka library. It basically meant I had to do all my development in a Linux VM because almost all our packages relied on this.</p> <h3 id="my-advice-5">My Advice</h3> <p>Avoid C dependencies at all costs.</p> <h2 id="error-handling">Error handling</h2> <p>Go error handling is not exception base but return based, and it’s got a lot of common idioms around it:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>myValue, err := doThing() if err != nil { return -1, fmt.Errorf(“unable to doThing %v”, err) } </code></pre></div></div> <p>Needless to say this can get very wordy when dealing with deeply nested exceptions or when you’re interacting a lot with external systems. It is definitely a mind shift if you’re used to the throwing exceptions wherever and have one single place to catch all exceptions where they’re handled appropriately.</p> <h3 id="my-advice-6">My Advice</h3> <p>I’ll be honest I never totally made my peace with this. I had good training from experienced opensource contributors to major Go projects, read all the right blog posts, definitely felt like I’d heard enough from the community on why the current state of Go error handling was great in their opinions, but the lack of stack traces was a deal breaker for me.</p> <p>On the positive side, I found Dave Cheney’s advice on error handling to be the most practical and he wrote <a href="https://github.com/pkg/errors">a package</a> containing a lot of that advice, we found it invaluable as it provided those stack traces we all missed but you had to remember to use it.</p> <h2 id="summary">Summary</h2> <p>A lot of people really love Go and are very productive with it, I just was never one of those people and that’s ok. However, I think the advice in this post is reasonably sound and uncontroversial. So, if you find yourself needing to write some code it Go, give this guide a quick perusal and you’ll waste a lot less time than I did getting productive in developing applications in Go.</p> Raspberry Pi Kubernetes Cluster - Part 2 https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-2/ Jason Meridth urn:uuid:0aef121f-48bd-476f-e09d-4ca0aa2ac602 Thu, 03 May 2018 02:13:07 +0000 <p>Howdy again.</p> <p>Alright, my 8 port switch showed up so I was able to connect my raspberry 3B+ boards to my home network. I plugged it in with 6 1ft CAT5 cables I had in my catch-all box that all of us nerds have. I’d highly suggest flexible CAT 6 cables instead if you can get them, like <a href="https://www.amazon.com/Cat-Ethernet-Cable-Black-Connectors/dp/B01IQWGKQ6">here</a>. I ordered them and they showed up before I finished this post, so I am using the CAT6 cables.</p> <!--more--> <p>The IP addresses they will receive initialy from my home router via DHCP can be determined with nmap. Lets imagine my subnet is 192.168.1.0/24.</p> <p>Once I got them on the network I did the following:</p> <script src="https://gist.github.com/64e7b08729ffe779f77a7bda0221c6e9.js"> </script> <h3 id="install-raspberrian-os-on-sd-cards">Install Raspberrian OS On SD Cards</h3> <p>You can get the Raspberry Pi Stretch Lite OS from <a href="https://www.raspberrypi.org/downloads/raspbian/">here</a>.</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/raspberry_pi_stretch_lite.png" alt="Raspberry Pi Stretch Lite" /></p> <p>Then use the <a href="https://etcher.io/">Etcher</a> tool to install it to each of the 6 SD cards.</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/etcher.gif" alt="Etcher" /></p> <h4 id="important">IMPORTANT</h4> <p>Before putting the cards into the Raspberry Pis you need to add a <code class="highlighter-rouge">ssh</code> folder to the root of the SD cards. This will allow you to ssh to each Raspberry Pi with default credentials (username: <code class="highlighter-rouge">pi</code> and password <code class="highlighter-rouge">raspberry</code>). Example: <code class="highlighter-rouge">ssh pi@raspberry_pi_ip</code> where <code class="highlighter-rouge">raspberry_ip</code> is obtained from the nmap command above.</p> <p>Next post will be setting up kubernetes. Thank you for reading.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-2/">Raspberry Pi Kubernetes Cluster - Part 2</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on May 02, 2018.</p> Multi-Environment Deployments with React https://derikwhittaker.blog/2018/04/10/multi-environment-deployments-with-react/ Maintainer of Code, pusher of bits… urn:uuid:4c0ae985-09ac-6d2e-0429-addea1632ea3 Tue, 10 Apr 2018 12:54:17 +0000 If you are using Create-React-App to scaffold your react application there is built in support for changing environment variables based on the NODE_ENV values, this is done by using .env files.  In short this process works by having a .env, .env.production, .env.development set of files.  When you run/build your application CRA will set the NODE_ENV value &#8230; <p><a href="https://derikwhittaker.blog/2018/04/10/multi-environment-deployments-with-react/" class="more-link">Continue reading <span class="screen-reader-text">Multi-Environment Deployments with&#160;React</span></a></p> <p>If you are using <a href="https://github.com/facebook/create-react-app" target="_blank" rel="noopener">Create-React-App</a> to scaffold your react application there is <a href="https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/template/README.md#adding-development-environment-variables-in-env" target="_blank" rel="noopener">built in support</a> for changing environment variables based on the NODE_ENV values, this is done by using .env files.  In short this process works by having a .env, .env.production, .env.development set of files.  When you run/build your application <a href="https://github.com/facebook/create-react-app" target="_blank" rel="noopener">CRA</a> will set the NODE_ENV value to either <code>development</code> or <code>production</code> and based on these values the correct .env file will be used.</p> <p>This works great, when you have a simple deploy setup. But many times in enterprise level applications you need support for more than just 2 environments, many times it is 3-4 environments.  Common logic would suggest that you can accomplish this via the built in mechanism by having additional .env files and changing the NODE_ENV value to the value you care about.  However, CRA does not support this with doing an <code>eject</code>, which will eject all the default conventions and leave it to you to configure your React application.  Maybe this is a good idea, but in my case ejecting was not something I wanted to do.</p> <p>Because I did not want to do an <code>eject</code> I needed to find another solution, and after a fair amount of searching I found a solution that seems to work for me and my needs and is about the amount of effort as I wanted <img src="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f642.png" alt=" Raspberry Pi Kubernetes Cluster - Part 1 https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-1/ Jason Meridth urn:uuid:bd3470f6-97d5-5028-cf12-0751f90915c3 Sat, 07 Apr 2018 14:01:00 +0000 <p>Howdy</p> <p>This is going to be the first post about my setup of a Raspberry Pi Kubernetes Cluster. I saw a post by <a href="https://harthoover.com/kubernetes-1.9-on-a-raspberry-pi-cluster/">Hart Hoover</a> and it finally motivated me to purchase his “grocery list” and do this finally. I’ve been using <a href="https://kubernetes.io/docs/getting-started-guides/minikube/">Minikube</a> for local Kubernetes testing but it doesn’t give you multi-host testing abilities. I’ve also been wanting to get deeper into my Raspberry Pi knowledge. Lots of learning and winning.</p> <p>The items I bought were:</p> <ul> <li>Six <a href="https://smile.amazon.com/dp/B07BFH96M3">Raspberry Pi 3 Model B+ Motherboards</a></li> <li>Six <a href="https://smile.amazon.com/gp/product/B010Q57T02/">SanDisk Ultra 32GB microSDHC UHS-I Card with Adapter, Grey/Red, Standard Packaging (SDSQUNC-032G-GN6MA)</a></li> <li>One <a href="https://smile.amazon.com/gp/product/B011KLFERG/ref=oh_aui_detailpage_o02_s01?ie=UTF8&amp;psc=1">Sabrent 6-Pack 22AWG Premium 3ft Micro USB Cables High Speed USB 2.0 A Male to Micro B Sync and Charge Cables Black CB-UM63</a></li> <li>One <a href="https://smile.amazon.com/gp/product/B01L0KN8OS/ref=oh_aui_detailpage_o02_s01?ie=UTF8&amp;psc=1">AmazonBasics 6-Port USB Wall Charger (60-Watt) - Black</a></li> <li>One <a href="https://smile.amazon.com/gp/product/B01D9130QC/ref=oh_aui_detailpage_o02_s00?ie=UTF8&amp;psc=1">GeauxRobot Raspberry Pi 3 Model B 6-layer Dog Bone Stack Clear Case Box Enclosure also for Pi 2B B+ A+ B A</a></li> <li>One <a href="http://amzn.to/2gNzLzi">Black Box 8-Port Switch</a></li> </ul> <p>Here is the tweet when it all arrived:</p> <div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet"><p lang="en" dir="ltr">I blame <a href="https://twitter.com/hhoover?ref_src=twsrc%5Etfw">@hhoover</a> ;). I will be building my <a href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">@kubernetesio</a> cluster once the 6pi case shows up next Wednesday. The extra pi is to upgrade my <a href="https://twitter.com/RetroPieProject?ref_src=twsrc%5Etfw">@RetroPieProject</a>. Touch screen is an addition I want to try. Side project here I come. <a href="https://t.co/EebIKbsCeH">pic.twitter.com/EebIKbsCeH</a></p>&mdash; Jason Meridth (@jmeridth) <a href="https://twitter.com/jmeridth/status/980075584725422080?ref_src=twsrc%5Etfw">March 31, 2018</a></blockquote> <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> </div> <p>I spent this morning finally putting it together.</p> <p>Here is me getting started on the “dogbone case” to hold all of the Raspberry Pis:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_2.jpg" alt="The layout" /></p> <p>The bottom and one layer above:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_3.jpg" alt="The bottom and one layer above" /></p> <p>And the rest:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_4.jpg" alt="3 Layers" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_11.jpg" alt="4 Layers" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_12.jpg" alt="5 Layers" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_13.jpg" alt="6 Layers and Finished" /></p> <p>Different angles completed:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_14.jpg" alt="Finished Angle 2" /></p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_15.jpg" alt="Finished Angle 3" /></p> <p>And connect the power:</p> <p><img src="https://blog.jasonmeridth.com/images/kubernetes_cluster/case_16.jpg" alt="Power" /></p> <p>Next post will be on getting the 6 sandisk cards ready and putting them in and watching the Raspberry Pis boot up and get a green light. Stay tuned.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/raspberry-pi-kubernetes-cluster-part-1/">Raspberry Pi Kubernetes Cluster - Part 1</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on April 07, 2018.</p> Building AWS Infrastructure with Terraform: S3 Bucket Creation https://derikwhittaker.blog/2018/04/06/building-aws-infrastructure-with-terraform-s3-bucket-creation/ Maintainer of Code, pusher of bits… urn:uuid:cb649524-d882-220f-c253-406a54762705 Fri, 06 Apr 2018 14:28:49 +0000 If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure.  In the AWS word you can use the native CloudFormation solution, but honestly I find this painful and the docs very lacking.  Personally, I prefer Terraform by Hashicorp.  In my experience &#8230; <p><a href="https://derikwhittaker.blog/2018/04/06/building-aws-infrastructure-with-terraform-s3-bucket-creation/" class="more-link">Continue reading <span class="screen-reader-text">Building AWS Infrastructure with Terraform: S3 Bucket&#160;Creation</span></a></p> <p>If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure.  In the AWS word you can use the native <a href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=DChcSEwjD-Lry6KXaAhUMuMAKHTB8AYwYABAAGgJpbQ&amp;ohost=www.google.com&amp;cid=CAESQeD2aF3IUBPQj5YF9K0xmz0FNtIhnq3PzYAHFV6dMZVIirR_psuXDSgkzxZ0jXoyWfpECufNNfbp7JzHQ73TTrQH&amp;sig=AOD64_1b_L781SLpKXqLTFFYIk5Zv3BcHA&amp;q=&amp;ved=0ahUKEwi1l7Hy6KXaAhWD24MKHQXSCQ0Q0QwIJw&amp;adurl=" target="_blank" rel="noopener">CloudFormation</a> solution, but honestly I find this painful and the docs very lacking.  Personally, I prefer <a href="https://www.terraform.io/" target="_blank" rel="noopener">Terraform</a> by <a href="https://www.hashicorp.com/" target="_blank" rel="noopener">Hashicorp</a>.  In my experience the simplicity and easy of use, not to mention the stellar documentation make this the product of choice.</p> <p>This is the initial post in what I hope to be a series of post about how to use Terraform to setup/build AWS Infrastructure.</p> <p>Terrform Documentation on S3 Creation -&gt; <a href="https://www.terraform.io/docs/providers/aws/d/s3_bucket.html" target="_blank" rel="noopener">Here</a></p> <p>In this post I will cover 2 things</p> <ol> <li>Basic bucket setup</li> <li>Bucket setup as Static website</li> </ol> <p>Setting up a basic bucket we can use the following</p> <div class="code-snippet"> <pre class="code-content">resource "aws_s3_bucket" "my-bucket" { bucket = "my-bucket" acl = "private" tags { Any_Tag_Name = "Tag value for tracking" } } </pre> </div> <p>When looking at the example above the only 2 values that are required are bucket and acl.</p> <p>I have added the use of Tags to show you can add custom tags to your bucket</p> <p>Another way to setup an S3 bucket is to act as a Static Web Host.   Setting this up takes a bit more configuration, but not a ton.</p> <div class="code-snippet"> <pre class="code-content">resource "aws_s3_bucket" "my-website-bucket" { bucket = "my-website-bucket" acl = "public-read" website { index_document = "index.html" error_document = "index.html" } policy = &lt;&lt;POLICY { "Version": "2012-10-17", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-website-bucket/*" } ] } POLICY tags { Any_Tag_Name = "Tag value for tracking" } } </pre> </div> <p>The example above has 2 things that need to be pointed out.</p> <ol> <li>The website settings.  Make sure you setup the correct pages here for index/error</li> </ol> <p>The Policy settings.  Here I am using just basic policy.  You can of course setup any policy here you want/need.</p> <p>As you can see, setting up S3 buckets is very simple and straight forward.</p> <p><strong><em>*** Reminder: S3 bucket names MUST be globally unique ***</em></strong></p> <p>Till next time,</p> SSH - Too Many Authentication Failures https://blog.jasonmeridth.com/posts/ssh-too-many-authentication-failures/ Jason Meridth urn:uuid:d7fc1034-1798-d75e-1d61-84fac635dda4 Wed, 28 Mar 2018 05:00:00 +0000 <h1 id="problem">Problem</h1> <p>I started seeing this error recently and had brain farted on why.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">Received disconnect from 123.123.132.132: Too many authentication failures <span class="k">for </span>hostname</code></pre></figure> <p>After a bit of googling it came back to me. This is because I’ve loaded too many keys into my ssh-agent locally (<code class="highlighter-rouge">ssh-add</code>). Why did you do that? Well, because it is easier than specifying the <code class="highlighter-rouge">IdentityFile</code> on the cli when trying to connect. But there is a threshhold. This is set by the ssh host by the <code class="highlighter-rouge">MaxAuthTries</code> setting in <code class="highlighter-rouge">/etc/ssh/sshd_config</code>. The default is 6.</p> <h1 id="solution-1">Solution 1</h1> <p>Clean up the keys in your ssh-agent.</p> <p><code class="highlighter-rouge">ssh-add -l</code> lists all the keys you have in your ssh-agent <code class="highlighter-rouge">ssh-add -d key</code> deletes the key from your ssh-agent</p> <h1 id="solution-2">Solution 2</h1> <p>You can solve this on the command line like this:</p> <p><code class="highlighter-rouge">ssh -o IdentitiesOnly=yes -i ~/.ssh/example_rsa foo.example.com</code></p> <p>What is IdentitiesOnly? Explained in Solution 3 below.</p> <h1 id="solution-3-best">Solution 3 (best)</h1> <p>Specifiy, explicitly, which key goes to which host(s) in your <code class="highlighter-rouge">.ssh/config</code> file.</p> <p>You need to configure which key (“IdentityFile”) goes with which domain (or host). You also want to handle the case when the specified key doesn’t work, which would usually be because the public key isn’t in ~/.ssh/authorized_keys on the server. The default is for SSH to then try any other keys it has access to, which takes us back to too many attempts. Setting “IdentitiesOnly” to “yes” tells SSH to only try the specified key and, if that fails, fall through to password authentication (presuming the server allows it).</p> <p>Your ~/.ssh/config would look like:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host *.myhost.com IdentitiesOnly yes IdentityFile ~/.ssh/myhost Host secure.myhost.com IdentitiesOnly yes IdentityFile ~/.ssh/mysecurehost_rsa Host *.myotherhost.domain IdentitiesOnly yes IdentityFile ~/.ssh/myotherhost_rsa </code></pre></div></div> <p><code class="highlighter-rouge">Host</code> is the host the key can connect to <code class="highlighter-rouge">IdentitiesOnly</code> means only to try <em>this</em> specific key to connect, no others <code class="highlighter-rouge">IdentityFile</code> is the path to the key</p> <p>You can try multiple keys if needed</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host *.myhost.com IdentitiesOnly yes IdentityFile ~/.ssh/myhost_rsa IdentityFile ~/.ssh/myhost_dsa </code></pre></div></div> <p>Hope this helps someone else.</p> <p>Cheers!</p> <p><a href="https://blog.jasonmeridth.com/posts/ssh-too-many-authentication-failures/">SSH - Too Many Authentication Failures</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on March 28, 2018.</p> Clear DNS Cache In Chrome https://blog.jasonmeridth.com/posts/clear-dns-cache-in-chrome/ Jason Meridth urn:uuid:6a2c8c0b-c91b-5f7d-dbc7-8065f0a2f1fd Tue, 27 Mar 2018 20:42:00 +0000 <p>I’m blogging this because I keep forgetting how to do it. Yeah, yeah, I can google it. I run this blog so I know it is always available…..anywho.</p> <p>Go To:</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chrome://net-internals/#dns </code></pre></div></div> <p>Click “Clear host cache” button</p> <p><img src="https://blog.jasonmeridth.com/images/clear_dns_cache_in_chrome.png" alt="clear_dns_cache_in_chrome" /></p> <p>Hope this helps someone else.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/clear-dns-cache-in-chrome/">Clear DNS Cache In Chrome</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on March 27, 2018.</p> Create Docker Container from Errored Container https://blog.jasonmeridth.com/posts/create-docker-container-from-errored-container/ Jason Meridth urn:uuid:33d5a6b5-4c48-ae06-deb6-a505edc6b427 Mon, 26 Mar 2018 03:31:00 +0000 <p>When I’m trying to “dockerize” an applciation I usually have to work through some wonkiness.</p> <p>To diagnose a container that has errored out, I, obviously, look at the logs via <code class="highlighter-rouge">docker logs -f [container_name]</code>. That is sometimes helpful. It will, at minimum tell me where I need to focus on the new container I’m going to create.</p> <p><img src="https://blog.jasonmeridth.com/images/diagnose.jpg" alt="diagnose" /></p> <p>Pre-requisites to being able to build a diagnosis container:</p> <ul> <li>You need to use <code class="highlighter-rouge">CMD</code>, <em>not</em> <code class="highlighter-rouge">ENTRYPOINT</code> in the Dockerfile <ul> <li>with <code class="highlighter-rouge">CMD</code> you’ll be able to start a shell, with <code class="highlighter-rouge">ENTRYPOINT</code> your diagnosis container will just keep trying to run that</li> </ul> </li> </ul> <p>To create a diagnosis container, do the following:</p> <ul> <li>Check your failed container ID by <code class="highlighter-rouge">docker ps -a</code></li> <li>Create docker image form the container with <code class="highlighter-rouge">docker commit</code> <ul> <li>example: <code class="highlighter-rouge">docker commit -m "diagnosis" [failed container id]</code></li> </ul> </li> <li>Check the newly create docker image ID by <code class="highlighter-rouge">docker images</code></li> <li><code class="highlighter-rouge">docker run -it [new container image id] sh</code> <ul> <li>this takes you into a container immediately after the error occurred.</li> </ul> </li> </ul> <p>Hope this helps someone else.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/create-docker-container-from-errored-container/">Create Docker Container from Errored Container</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on March 25, 2018.</p> Log Early, Log Often… Saved my butt today https://derikwhittaker.blog/2018/03/21/log-early-log-often-saved-my-butt-today/ Maintainer of Code, pusher of bits… urn:uuid:395d9800-e7ce-27fd-3fc1-5e68628bc161 Wed, 21 Mar 2018 13:16:03 +0000 In a prior posting (AWS Lambda:Log Early Log often, Log EVERYTHING) I wrote about the virtues and value about having really in depth logging, especially when working with cloud services.  Well today this logging saved my ASS a ton of detective work. Little Background I have a background job (Lambda that is called on a schedule) &#8230; <p><a href="https://derikwhittaker.blog/2018/03/21/log-early-log-often-saved-my-butt-today/" class="more-link">Continue reading <span class="screen-reader-text">Log Early, Log Often&#8230; Saved my butt&#160;today</span></a></p> <p>In a prior <a href="https://derikwhittaker.blog/2018/03/06/aws-lambda-log-early-log-often-log-everything/" target="_blank" rel="noopener">posting (AWS Lambda:Log Early Log often, Log EVERYTHING)</a> I wrote about the virtues and value about having really in depth logging, especially when working with cloud services.  Well today this logging saved my ASS a ton of detective work.</p> <p><strong>Little Background</strong><br /> I have a background job (Lambda that is called on a schedule) to create/update data cache in a <a href="https://aws.amazon.com/dynamodb/" target="_blank" rel="noopener">DynamoDB</a> table.  Basically this job will pull data from one data source and attempt to push it as create/update/delete to our Dynamo table.</p> <p>Today when I was running our application I noticed things were not loading right, in fact I had javascript errors because of null reference errors.  I knew that the issue had to be in our data, but was not sure what was wrong.  If I had not had a ton of logging (debug and info) I would have had to run our code locally and step though/debug code for hundreds of items of data.</p> <p>However, because of in depth logging I was able to quickly go to <a href="https://aws.amazon.com/cloudwatch/" target="_blank" rel="noopener">CloudWatch</a> and filter on a few key words and narrow hundreds/thousands of log entries down to 5.  Once I had these 5 entries I was able to expand a few of those entries and found the error within seconds.</p> <p>Total time to find the error was less than 5 minutes and I never opened a code editor or stepped into code.</p> <p>The moral of this story, because I log everything, including data (no PII of course) I was able to quickly find the source of the error.  Now to fix the code&#8230;.</p> <p>Till next time,</p> AWS Lambda: Log early, Log often, Log EVERYTHING https://derikwhittaker.blog/2018/03/06/aws-lambda-log-early-log-often-log-everything/ Maintainer of Code, pusher of bits… urn:uuid:6ee7f59b-7f4c-1312-bfff-3f9c46ec8701 Tue, 06 Mar 2018 14:00:58 +0000 In the world of building client/server applications logs are important.  They are helpful when trying to see what is going on in your application.  I have always held the belief  that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code. But lets &#8230; <p><a href="https://derikwhittaker.blog/2018/03/06/aws-lambda-log-early-log-often-log-everything/" class="more-link">Continue reading <span class="screen-reader-text">AWS Lambda: Log early, Log often, Log&#160;EVERYTHING</span></a></p> <p>In the world of building client/server applications logs are important.  They are helpful when trying to see what is going on in your application.  I have always held the belief  that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code.</p> <p>But lets be honest, in most cases when building client/server applications logs are an afterthought.  Often this is because you can pretty easily (in most cases) debug your application and step through the code.</p> <p>When building a <a href="https://aws.amazon.com/serverless/" target="_blank" rel="noopener">serverless</a> applications with technologies like <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener">AWS Lambda</a> functions (holds true for Azure Functions as well) your logging game really needs to step up.</p> <p>The reason for this is that you cannot really debug your Lambda in the wild (you can to some degree locally with AWS SAM or the Serverless framework).  Because of this you need produce detailed enough logs to allow you to easily determine the WHAT and WHERE.</p> <p>When I build my serverless functions I have a few guidelines I follow</p> <ol> <li>Info Log calls to methods, output argument data (make sure no <a href="https://en.wikipedia.org/wiki/Personally_identifiable_information" target="_blank" rel="noopener">PII</a>/<a href="https://en.wikipedia.org/wiki/Protected_health_information" target="_blank" rel="noopener">PHI</a>)</li> <li>Error Log any failures (in try/catch or .catch for promises)</li> <li>Debug Log any critical decision points</li> <li>Info Log exit calls at top level methods</li> </ol> <p>I also like to setup a simple and consistent format for my logs.  The example I follow for my Lambda logs is as seen below</p> <div class="code-snippet"> <pre class="code-content">timestamp: [logLevel] : [Class.Method] - message {data points} </pre> </div> <p>I have found that if I follow these general guidelines the pain of determine failure points in serverless environments is heavily reduced.</p> <p>Till next time,</p> Sinon Error: Attempted to wrap undefined property ‘XYZ as function https://derikwhittaker.blog/2018/02/27/sinon-error-attempted-to-wrap-undefined-property-xyz-as-function/ Maintainer of Code, pusher of bits… urn:uuid:b41dbd54-3804-6f6d-23dc-d2a04635033a Tue, 27 Feb 2018 13:45:29 +0000 I ran into a fun little error recently when working on a ReactJs application.  In my application I was using SinonJs to setup some spies on a method, I wanted to capture the input arguments for verification.  However, when I ran my test I received the following error. Attempted to wrap undefined property handlOnAccountFilter as &#8230; <p><a href="https://derikwhittaker.blog/2018/02/27/sinon-error-attempted-to-wrap-undefined-property-xyz-as-function/" class="more-link">Continue reading <span class="screen-reader-text">Sinon Error: Attempted to wrap undefined property &#8216;XYZ as&#160;function</span></a></p> <p>I ran into a fun little error recently when working on a <a href="https://reactjs.org/" target="_blank" rel="noopener">ReactJs</a> application.  In my application I was using <a href="http://sinonjs.org/" target="_blank" rel="noopener">SinonJs</a> to setup some spies on a method, I wanted to capture the input arguments for verification.  However, when I ran my test I received the following error.</p> <blockquote><p>Attempted to wrap undefined property handlOnAccountFilter as function</p></blockquote> <p>My method under test is setup as such</p> <div class="code-snippet"> <pre class="code-content">handleOnAccountFilter = (filterModel) =&gt; { // logic here } </pre> </div> <p>I was using the above syntax is the <a href="https://github.com/jeffmo/es-class-public-fields" target="_blank" rel="noopener">proposed class property</a> feature, which will automatically bind the <code>this</code> context of the class to my method.</p> <p>My sinon spy is setup as such</p> <div class="code-snippet"> <pre class="code-content">let handleOnAccountFilterSpy = null; beforeEach(() =&gt; { handleOnAccountFilterSpy = sinon.spy(AccountsListingPage.prototype, 'handleOnAccountFilter'); }); afterEach(() =&gt; { handleOnAccountFilterSpy.restore(); }); </pre> </div> <p>Everything looked right, but I was still getting this error.  It turns out that this error is due in part in the way that the Class Property feature implements the handlOnAccountFilter.  When you use this feature the method/property is added to the class as an instance method/property, not as a prototype method/property.  This means that sinon is not able to gain access to it prior to creating an instance of the class.</p> <p>To solve my issue I had to make a change in the implementation to the following</p> <div class="code-snippet"> <pre class="code-content">handleOnAccountFilter(filterModel) { // logic here } </pre> </div> <p>After make the above change I needed to determine how I wanted to bind <code>this</code> to my method (Cory show 5 ways to do this <a href="https://medium.freecodecamp.org/react-binding-patterns-5-approaches-for-handling-this-92c651b5af56" target="_blank" rel="noopener">here</a>).  I chose to bind <code>this</code> inside the constructor as below</p> <div class="code-snippet"> <pre class="code-content">constructor(props){ super(props); this.handleOnAccountFilter = this.handleOnAccountFilter.bind(this); } </pre> </div> <p>I am not a huge fan of having to do this (pun intended), but oh well.  This solved my issues.</p> <p>Till next time</p> Ensuring componentDidMount is not called in Unit Tests https://derikwhittaker.blog/2018/02/22/ensuring-componentdidmount-is-not-called-in-unit-tests/ Maintainer of Code, pusher of bits… urn:uuid:da94c1a3-2de4-a90c-97f5-d7361397a33c Thu, 22 Feb 2018 19:45:53 +0000 If you are building a ReactJs you will often times implement componentDidMount on your components.  This is very handy at runtime, but can pose an issue for unit tests. If you are building tests for your React app you are very likely using enzyme to create instances of your component.  The issue is that when enzyme creates &#8230; <p><a href="https://derikwhittaker.blog/2018/02/22/ensuring-componentdidmount-is-not-called-in-unit-tests/" class="more-link">Continue reading <span class="screen-reader-text">Ensuring componentDidMount is not called in Unit&#160;Tests</span></a></p> <p>If you are building a <a href="https://reactjs.org/" target="_blank" rel="noopener">ReactJs</a> you will often times implement <code>componentDidMount</code> on your components.  This is very handy at runtime, but can pose an issue for unit tests.</p> <p>If you are building tests for your React app you are very likely using <a href="http://airbnb.io/projects/enzyme/" target="_blank" rel="noopener">enzyme</a> to create instances of your component.  The issue is that when enzyme creates the component it invokes the lifecyle methods, like <code>componentDidMount</code>.  Sometimes we do not want this to be called, but how to suppress this?</p> <p>I have found 2 different ways to suppress/mock <code>componentDidMount</code>.</p> <p>Method one is to redefine <code>componentDidMount</code> on your component for your tests.  This could have interesting side effects so use with caution.</p> <div class="code-snippet"> <pre class="code-content"> describe('UsefullNameHere', () =&gt; { beforeAll(() =&gt; { YourComponent.prototype.componentDidMount = () =&gt; { // can omit or add custom logic }; }); }); </pre> </div> <p>Basically above I am just redefining the componentDidMount method on my component.  This works and allows you to have custom logic.  Be aware that when doing above you will have changed the implementation for your component for the lifetime of your test session.</p> <p>Another solution is to use a mocking framework like <a href="http://sinonjs.org/" target="_blank" rel="noopener">SinonJs</a>.  With Sinon you can stub out the <code>componentDidMount</code> implementation as seen below</p> <div class="code-snippet"> <pre class="code-content"> describe('UsefullNameHere', () =&gt; { let componentDidMountStub = null; beforeAll(() =&gt; { componentDidMountStub = sinon.stub(YourComponent.prototype, 'componentDidMount').callsFake(function() { // can omit or add custom logic }); }); afterAll(() =&gt; { componentDidMountStub.restore(); }); }); </pre> </div> <p>Above I am using .stub to redefine the method.  I also added .<a href="http://sinonjs.org/releases/v4.3.0/stubs/" target="_blank" rel="noopener">callsFake</a>() but this can be omitted if you just want to ignore the call.  You will want to make sure you restore your stub via the afterAll, otherwise you will have stubbed out the call for the lifetime of your test session.</p> <p>Till next time,</p> Los Techies Welcomes Derik Whittaker https://lostechies.com/derekgreer/2018/02/21/los-techies-welcomes-derik-whittaker/ Los Techies urn:uuid:adc9a1c8-48ea-3bea-1aa7-320d51db12a1 Wed, 21 Feb 2018 11:00:00 +0000 Los Techies would like to introduce, and extend a welcome to Derik Whittaker. Derik is a C# MVP, member of the AspInsiders group, community speaker, and Pluralsight author. Derik was previously a contributor at CodeBetter.com. Welcome, Derik! <p>Los Techies would like to introduce, and extend a welcome to Derik Whittaker. Derik is a C# MVP, member of the AspInsiders group, community speaker, and Pluralsight author. Derik was previously a contributor at <a href="http://codebetter.com/">CodeBetter.com</a>. Welcome, Derik!</p> Ditch the Repository Pattern Already https://lostechies.com/derekgreer/2018/02/20/ditch-the-repository-pattern-already/ Los Techies urn:uuid:7fab2063-d833-60ce-9e46-e4a413ec8391 Tue, 20 Feb 2018 21:00:00 +0000 One pattern that still seems particularly common among .Net developers is the Repository pattern. I began using this pattern with NHibernate around 2006 and only abandoned its use a few years ago. <p>One pattern that still seems particularly common among .Net developers is the <a href="https://martinfowler.com/eaaCatalog/repository.html">Repository pattern.</a> I began using this pattern with NHibernate around 2006 and only abandoned its use a few years ago.</p> <p>I had read several articles over the years advocating abandoning the Repository pattern in favor of other suggested approaches which served as a pebble in my shoe for a few years, but there were a few design principles whose application seemed to keep motivating me to use the pattern.  It wasn’t until a change of tooling and a shift in thinking about how these principles should be applied that I finally felt comfortable ditching the use of repositories, so I thought I’d recount my journey to provide some food for thought for those who still feel compelled to use the pattern.</p> <h2 id="mental-obstacle-1-testing-isolation">Mental Obstacle 1: Testing Isolation</h2> <p>What I remember being the biggest barrier to moving away from the use of repositories was writing tests for components which interacted with the database.  About a year or so before I actually abandoned use of the pattern, I remember trying to stub out a class derived from Entity Framework’s DbContext after reading an anti-repository blog post.  I don’t remember the details now, but I remember it being painful and even exploring use of a 3rd-party library designed to help write tests for components dependent upon Entity Framework.  I gave up after a while, concluding it just wasn’t worth the effort.  It wasn’t as if my previous approach was pain-free, as at that point I was accustomed to stubbing out particularly complex repository method calls, but as with many things we often don’t notice friction to which we’ve become accustomed for one reason or another.  I had assumed that doing all that work to stub out my repositories was what I should be doing.</p> <p>Another principle that I picked up from somewhere (maybe the big <a href="http://xunitpatterns.com/">xUnit Test Patterns</a> book? … I don’t remember) that seemed to keep me bound to my repositories was that <a href="http://aspiringcraftsman.com/2012/04/01/tdd-best-practices-dont-mock-others/">you shouldn’t write tests that depend upon dependencies you don’t own</a>.  I believed at the time that I should be writing tests for Application Layer services (which later morphed into discrete dispatched command handlers) and the idea of stubbing out either NHIbernate or Entity Framework violated my sensibilities.</p> <h2 id="mental-obstacle-2-the-dependency-inversion-principle-adherence">Mental Obstacle 2: The Dependency Inversion Principle Adherence</h2> <p>The Dependency Inversion Principle seems to be a source of confusion for many which stems in part from the similarity of wording with the practice of <a href="https://lostechies.com/derickbailey/2011/09/22/dependency-injection-is-not-the-same-as-the-dependency-inversion-principle/">Dependency Injection</a> as well as from the fact that the pattern’s formal definition reflects the platform from whence the principle was conceived (i.e. C++).  One might say that the abstract definition of the Dependency Inversion Principle was too dependent upon the details of its origin (ba dum tss).  I’ve written about the principle a few times (perhaps my most succinct being <a href="https://stackoverflow.com/a/1113937/1219618">this Stack Overflow answer</a>), but put simply, the Dependency Inversion Principle has at its primary goal the decoupling of the portions of your application which define <i>policy</i> from the portions which define <i>implementation</i>.  That is to say, this principle seeks to keep the portions of your application which govern what your application does (e.g. workflow, business logic, etc.) from being tightly coupled to the portions of your application which govern the low level details of how it gets done (e.g. persistence to an Sql Server database, use of Redis for caching, etc.).</p> <p>A good example of a violation of this principle, which I recall from my NHibernate days, was that once upon a time NHibernate was tightly coupled to log4net.  This was later corrected, but at one time the NHibernate assembly had a hard dependency on log4net.  You could use a different logging library for your own code if you wanted, and you could use binding redirects to use a different version of log4net if you wanted, but at one time if you had a dependency on NHibernate then you had to deploy the log4net library.  I think this went unnoticed by many due to the fact that most developers who used NHibernate also used log4net.</p> <p>When I first learned about the principle, I immediately recognized that it seemed to have limited advertized value for most business applications in light of what Udi Dahan labeled<a href="http://udidahan.com/2009/06/07/the-fallacy-of-reuse/"> The Fallacy Of ReUse</a>.  That is to say, <i>properly understood</i>, the Dependency Inversion Principle has as its primary goal the reuse of components and keeping those components decoupled from dependencies which would keep them from being easily reused with other implementation components, but your application and business logic isn’t something that is likely to ever be reused in a different context.  The take away from that is basically that the advertized value of adhering to the Dependency Inversion Principle is really more applicable to libraries like NHibernate, Automapper, etc. and not so much to that workflow your team built for Acme Inc.’s distribution system.  Nevertheless, the Dependency Inversion Principle had a practical value of implementing an architecture style Jeffrey Palermo labeled <a href="http://jeffreypalermo.com/blog/the-onion-architecture-part-1/">the Onion Architecture.</a> Specifically, in contrast to <a href="https://msdn.microsoft.com/en-us/library/ff650258.aspx"> traditional 3-layered architecture models</a> where UI, Business, and Data Access layers precluded using something like <a href="https://msdn.microsoft.com/en-us/library/ff648105.aspx?f=255&amp;MSPPError=-2147217396">Data Access Logic Components</a> to encapsulate an ORM to map data directly to entities within the Business Layer, inverting the dependencies between the Business Layer and the Data Access layer provided the ability for the application to interact with the database while also <i>seemingly </i>abstracting away the details of the data access technology used.</p> <p>While I always saw the fallacy in strictly trying to apply the Dependency Inversion Principle to invert the implementation details of how I got my data from my application layer so that I’d someday be able to use the application in a completely different context, it seemed the academically astute and in vogue way of doing Domain-driven Design at the time, seemed consistent with the GoF’s advice to program to an interface rather than an implementation, and provided an easier way to write isolation tests than trying to partially stub out ORM types.</p> <h2 id="the-catalyst">The Catalyst</h2> <p>For the longest time, I resisted using Entity Framework.  I had become fairly proficient at using NHibernate and I just saw it as plain stupid to use a framework that was years behind NHibernate in features and maturity, especially when it had such a steep learning curve.  A combination of things happened, though.  A lot of the NHibernate supporters (like many within the Alt.Net crowd) moved on to other platforms like Ruby and Node; anything with Microsoft’s name on it eventually seems to gain market share whether it’s better or not; and Entity Framework eventually did seem to mostly catch up with NHibernate in features, and surpassed it in some areas. So, eventually I found it impossible to avoid using Entity Framework which led to me trying to apply the same patterns I’d used before with this newer-to-me framework.</p> <p>To be honest, everything mostly worked, especially for the really simple stuff.  Eventually, though, I began to see little ways I had to modify my abstraction to accommodate differences in how Entity Framework did things from how NHibernate did things.  What I discovered was that, while my repositories allowed my application code to be physically decoupled from the ORM, the way I was using the repositories was in small ways semantically coupled to the framework.  I wish I had kept some sort of record every time I ran into something, as the only real thing I can recall now were motivations with certain design approaches to expose the SaveChanges method for <a href="https://lostechies.com/derekgreer/2015/11/01/survey-of-entity-framework-unit-of-work-patterns/"> Unit of Work implementations</a> I don’t want to make more of the semantic coupling argument against repositories than it’s worth, but observing little places where <a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">my abstractions were leaking</a>, combined with the pebble in my shoe of developers who I felt were far better than me were saying I shouldn’t use them lead me to begin rethinking things.</p> <h2 id="more-effective-testing-strategies">More Effective Testing Strategies</h2> <p>It was actually a few years before I stopped using repositories that I stopped stubbing out repositories.  Around 2010, I learned that you can use Test-Driven Development to achieve 100% test coverage for the code for which you’re responsible, but when you plug your code in for the first time with that team that wasn’t designing to the same specification and not writing any tests at all that things may not work.  It was then that I got turned on to Acceptance Test Driven Development.  What I found was that writing high-level subcutaneous tests (i.e. skipping the UI layer, but otherwise end-to-end) was overall easier, was possible to align with acceptance criteria contained within a user story, provided more assurance that everything worked as a whole, and was easier to get teams on board with.  Later on, I surmised that I really shouldn’t have been writing isolation tests for components which, for the most part, are just specialized facades anyway.  All an isolation test for a facade really says is “did I delegate this operation correctly” and if you’re not careful you can end up just writing a whole bunch of tests that basically just validate whether you correctly configured your mocking library.</p> <p>So, by the time I started rethinking my use of repositories, I had long since stopped using them for test isolation.</p> <h2 id="taking-the-plunge">Taking the Plunge</h2> <p>It was actually about a year after I had become convinced that repositories were unnecessary, useless abstractions that I started working with a new codebase I had the opportunity to steer.  Once I eliminated them from the equation, everything got so much simpler.   Having been repository-free for about two years now, I think I’d have a hard time joining a team that had an affinity for them.</p> <h2 id="conclusion">Conclusion</h2> <p>If you’re still using repositories and you don’t have some other hangup you still need to get over like writing unit tests for your controllers or application services then give the repository-free lifestyle a try.  I bet you’ll love it.</p> Using Manual Mocks to test the AWS SDK with Jest https://derikwhittaker.blog/2018/02/20/using-manual-mocks-to-test-the-aws-sdk-with-jest/ Maintainer of Code, pusher of bits… urn:uuid:3a424860-3707-7327-2bb1-a60b9f3be47d Tue, 20 Feb 2018 13:56:45 +0000 Anytime you build Node applications it is highly suggested that your cover your code with tests.  When your code interacts with 3rd party API&#8217;s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests &#8230; <p><a href="https://derikwhittaker.blog/2018/02/20/using-manual-mocks-to-test-the-aws-sdk-with-jest/" class="more-link">Continue reading <span class="screen-reader-text">Using Manual Mocks to test the AWS SDK with&#160;Jest</span></a></p> <p>Anytime you build Node applications it is highly suggested that your cover your code with tests.  When your code interacts with 3rd party API&#8217;s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests not unit tests.</p> <p>If you are using <a href="http://bit.ly/jest-get-started" target="_blank" rel="noopener">Jest</a>, one solution is utilize the built in support for <a href="http://bit.ly/jest-manual-mocks" target="_blank" rel="noopener">manual mocks.</a>  I have found the usage of manual mocks invaluable while testing 3rd party API&#8217;s such as the AWS.  Keep in mind just because I am using manual mocks this will remove the need for using libraries like <a href="http://bit.ly/sinon-js" target="_blank" rel="noopener">SinonJs</a> (a JavaScript framework for creating stubs/mocks/spies).</p> <p>The way that manual mocks work in Jest is as follows (from the Jest website&#8217;s documentation).</p> <blockquote><p><em>Manual mocks are defined by writing a module in a <code>__mocks__/</code> subdirectory immediately adjacent to the module. For example, to mock a module called <code>user</code> in the <code>models</code> directory, create a file called <code>user.js</code> and put it in the <code>models/__mocks__</code> directory. Note that the <code>__mocks__</code> folder is case-sensitive, so naming the directory <code>__MOCKS__</code> will break on some systems. If the module you are mocking is a node module (eg: <code>fs</code>), the mock should be placed in the <code>__mocks__</code> directory adjacent to <code>node_modules</code> (unless you configured <a href="https://facebook.github.io/jest/docs/en/configuration.html#roots-array-string"><code>roots</code></a> to point to a folder other than the project root).</em></p></blockquote> <p>In my case I want to mock out the usage of the <a href="http://bit.ly/npm-aws-sdk" target="_blank" rel="noopener">AWS-SDK</a> for <a href="http://bit.ly/aws-sdk-node" target="_blank" rel="noopener">Node</a>.</p> <p>To do this I created a __mocks__ folder at the root of my solution.  I then created a <a href="http://bit.ly/gist-aws-sdk-js" target="_blank" rel="noopener">aws-sdk.js</a> file inside this folder.</p> <p>Now that I have my mocks folder created with a aws-sdk.js file I am able to consume my manual mock in my jest test by simply referencing the aws-sdk via a <code>require('aws-sdk')</code> command.</p> <div class="code-snippet"> <pre class="code-content">const AWS = require('./aws-sdk'); </pre> </div> <p>With declaration of AWS above my code is able to a use the <a href="http://bit.ly/npm-aws-sdk" target="_blank" rel="noopener">NPM </a>package during normal usage, or my aws-sdk.js mock when running under the Jest context.</p> <p>Below is a small sample of the code I have inside my aws-sdk.js file for my manual mock.</p> <div class="code-snippet"> <pre class="code-content">const stubs = require('./aws-stubs'); const AWS = {}; // This here is to allow/prevent runtime errors if you are using // AWS.config to do some runtime configuration of the library. // If you do not need any runtime configuration you can omit this. AWS.config = { setPromisesDependency: (arg) =&gt; {} }; AWS.S3 = function() { } // Because I care about using the S3 service's which are part of the SDK // I need to setup the correct identifier. // AWS.S3.prototype = { ...AWS.S3.prototype, // Stub for the listObjectsV2 method in the sdk listObjectsV2(params){ const stubPromise = new Promise((resolve, reject) =&gt; { // pulling in stub data from an external file to remove the noise // from this file. See the top line for how to pull this in resolve(stubs.listObjects); }); return { promise: () =&gt; { return stubPromise; } } } }; // Export my AWS function so it can be referenced via requires module.exports = AWS; </pre> </div> <p>A few things to point out in the code above.</p> <ol> <li>I chose to use the <a href="http://bit.ly/sdk-javascript-promises" target="_blank" rel="noopener">promise</a>s implementation of the listObjectsV2.  Because of this I need to return a promise method as my result on my listObjectsV2 function.  I am sure there are other ways to accomplish this, but this worked and is pretty easy.</li> <li>My function is returning stub data, but this data is described in a separate file called aws-stubs.js which sites along side of my aws-sdk.js file.  I went this route to remove the noise of having the stub data inside my aws-adk file.  You can see a full example of this <a href="http://bit.ly/gist-aws-stub-data" target="_blank" rel="noopener">here</a>.</li> </ol> <p>Now that I have everything setup my tests will no longer attempt to hit the actually aws-sdk, but when running in non-test mode they will.</p> <p>Till next time,</p> Configure Visual Studio Code to debug Jest Tests https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/ Maintainer of Code, pusher of bits… urn:uuid:31928626-b984-35f6-bf96-5bfb71e16208 Fri, 16 Feb 2018 21:33:03 +0000 If you have not given Visual Studio Code a spin you really should, especially if  you are doing web/javascript/Node development. One super awesome feature of VS Code is the ability to easily configure the ability to debug your Jest (should work just fine with other JavaScript testing frameworks) tests.  I have found that most of &#8230; <p><a href="https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/" class="more-link">Continue reading <span class="screen-reader-text">Configure Visual Studio Code to debug Jest&#160;Tests</span></a></p> <p>If you have not given <a href="https://code.visualstudio.com/" target="_blank" rel="noopener">Visual Studio Code</a> a spin you really should, especially if  you are doing web/javascript/Node development.</p> <p>One super awesome feature of VS Code is the ability to easily configure the ability to debug your <a href="https://facebook.github.io/jest/" target="_blank" rel="noopener">Jest </a>(should work just fine with other JavaScript testing frameworks) tests.  I have found that most of the time I do not need to actually step into the debugger when writing tests, but there are times that using <code>console.log</code> is just too much friction and I want to step into the debugger.</p> <p>So how do we configure VS Code?</p> <p>First you  will need to install the <a href="https://www.npmjs.com/package/jest-cli" target="_blank" rel="noopener">Jest-Cli</a> NPM package (I am assuming you already have Jest setup to run your tests, if you do not please read the <a href="https://facebook.github.io/jest/docs/en/getting-started.html" target="_blank" rel="noopener">Getting-Started</a> docs).  If you fail to do this step you will get the following error in Code when you try to run the debugger.</p> <p><img data-attachment-id="78" data-permalink="https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/jestcli/" data-orig-file="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640" data-orig-size="702,75" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="JestCLI" data-image-description="" data-medium-file="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640?w=300" data-large-file="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640?w=640" class="alignnone size-full wp-image-78" src="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640" alt="JestCLI" srcset="https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=640 640w, https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=150 150w, https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png?w=300 300w, https://derikwhittaker.files.wordpress.com/2018/02/jestcli.png 702w" sizes="(max-width: 640px) 100vw, 640px" /></p> <p>After you have Jest-Cli installed you will need to configure VS Code for debugging.  To do this open up the configuration by clicking Debug -&gt; Open Configurations.  This will open up a file called launch.json.</p> <p>Once launch.json is open add the following configuration</p> <div class="code-snippet"> <pre class="code-content"> { "name": "Jest Tests", "type": "node", "request": "launch", "program": "${workspaceRoot}/node_modules/jest-cli/bin/jest.js", "stopOnEntry": false, "args": ["--runInBand"], "cwd": "${workspaceRoot}", "preLaunchTask": null, "runtimeExecutable": null, "runtimeArgs": [ "--nolazy" ], "env": { "NODE_ENV": "development" }, "console": "internalConsole", "sourceMaps": false, "outFiles": [] } </pre> </div> <p>Here is a gist of a working <a href="https://gist.github.com/derikwhittaker/331d4a5befddf7fc6b2599f1ada5d866" target="_blank" rel="noopener">launch.json</a> file.</p> <p>After you save the file you are almost ready to start your debugging.</p> <p>Before you can debug you will want to open the debug menu (the bug icon on the left toolbar).   This will show a drop down menu with different configurations.  Make sure &#8216;Jest Test&#8217; is selected.</p> <p><img data-attachment-id="79" data-permalink="https://derikwhittaker.blog/2018/02/16/configure-visual-studio-code-to-debug-jest-tests/jesttest/" data-orig-file="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640" data-orig-size="240,65" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="JestTest" data-image-description="" data-medium-file="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640?w=240" data-large-file="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640?w=240" class="alignnone size-full wp-image-79" src="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=640" alt="JestTest" srcset="https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png 240w, https://derikwhittaker.files.wordpress.com/2018/02/jesttest.png?w=150 150w" sizes="(max-width: 240px) 100vw, 240px" /></p> <p>If you have this setup correctly you should be able to set breakpoints and hit F5.</p> <p>Till next time,</p> Going Async with Node AWS SDK with Express https://derikwhittaker.blog/2018/02/13/going-async-with-node-aws-sdk-with-express/ Maintainer of Code, pusher of bits… urn:uuid:d4750cda-8c6e-8b2f-577b-78c746ee6ebd Tue, 13 Feb 2018 13:00:30 +0000 When building applications in Node/Express you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary.  The 'old school' way was to use call backs, which often led to callback hell.  Than came along Promises which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything.  Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for async/await was introduced and this really has cleaned up and enhanced the readability of your code. <p>When building applications in <a href="https://nodejs.org/en/" target="_blank" rel="noopener">Node</a>/<a href="http://expressjs.com/" target="_blank" rel="noopener">Express </a>you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary.  The &#8216;old school&#8217; way was to use call backs, which often led to <a href="http://callbackhell.com/" target="_blank" rel="noopener">callback hell</a>.  Than came along <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promises</a> which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything.  Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function" target="_blank" rel="noopener">async</a>/<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await" target="_blank" rel="noopener">await</a> was introduced and this really has cleaned up and enhanced the readability of your code.</p> <p>Having the ability to use async/await is great, and is supported out of the box w/ Express.  But what do you do when you using a library which still wants to use promises or callbacks? The case in point for this article is <a href="https://aws.amazon.com/sdk-for-node-js/" target="_blank" rel="noopener">AWS Node SDK</a>.</p> <p>By default if you read through the AWS SDK documentation the examples lead you to believe that you need to use callbacks when implementing the SDK.  Well this can really lead to some nasty code in the world of Node/Express.  However, as of <a href="https://aws.amazon.com/blogs/developer/support-for-promises-in-the-sdk/" target="_blank" rel="noopener">v2.3.0</a> of the AWS SDK there is support for Promises.  This is much cleaner than using callbacks, but still poses a bit of an issue if you want to use async/await in your Express routes.</p> <p>However, with a bit of work you can get your promise based AWS calls to play nicely with your async/await based Express routes.  Lets take a look at how we can accomplish this.</p> <p>Before you get started I am going to make a few assumptions.</p> <ol> <li>You already have a Node/Express application setup</li> <li>You already have the AWS SDK for Node installed, if not read <a href="https://aws.amazon.com/sdk-for-node-js/" target="_blank" rel="noopener">here</a></li> </ol> <p>The first thing we are going to need to do is add reference to our AWS SDK and configure it to use promises.</p> <div class="code-snippet"> <pre class="code-content">const AWS = require('aws-sdk'); AWS.config.setPromisesDependency(null); </pre> </div> <p>After we have our SDK configured we can implement our route handler.  In my example here I am placing all the logic inside my handler.  In a real code base I would suggest better deconstruction of this code into smaller parts.</p> <div class="code-snippet"> <pre class="code-content">const express = require('express'); const router = express.Router(); const s3 = new AWS.S3(); router.get('/myRoute', async (req, res) =&gt; { const controller = new sitesController(); const params = req.params; const params = { Bucket: "bucket_name_here" }; let results = {}; var listPromise = s3.listObjects(params).promise(); listPromise.then((data) =&gt; { results = data; }); await Promise.all([listPromise]); res.json({data: results }) }) module.exports = router; </pre> </div> <p>Lets review the code above and call out a few important items.</p> <p>The first thing to notice is the addition of the <code>async</code> keyword in my route handler.  This is what allows us to use async/await in Node/Express.</p> <p>The next thing to look at is how I am calling the s3.listObjects.  Notice I am <strong>NOT </strong>providing a callback to the method, but instead I am chaining with .promise().  This is what instructs the SDK to use promises vs callbacks.  Once I have my callback I chain a &#8216;then&#8217; in order to handle my response.</p> <p>The last thing to pay attention to is the line with <code>await Promise.All([listPromise]);</code> This is the magic forces our route handler to not return prior to the resolution of all of our Promises.  Without this your call would exit prior to the listObjects call completing.</p> <p>Finally, we are simply returning our data from the listObjects call via <code>res.json</code> call.</p> <p>That&#8217;s it, pretty straight forward, once you learn that the AWS SDK supports something other than callbacks.</p> <p>Till next time,</p> Unable To Access Mysql With Root and No Password After New Install On Ubuntu https://blog.jasonmeridth.com/posts/unable-to-access-mysql-with-root-and-no-password-after-new-install-on-ubuntu/ Jason Meridth urn:uuid:f81a51eb-8405-7add-bddb-f805b183347e Wed, 31 Jan 2018 00:13:00 +0000 <p>This bit me in the rear end again today. Had to reinstall mysql-server-5.7 for other reasons.</p> <p>You just installed <code class="highlighter-rouge">mysql-server</code> locally for your development environment on a recent version of Ubuntu (I have 17.10 artful installed). You did it with a blank password for <code class="highlighter-rouge">root</code> user. You type <code class="highlighter-rouge">mysql -u root</code> and you see <code class="highlighter-rouge">Access denied for user 'root'@'localhost'</code>.</p> <p><img src="https://blog.jasonmeridth.com/images/wat.png" alt="wat" /></p> <p>Issue: Because you chose to not have a password for the <code class="highlighter-rouge">root</code> user, the <code class="highlighter-rouge">auth_plugin</code> for my MySQL defaulted to <code class="highlighter-rouge">auth_socket</code>. That means if you type <code class="highlighter-rouge">sudo mysql -u root</code> you will get in. If you don’t, then this is NOT the fix for you.</p> <p>Solution: Change the <code class="highlighter-rouge">auth_plugin</code> to <code class="highlighter-rouge">mysql_native_password</code> so that you can use the root user in the database.</p> <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo mysql -u root mysql&gt; USE mysql; mysql&gt; UPDATE user SET plugin='mysql_native_password' WHERE User='root'; mysql&gt; FLUSH PRIVILEGES; mysql&gt; exit; $ sudo systemctl restart mysql $ sudo systemctl status mysql </code></pre></div></div> <p><strong>NB</strong> ALWAYS set a password for mysql-server in staging/production.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/unable-to-access-mysql-with-root-and-no-password-after-new-install-on-ubuntu/">Unable To Access Mysql With Root and No Password After New Install On Ubuntu</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on January 30, 2018.</p> New Job https://blog.jasonmeridth.com/posts/new-job/ Jason Meridth urn:uuid:102e69a7-2b63-e750-2fa5-f46372d4d7c1 Mon, 08 Jan 2018 18:13:00 +0000 <p>Well, it is a new year and I’ve started a new job. I am now a Senior Software Engineer at <a href="https://truelinkfinancial.com">True Link Financial</a>.</p> <p><img src="https://blog.jasonmeridth.com/images/tllogo.png" alt="true link financial logo" /></p> <p>After interviewing with the co-founders Kai and Claire and their team, I knew I wanted to work here.</p> <p><strong>TL;DR</strong>: True Link: We give elderly and disable (really, anyone) back their financial freedom where they may not usually have it.</p> <p>Longer Version: Imagine you have an elderly family member who may start showing signs of dimensia. You can give them a True Link card and administer their card. You link it to their bank account or another source of funding and you can set limitations on when, where and how the card can be used. The family member feels freedom by not having to continually ask for money but is also protected by scammers and non-friendly people (yep, they exist).</p> <p>The customer service team, the marketing team, the product team, the engineering team and everyone else at True Link are amazing.</p> <p>For any nerd readers, the tech stack is currently Rails, React, AWS, Ansible. We’ll be introducing Docker and Kubernetes soon hopefully, but always ensuring the right tools for the right job.</p> <p>Looking forward to 2018.</p> <p>Cheers.</p> <p><a href="https://blog.jasonmeridth.com/posts/new-job/">New Job</a> was originally published by Jason Meridth at <a href="https://blog.jasonmeridth.com">Jason Meridth</a> on January 08, 2018.</p>