Los Techies

Becoming a Technical Fellow at Headspring, and New Beginnings

Tue, 23 Jun 2020 19:46:43 +0000

My journey at Headspring started for me at a local developer meetup, the "Austin Agile Lunch" group, way back in 2007. I had started blogging and getting involved in the community, attending the local .NET user group, code camp, and most influentially, the first ALT.NET open space conference. At this lunch, Jeffrey Palermo, who had started at Headspring that year, asked the innocuous question "Do you know anyone looking for a job?" And the naive me answered "No, but if I know of anyone, I'll be glad to point them your way!" Jeffrey, always direct, tried again, "Do you know anyone named JIMMY looking for a job?"

I had been quite unhappy at my current gig (as anyone who's seen my talks about it can attest), but wasn't quite sure what was next. I had done a cash-flow challenged startup, a product company, and Big Corp IT, all in the span of the first 4 years of my career. Although Headspring was only 5 people at the time, it offered something I couldn't get anywhere else. An opportunity to drive for excellence in an environment where everyone else had a similar passion, and an ability to affect positive change at a level I couldn't dream of before.

Within a year or so after I started, thanks to support and encouragement from coworkers and the opportunities I now had, I received my first Microsoft MVP award, co-authored my first book, published my first OSS project used in production, and wrote dozens of blog posts of everything I was working on, all in trying to live up to a promise I made to myself to pass on what I had learned as I had indebted myself to so many others who shared their own learnings.

To pay it forward.

This culminated in Headspring recognizing me as the first-ever Technical Fellow, an honor I never set out to receive, and I'm even more indebted to others to repay.

And now, 13 years later, my Headspring journey shifts (again).

For some years, I looked at going independent as the next logical step for my career. However, I love working at Headspring, the people, the clients, and the work, and could not imaging giving that up, for almost any opportunity. I recognized that although I could ask for the flexibility of an independent consultant, I would not ask for, nor accept, that sort of special treatment. As it became clear that I could go independent, I approached the Headspring leadership to look at possibilities of being independent while also at Headspring.

Thanks to the great support of Dustin and rest of the Headspring leadership, we were able to come to a mutually beneficial agreement. While I'm continuing my role as the Chief Architect at Headspring, I've stepped back as a full-time employee and staying on as an independent contractor (Jimmy Bogard Consulting LLC, I'm not going to win any awards for creativity).

My role as Chief Architect I hope (and plan) to continue for years to come, as I believe in Headspring's vision and future, and we've signed a partnership agreement to that effect. What changes for me is my day-to-day projects and clients may or may not be strictly Headspring projects and clients, or any projects at all (maybe become a true brisket master?). My hope is to be able to have the flexibility to consult on the kinds of opportunities that are only feasible as an independent consultant, while still helping to fulfill the vision and mission of Headspring that I've helped to define and shape for over a decade.

As my grandfather was fond of saying, "Onward and upward".

Building End-to-End Diagnostics: User-Defined Context with Correlation Context

Mon, 22 Jun 2020 13:06:04 +0000

Posts in this series:

Source Code

With a brief detour to push out some NuGet packages, I wanted to pick up with a common issue folks run into once they start implementing distributed tracing. And that problem is one of locality - trying to find one trace among many, the needle in the haystack. In the past, I've resorted to combing through logs, using timestamps as a very poor means of trying to diagnose an issue.

We can decorate our logs with context information to make searching easier, but when it comes to our traces, how can we triangulate something a user did with a workflow and the associated traces?

This is where the emerging Correlation Context standard comes in to play - the ability for application code to add arbitrary information to traces - and critically - have that information flow through subsequent spans.

With distributed trace spans, you can add information to a span so that it eventually shows up in a collector/exporter. This is possible today with the System.Diagnostics.Activity API through tags:

Activity.Current.AddTag("user.id", user.Id);

But that information does not flow through to any subsequent spans within a process, or to subsequent processes. It exists for a single Activity/span, then it's gone.

This is where Correlation Context comes in. Trace context spec defines a "tracestate" header for vendor-specific trace information to propagate, and Correlation Context allows application code to add application-specific trace information to propagate.

Propagating Correlation Context in NServiceBus

ASP.NET Core will automatically parse correlation context header information, and places this in Activity.Baggage:

string[] baggage = headers.GetCommaSeparatedValues(HeaderNames.CorrelationContext);
if (baggage.Length > 0)
{
    foreach (var item in baggage)
    {
        if (NameValueHeaderValue.TryParse(item, out var baggageItem))
        {
            activity.AddBaggage(baggageItem.Name.ToString(), HttpUtility.UrlDecode(baggageItem.Value.ToString()));
        }
    }
}

Baggage, unlike Tags, will flow through to child Activities. When an Activity starts, the parent Activity.Current will flow its ParentId through to the new Activity. When you access an Activity's Baggage, the implementation pulls the current baggage, its Parent's baggage, and every parent up the chain.

With NServiceBus, we want to also parse incoming baggage, and propagate outgoing baggage through the Correlation-Context header. And although this header is still in draft mode with the W3C, it already has implementation support in ASP.NET Core 3.0.

To parse incoming headers, we can do the same operation that ASP.NET Core does, in our code that parses the incoming traceparent, we can look for the Correlation-Context header and place the values in Activity.Baggage:

if (context.MessageHeaders.TryGetValue(Headers.CorrelationContextHeaderName, out var correlationContext))
{
    var baggage = correlationContext.Split(',');
    if (baggage.Length > 0)
    {
        foreach (var item in baggage)
        {
            if (NameValueHeaderValue.TryParse(item, out var baggageItem))
            {
                activity.AddBaggage(baggageItem.Name, HttpUtility.UrlDecode(baggageItem.Value));
            }
        }
    }
}

Now that we have correlation context in our Baggage, any other child activities will have this baggage, too.

The last piece is propagation, in our original outgoing NServiceBus behavior that propagates traceparent:

if (!context.Headers.ContainsKey(Headers.CorrelationContextHeaderName))
{
    var baggageItems = activity.Baggage.Select(item => $"{item.Key}={item.Value}");
    var headerValue = string.Join(",", baggageItems);
    if (!string.IsNullOrEmpty(headerValue))
    {
        context.Headers[Headers.CorrelationContextHeaderName] = headerValue;
    }
}

Will now propagate the baggage out through the Correlation-Context header. With the incoming and outgoing header behaviors in place, any service can drop some data into baggage and have it propagate to all downstream services.

So all done, right? Well, not quite, as even though we added information to the Activity.Baggage, it doesn't necessarily mean that those values get exported to our tracing tools. Unfortunately today, OpenTelemetry exporters only consider the Tags portion of an Activity for exporting. This will be opt-in in the future, but for now, we'll need to manually copy our Baggage to Tags during the export process (or in child activities).

In the next post, we'll walk through exactly that - creating a "breadcrumb" Activity that piggybacks on events of other activities.

Diagnostics and Instrumentation Packages for MongoDB and NServiceBus Published

Wed, 17 Jun 2020 14:25:00 +0000

As part of the end-to-end diagnostics and tracing blog series, I had an end goal of eventually publishing out NuGet packages for diagnostics (Activity and DiagnosticSource). I'm happy to announce that I've released 4 packages to NuGet:

These packages add:

Activity and DiagnosticsSource support
OpenTelemetry adapters
W3C Trace Context and Correlation Context support for NServiceBus

With the diagnostics packages, you can write your own DiagnosticListeners to subscribe to diagnostics Activity events. The OpenTelemetry packages listen to these Activity events and adapt them to OpenTelemetry spans. The various existing exporters can output these spans to Zipkin, Jaeger, Application Insights, and many other observability tools.

I've updated the ReadMe's for each repo to include instructions for how to use:

With these packages, you can easily configure OpenTelemetry to include MongoDB and NServiceBus support:

services.AddOpenTelemetry(builder => 
    builder
        .UseZipkin(o =>
        {
            o.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
            o.ServiceName = EndpointName;
        })
        .AddNServiceBusAdapter(opt => opt.CaptureMessageBody = true)
        .AddMongoDBAdapter(opt => opt.CaptureCommandText = true)
        .AddRequestAdapter()
        .AddDependencyAdapter(configureSqlAdapterOptions: opt => opt.CaptureTextCommandContent = true);
);

Enjoy!

Picking a Web Microframework

Wed, 27 May 2020 00:23:00 +0000

I’ve had to use this at work the last couple of weeks. We had a “home grown” framework for a new application we’re working and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization anyway to do routes, etc).

However, being that the group I was working with is pretty “anti framework” I had to settle on something that was light weight, integrated with jetty and allowed us to work the way that was comfortable for us as team (also it had to work with Scala).

Microframeworks

The team had shown a lot of disdain for Play (which I had actually quite a lot when I last was leading a JVM based tech stack) and Spring Boot as being too heavy weight, so these were definitely out.

Fortunately, in the JVM world there is a big push back now on heavy web frameworks so meant I had lots of choices for “non frameworks” but could still do some basic security, routing, authentication but not hurt the existing team’s productivity.

There are probably 3 dozen microframeworks to choose from with varying degrees of value but the two that seemed to easiest to start with today were:

My Attempt with Quarkus

Quarkus has a really great getting started story but it’s harder to get started on an existing project with it, it was super trivial to add, and after a couple of days of figuring out the magic incantation I just decided to punt on it. I think because of it’s popularity in the Cloud Native space (which we’re trying to target), the backing of Red Hat, and the pluggable nature of the stack there are a lot of reasons to want this to work. In the end because of the timeline it didn’t make the cut. But it may come back.

My Attempt with Javalin

Javalin despite being a less popular project than Quarkus it is getting some buzz. It also looks like it just slides into the team’s existing Servlet code base. I wanted this to work very badly but stopped before I even started because of this issue so this was out despite being on paper a really execellent framework.

My Attempt with Scalatra

Scalatra has been around for a number of years and is inspired by Sinatra which I used quite a bit in my Ruby years. This took a few minutes to get going just following their standalone directions and then some more to successful convert the routes and account for learning curves with routes.

Some notes:

The routing API and parameters etc are very nice to work with IMO.
It was very easy to get json by default support setup.
Metrics were very easy to wire up.
Swagger integration was pretty rough, while it looks good on paper I could not get an example to show up, and it is unable to handle case classes or enums which we use.
Benchmark performance when I’ve looked around the web was pretty bad, I’ve not done enough to figure out if this is real or not. I’ve seen first hand a lot of benchmarking are just wrong.
Integration with JUnit has been rough and I cannot seem to get the correct port to fire, I suspect I have to stop using the @Test annotation is all (which I’m not enjoying).
Http/2 support is still lacking despite being available in the version of Jetty they’re on, I’ve read a few places that an issue is keeping web sockets working but either way there is no official support in the project yet.

Conclusion

I think we’re going to stick with Scalatra for the time being as it is a muture framework that works well for our current goals. However, the lack of http/2 support maybe a deal breaker in the medium term.

Building End-to-End Diagnostics: Visualization with Exporters

Tue, 19 May 2020 14:46:42 +0000

Posts in this series:

Source Code

In the last post, we looked at surfacing our diagnostics events from the Activity and DiagnosticSource APIs on through to OpenTelemetry Spans.

Side note - the .NET team is working to close the gap between the OTel "Span" concept and the Activity API. The end goal is that the Activity class represents an OTel "Span".

With all of these pieces in place, we can now export these traces to...well, something! The benefit of this common OTel API means that it should be much easier to export your traces to some collector that can then store, index, and visualize your traces. But before we look at visualization, let's build a couple of complex but contrived scenarios for a more real-world scenario distributed system.

Orchestration Sample

I wanted to try to use as many different distributed components as possible that I typically see in my client work, in an arrangement of different conversation patterns, so I wanted an example system that did:

Request/Response with HTTP
Request/Response with AMQP
Pub/Sub with AMQP
Orchestration and Choreography with AMQP
Database interaction with SQL and MongoDB

There's probably someone winning a distributed systems bingo at this point. My sample application will have 3 different endpoints:

Web Application (receives the first request)
Worker Application (Worker SDK)
Child Worker Application (another Worker SDK)

Because I'm lazy, I'll kick things off with the Swagger UI, and initiate 2 kinds of interactions:

Process manager using orchestration
Process manager using choreography

First, let's look at an orchestration conversation:

Conversation using orchestration with HTTP and AMQP

The code is a bit much, but let's walk through the interactions:

HTTP POST to Web Application
AMQP Send to Worker Application
Worker Application receives message and starts process manager; AMQP Send to itself
Worker Application receives message, makes HTTP POST to Web Application
Web Application makes SQL call
Worker Application receives HTTP reply, makes AMQP Reply
Worker Application process manager receives reply, makes AMQP Send to Child Application
Child Application receives message, makes MongoDB call and makes AMQP Reply to Worker Application
Worker Application process manager receives reply, marks process as complete and makes AMQP Reply to process manager initiator
Web Application receives message and conversation completes

WHEW. Now this conversation does resemble ones I've built, but it is a bit contrived as I don't usually have these back-and-forth communications. But it does show the different kinds of communications in a complex conversation.

If something were to go awry in this conversation, we wouldn't have much recourse to fix anything without distributed tracing.

Exporting Traces with OpenTelemetry

Configuring our applications to export their traces and spans is easy, first we add the appropriate exporter packages:

<PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="0.2.0-alpha.283" />
<PackageReference Include="OpenTelemetry.Exporter.Zipkin" Version="0.2.0-alpha.283" />

Once we have our packages referenced, we can configure them with the rest of our OpenTelemetry setup:

services.AddOpenTelemetry(builder => builder
    .UseZipkin(o =>
    {
        o.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
        o.ServiceName = Program.EndpointName;
    })
    .UseJaeger(c =>
    {
        c.AgentHost = "localhost";
        c.AgentPort = 6831;
        c.ServiceName = Program.EndpointName;
    })
    .AddNServiceBusAdapter()
    .AddRequestAdapter()
    .AddDependencyAdapter(configureSqlAdapterOptions: 
        opt => opt.CaptureTextCommandContent = true));

Each call to an exporter needs some configuration on how to connect to it. Zipkin and Jaeger will need a URI, while Application Insights would need our instrumentation key.

That's it! Once we've configured our exporters, our application will export them seamlessly as it handles requests. Let's look at traces for our orchestration-style communication (and we should see all the steps above), first in Zipkin:

Zipkin trace of orchestration conversation

And next in Jaeger:

Jaeger trace of orchestration conversation

In each of these traces, we can follow the conversation all the way from the initial HTTP POST all the way down to the final AMQP "reply" back to the process manager originator. And because this is an orchestration-style process, the execution times are completely linear - no process hands off to another until it completes.

Without our integration into NServiceBus, we wouldn't have this connected conversation - only disconnected individual traces for each HTTP request. With OpenTelemetry, we've got a complete view of our entire conversation, with each step recorded along the way.

Let's contrast this with a choreography-style conversation to see how that might look.

Choreography Sample

In our orchestration sample, we used commands and replies to dictate a flow for our conversation. With choreography, our communication is more event-oriented, and multiple steps can execute in parallel. Our choreography conversation looks like:

Conversation using choreography with HTTP and AMQP

This conversation is a bit different than the orchestration conversation because we're using events, rather than commands, to coordinate an activity. This allows some processes to execute in parallel. The worker application and child application now handle the published event simultaneously, and publish an event when their step completes.

Finally, our process manager subscribes to the "complete" event from both processes, and when both events are received, the process manager publishes its final event back to the Web Application.

We can see that there are fewer messages in this approach, as each application uses an event to fire, as opposed to the two messages involved in request/response. With our choreography application set up, let's look to see how this conversation manifests in Zipkin:

Zipkin trace of choreography conversation

And in Jaeger:

Jaeger trace for choreography conversation

In each of these traces, we see the interaction kick off with the Web Application publishing an event. Next, both the Child Worker Service and Worker Service execute in parallel, making requests and doing work. We also see the Child Worker Service publishing an event to the Worker Service process manager, but it is not until the second event that the process manager receives that the final publish and receive from the Web Application happens.

Looking at each of these traces, we can see how the conversation patterns differ in their communication style and processing.

With OpenTelemetry, we've made it simple to add end-to-end distributed tracing to our system. None of our application code is aware of OpenTelemetry even existing, and with the upcoming changes to the Activity API, adding distributed tracing will be made even easier.

In the next (and last) post, I'll share some future direction of OpenTelemetry, as well as links to the sample extensions made into released NuGet packages.

Building End-to-End Diagnostics: Activity and Span Correlation

Wed, 13 May 2020 14:23:28 +0000

Posts in this series:

Source Code

In the last post, we looked at hooking up our diagnostics events (and Activities) to OpenTelemetry, where our main task was creating the appropriate span attributes based on the OpenTelemetry Span conventions. When everything runs, we get lovely correlated events based on our W3C tracecontext headers flowing through, as our Activity maintains a link to its parent, regardless of where that parent activity came from.

The Activity to Span connection works great, as long as we can maintain a link between the OTel "Span" and the Activity Start/Stop events themselves, and we're effectively able to start and stop the right activity when something interesting happens in our observed code.

Sometimes, it's not so easy to correlate the "start" and "stop" events for a given observed event. Consider our NServiceBus consumer activity logic:

public override async Task Invoke(
    IIncomingPhysicalMessageContext context,
    Func<Task> next)
{
    var activity = StartActivity(context);

    try
    {
        await next().ConfigureAwait(false);
    }
    finally
    {
        StopActivity(activity, context);
    }
}

For the Start and Stop activities, we have the reference to the activity we started in that activity variable. We can do this because the infrastructure hook for consuming a message wraps the entire operation, and we're passed a continuation (that next parameter). But what happens if the infrastructure hooks we have for "start" and "stop" are physically separate? How do we know which Activity to stop once the observed event completes?

I ran into this exact situation with the MongoDB .NET Driver, which does have a mechanism to raise internal diagnostic events, but your handling of these events is physically separate in your code.

Correlating Events

The main interface to listen to internal diagnostic events in the MongoDB .NET Driver is the IEventSubscriber interface:

public interface IEventSubscriber
{
    bool TryGetEventHandler<TEvent>(out Action<TEvent> handler);
}

We can register subscribers, but we only get this one method that gets called based on different TEvent types. That means that we don't get a single method for opening a connection or issuing a command. Those are separate calls with separate callbacks.

First, let's get some boilerplate to handle these events. There's a helper class, ReflectionEventSubscriber, that lets us define methods with a signature that accepts the event objects and the subscriber uses reflection to find these events. The events we're looking to handle are CommandStartedEvent, CommandSucceededEvent, and CommandFailedEvent. Our event subscriber implementation can wrap this reflection-based event subscriber:

public class DiagnosticsActivityEventSubscriber : IEventSubscriber
{
    private readonly ReflectionEventSubscriber _subscriber;

    public DiagnosticsActivityEventSubscriber() =>
        _subscriber = new ReflectionEventSubscriber(this, 
            bindingFlags: BindingFlags.Instance | BindingFlags.NonPublic);

    public bool TryGetEventHandler<TEvent>(out Action<TEvent> handler) 
        => _subscriber.TryGetEventHandler(out handler);

    private void Handle(CommandStartedEvent @event)
    {
        // Start activity
    }

    private void Handle(CommandSucceededEvent @event)
    {
        // Stop activity
    }

    private void Handle(CommandFailedEvent @event)
    {
        // Stop activity
    }
}

As we can see above, the methods to handle the start/stop activities are physically separate, but we need to share our started Activity between these! We can store the started activity in a field, but when the stop events come in, we need to make sure we stop the right activity.

To find the "right" activity, we need some way to correlate the information in the CommandStartedEvent object and the CommandSucceededEvent/ CommandFailedEvent objects. Luckily, the Mongo drivers for other languages are quite similar to .NET, and some of those other drivers have OpenTelemetry implementations. From that code, we can see that these events have a RequestId that uniquely identifies this request against the MongoDB server. This could work!

Implementing our Event Subscriber

First, we need the plumbing of our diagnostic source in our DiagnosticsActivityEventSubscriber class to raise the diagnostic events:

public class DiagnosticsActivityEventSubscriber : IEventSubscriber
{
    public const string ActivityName = "MongoDB.Driver.Core.Events.Command";

    private static readonly DiagnosticSource _diagnosticListener
        = new DiagnosticListener(ActivityName);

Next, we somewhere to place our started Activity objects, correlated by the RequestId property. The most straightforward solution would be a ConcurrentDictionary for this value:

private readonly ConcurrentDictionary<int, Activity> _activityMap
    = new ConcurrentDictionary<int, Activity>();

Now, when we start our activity, we follow the normal steps we did in our NServiceBus example, with the added step of adding the started activity to our dictionary:

private void Handle(CommandStartedEvent @event)
{
    var activity = new Activity(ActivityName);

    if (_diagnosticListener.IsEnabled(CommandStarted.EventName, @event))
    {
        _diagnosticListener.StartActivity(activity, new CommandStarted(@event));
    }
    else
    {
        activity.Start();
    }

    _activityMap.TryAdd(@event.RequestId, activity);
}

With the event started and stored locally, we can stop our activity based on pulling the activity back out of the local dictionary:

private void Handle(CommandSucceededEvent @event)
{
    if (_activityMap.TryRemove(@event.RequestId, out var activity))
    {
        _diagnosticListener.StopActivity(activity, new CommandSucceeded(@event));
    }
}

private void Handle(CommandFailedEvent @event)
{
    if (_activityMap.TryRemove(@event.RequestId, out var activity))
    {
        _diagnosticListener.StopActivity(activity, new CommandFailed(@event));
    }
}

Initially, I tried to use Activity.Current to pull out the activity, but that won't always be correct if the async context hasn't flowed through to my event handler (it did not). Instead, with this correlated dictionary, I can ensure I stop the correct activities and my eventual OTel spans will correlate as well.

OpenTelemetry Integration

On the OTel side, we have a similar issue where the "Current Span" assumes a common execution context. But since context isn't shared, we have to implement our ListenerHandler in a similar fashion, but this time, correlate a TelemetrySpan:

internal class CommandListener : ListenerHandler
{
    public CommandListener(string sourceName, Tracer tracer) 
        : base(sourceName, tracer)
    {
    }

    private readonly ConcurrentDictionary<int, TelemetrySpan> _spanMap
        = new ConcurrentDictionary<int, TelemetrySpan>();

Now when we receive the OnStartActivity call, we have to perform a similar operation to store our TelemetrySpan based on that OperationId:

public override void OnStartActivity(Activity activity, object payload)
{
    if (!(payload is CommandStarted message))
    {
        AdapterEventSource.Log.NullPayload("CommandListener.OnStartActivity");
        return;
    }

    Tracer.StartActiveSpanFromActivity($"mongodb.{message.Event.CommandName}",
        activity, 
        SpanKind.Client, 
        out var span);

    SetSpanAttributes(span, message);

    _spanMap.TryAdd(message.Event.RequestId, span);
}

From here, stopping the activity will mean, instead of us accessing the CurrentSpan, pulling it out of our dictionary:

public override void OnStopActivity(Activity activity, object payload)
{
    if (!(payload is CommandSucceeded message))
    {
        AdapterEventSource.Log.NullPayload("CommandListener.OnStopActivity");
        return;
    }

    if (_spanMap.TryRemove(message.Event.RequestId, out var span))
    {
        span.End();
    }
}

public override void OnException(Activity activity, object payload)
{
    if (!(payload is CommandFailed message))
    {
        AdapterEventSource.Log.NullPayload("CommandListener.OnExceptionActivity");
        return;
    }

    if (_spanMap.TryRemove(message.Event.RequestId, out var span))
    {
        span.Status = Status.Unknown.WithDescription(message.Event.Failure.Message);
        SetSpanAttributes(span, message);
        span.End();
    }
}

Although it took a bit more work to store the Activity and TelemetrySpan locally, doing so ensured that we correlated the correct instances for eventual publishing to our tracing adapters. If we only went with the Current properties, we'd be stopping the wrong activities and spans, resulting in a very wonky looking request graph.

The last piece is registering my subscriber with MongoDB, but it's quite specific to that driver so you can check out the source code to see how it's all registered and hooked up.

In the next (and last) post, I'll walk through my sample application and see how these adapters will be hooked up and used to visualize complicated traces in Zipkin, Jaeger, and Application Insights.

Building End-to-End Diagnostics: OpenTelemetry Integration

Thu, 07 May 2020 13:06:39 +0000

Posts in this series:

Source Code

In the last post, we looked at providing diagnostic event hooks into our code at specific points so that "something" could listen in. For our purposes, we want to listen in to surface telemetry data, but the uses are far wider for diagnostic events, such as logging, testing, and metrics. We want to focus on telemetry through the OpenTelemetry project.

So why OpenTelemetry? Just another standard we need to follow? As someone that uses NServiceBus on a wide variety of clients, I'm exposed to a number of different observability tools, to name a few:

Zipkin
Jaeger
Dynatrace
Application Insights

If you're a library author, like NServiceBus, and you want to enable observability through these tools, you'd have to create and maintain packages for each one of those products. Instead, if you could target some common API, like you can today for:

Logging
Dependency Injection
Configuration

Then you don't have to have some extensive matrix of support. You target one common API, and the implementation providers can plug in to that common model. OpenTelemetry is just this - providing a standardized API for tracing primitives (and eventually, much more). The .NET SDK for OpenTelemetry (in alpha at the moment, in beta for most other runtimes/platforms) provides that SDK, as well as convenient bridges for the diagnostics event API

Plugging into OpenTelemetry

The diagnostic events by themselves provide quite a bit of value, but we need to listen to them in order to do something. That's where OpenTelemetry comes in - the "something" listening to diagnostic events. Since OpenTelemetry .NET SDK is currently in alpha, I'm going to reference the OpenTelemetry MyGet repository and reference the OpenTelemetry alpha package:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="OpenTelemetry" Version="0.2.0-alpha.231" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\NServiceBus.Diagnostics\NServiceBus.Diagnostics.csproj" />
  </ItemGroup>

</Project>

This library I created serves as the listener to diagnostic events, as well as provide an extension to OpenTelemetry to register those listeners. But first, what do we need to create? With the current OpenTelemetry SDK, we need 3 things:

A ListenerHandler that receives Activity-based diagnostic events
An Adapter that subscribes a ListenerHandler to diagnostic events
Extension method to TracerBuilder that registers our Adapter with OpenTelemetry

ListenerHandlers

For NServiceBus, we have two kinds of activities - sending and processing messages. That means we'll have two telemetry events, two spans, and two listeners. The OpenTelemetry SDK includes a helper for dealing with the Activity/OpenTelemetry bridge, and that's a base ListenerHandler class. Note: this is all subject to change as it's all alpha, but the primitives defined in the OpenTelemetry Specification are at least in beta.

That base ListenerHandler class has methods to override - when an Activity starts, stops, raises an exception event, or any other custom event. For us, we only have start/stop events, so we can create a class that overrides those two methods:

internal class SendMessageListener : ListenerHandler
{
    public SendMessageListener(string sourceName, Tracer tracer) : base(sourceName, tracer)
    {
    }

    public override void OnStartActivity(Activity activity, object payload)
    {
        ProcessEvent(activity, payload as BeforeSendMessage);
    }

    public override void OnStopActivity(Activity activity, object payload)
    {
        ProcessEvent(activity, payload as AfterSendMessage);
    }

The payload is the argument fed into StartActivity in our diagnostic listener, allowing our OpenTelemetry listener to have an actual class to work with. The processing of those events needs to create OpenTelemetry spans from the activities, and add attributes to the span.

Our ProcessEvent method is a bit long, so let's break it up into parts:

private void ProcessEvent(Activity activity, BeforeSendMessage payload)
{
    if (payload == null)
    {
        AdapterEventSource.Log.NullPayload("SendMessageListener.OnStartActivity");
        return;
    }

    var span = StartSpanFromActivity(activity, payload);

    if (span.IsRecording)
    {
        SetSpanAttributes(activity, payload, span);
    }
}

If the payload wasn't recognized as what we expect, then we simply return. Next, we start an OpenTelemetry span from the activity. Finally, if we detect that the span is recording, then we apply the span attributes.

The StartSpanFromActivity is:

private TelemetrySpan StartSpanFromActivity(Activity activity, BeforeSendMessage payload)
{
    payload.Context.Headers.TryGetValue(Headers.MessageIntent, out var intent);

    var routes = payload.Context.RoutingStrategies
        .Select(r => r.Apply(payload.Context.Headers))
        .Select(t =>
        {
            switch (t)
            {
                case UnicastAddressTag u:
                    return u.Destination;
                case MulticastAddressTag m:
                    return m.MessageType.Name;
                default:
                    return null;
            }
        })
        .ToList();

    var operationName = $"{intent ?? activity.OperationName} {string.Join(", ", routes)}";

    Tracer.StartActiveSpanFromActivity(operationName, activity, SpanKind.Producer, out var span);
    return span;
}

I wanted to have a more meaningful name here, so I had to do a bit of munging of the incoming data to detect what "kind" of message we're sending here from the intent of the message (Send/Publish/Reply etc.). I also wanted to record where this operation was being sent to, so I also record the routes. The reason this is a little wonky is that NServiceBus is an abstraction over messaging, so I don't have access for example to the inner workings of RabbitMQ or Azure Service Bus or MSMQ.

The final result however is a Span with a kind of SpanKind.Producer started from the Activity we pass in. The basic guidance of setting the span and its attributes comes from the OpenTelemetry span conventions around messaging.

With the span started, we now want to apply the span attributes. Following the span guidance, we set several well-known attributes:

private static void SetSpanAttributes(Activity activity, BeforeSendMessage payload, TelemetrySpan span)
{
    span.SetAttribute("messaging.message_id", payload.Context.MessageId);
    span.SetAttribute("messaging.message_payload_size_bytes", payload.Context.Body.Length);

    span.ApplyContext(payload.Context.Builder.Build<ReadOnlySettings>(), payload.Context.Headers);

    foreach (var tag in activity.Tags)
    {
        span.SetAttribute($"messaging.nservicebus.{tag.Key.ToLowerInvariant()}", tag.Value);
    }
}

As well as fill in the span with any tags passed in through the Activity. Those extra tags translated into span attributes will allow the diagnostics hooks to add any extra details they want that can finally show up in telemetry events. There's a lot more attributes being set in the ApplyContext method, but it's very specific to NServiceBus internals.

Finally, to end the span, the ProcessEvent method takes in the AfterSendMessage payload but doesn't really do anything with it - the current Span already tracks the given Activity so we don't have anything additional to add in its data:

private void ProcessEvent(Activity activity, AfterSendMessage payload)
{
    Tracer.CurrentSpan.End();
}

The corresponding ProcessMessageListener is quite similar, and you can find that code over on GitHub. With our listeners in place, we now need the hooks into OpenTelemetry to register our listeners with its configuration.

Registering Our Listeners

The last two steps in OpenTelemetry integration are to provide the bridge from "what diagnostic events should I be listening to" to the ListenerHandler classes we created above. Listening to diagnostic events isn't the most straightforward thing in the world - it uses a global registry and a series of nested observables to subscribe to (more on that in a future post), and you need to be mindful to clean up your subscriptions to avoid memory leaks.

Luckily, OpenTelemetry .NET SDK has this nailed down for us, we just need to create a disposable class that uses a DiagnosticSourceSubscriber to listen to specific named diagnostic events. Here's the one for the "Send" messages:

public class NServiceBusSendAdapter : IDisposable
{
    private readonly DiagnosticSourceSubscriber _diagnosticSourceSubscriber;

    public NServiceBusSendAdapter(Tracer tracer)
    {
        _diagnosticSourceSubscriber = new DiagnosticSourceSubscriber(
            new SendMessageListener("NServiceBus.Diagnostics.Send", tracer),
            null);
        _diagnosticSourceSubscriber.Subscribe();
    }

    public void Dispose() 
        => _diagnosticSourceSubscriber?.Dispose();
}

We create an instance of the DiagnosticSourceSubscriber, which is the main helper for bridging diagnostic events and activities, and pass in an instance of our SendMessageListener class we saw earlier. The name of the source, NServiceBus.Diagnostics.Send, is the root name of our diagnostic event.

When the diagnostic events get raised, they are then suffixed with "Start" and "Stop" in their name, and the base listener class uses that convention to call the OnStart and OnStop methods. If you compare this code with say, the Azure SDK code for listening to diagnostic events, ours is MUCH easier to make sense of.

Finally, we need to create an extension to OpenTelemetry configuration for our adapters to register them with OpenTelemetry:

public static class TraceBuilderExtensions
{
    public static TracerBuilder AddNServiceBusAdapter(this TracerBuilder builder)
        => builder
            .AddAdapter(t => new NServiceBusReceiveAdapter(t))
            .AddAdapter(t => new NServiceBusSendAdapter(t));
}

With that in place, we can now register our adapters in our application startup. But before we get to that, what happens when our underlying infrastructure does not play nicely with this model? In the next post, I'll walk through adding telemetry to the Mongo .NET driver, where async context is not preserved and we have to do some more work to correlate diagnostic events.

Building End-to-End Diagnostics and Tracing: Diagnostic Events

Mon, 20 Apr 2020 15:28:47 +0000

Posts in this series:

In the last post, we looked at the W3C Trace Context standard, and how we can extend NServiceBus to propagate these new headers using middleware. With many monitoring and observability tools, this can be enough for tracing tooling to stitch together the entire request timeline.

These tools rely on agents installed in your environments, either as background services or container sidecars to "listen in" on your traffic and report back to some centralized reporting store that we can then use to view traces. Although this approach can seem a bit intrusive, it does not require any changes to existing code.

One disadvantage to this approach is that it can be difficult to report more fine grained metrics - things like:

How long did this operation take?
What was the nature of this communication?
What was the cycle time/processing time?
Did this operation complete successfully? Was it retried? How many times?

For more complex observability scenarios, merely attaching headers to requests isn't going to suffice.

Instead, we need to surface diagnostics out of our system into "something". Eventually, that "something" will be OpenTelemetry, but first, we need to surface diagnostics.

Diagnostics Events

With the Activity API, we have a central place where we can store diagnostic information across requests, but the Activity API can do quite a bit more. We want to surface diagnostic information out to OpenTelemetry, but I don't want to tie my middleware directly to one kind of telemetry listener. Even though OpenTelemetry is an emerging standard, I'd like some indirection from my events to OpenTelemetry so that any tool that cares about diagnostic events can listen in to what's going on.

Similar to Activity, we need a fundamental component to expose diagnostic events. With .NET Core, that component is a DiagnosticListener. A DiagnosticListener exposes an observable (rather than pure .NET events) so that interested subscribers can listen in to diagnostic events, completely decoupled from our code raising the events.

Of course, we won't raise events in a vacuum - we actually need to design what these events look like.

In the .NET Core codebase, one can find many different examples of raising diagnostic events:

You can find others in the EF Core codebase, and some Azure SDKs as well. The general idea is we create a listener with a well-known name, and write to the listener at appropriate times (typically, when our activity starts/stops).

The Activity API and diagnostic listener API roughly correspond to the OpenTelemetry, but there is an open GitHub issue for .NET 5 to strengthen this bond.

Raising a diagnostic event is fairly straightforward, we call the Write method:

 _listener.Write("EventName", <event object>);

Now we need to figure out two things - what to call our event and what that event object should be!

Diagnostics Events

In some usages of raising the diagnostic events inside the .NET Core codebase, there wasn't any correlation between the activity name and event name, and worse - that event object was an anonymous type!

_listener.Write("My Cool Event", new {
    SomeValue = someValue,
    AnotherValue = anotherValue
});

This causes major headaches with consumers of the event who only receive something of type object but have to use reflection instead of casting to get the information out.

With our events, I don't want to fall into that trap, we want our consumers to have

Some type that they can cast the object to
That type should include all available information

We can find some more guidelines in the Diagnostic Source Users Guide, which recommends how we design and use diagnostic listeners and events.

We also need to decide what interesting points we want to raise the diagnostic events, but that's somewhat straightforward for us - when the activities we create start and stop. We can always add other diagnostic events in the future if we like. With that in mind, we can create two consumer and two producer events, representing the start/stop of the activity:

public class BeforeSendMessage
{
    public const string EventName = Constants.ProducerActivityName + ".Start";

    public BeforeSendMessage(IOutgoingPhysicalMessageContext context) 
        => Context = context;

    public IOutgoingPhysicalMessageContext Context { get; }
}

public class AfterSendMessage
{
    public const string EventName = Constants.ProducerActivityName + ".Stop";

    public AfterSendMessage(IOutgoingPhysicalMessageContext context)
        => Context = context;

    public IOutgoingPhysicalMessageContext Context { get; }
}

public class BeforeProcessMessage
{
    public const string EventName = Constants.ConsumerActivityName + ".Start";

    public BeforeProcessMessage(IIncomingPhysicalMessageContext context)
        => Context = context;

    public IIncomingPhysicalMessageContext Context { get; }
}

public class AfterProcessMessage
{
    public const string EventName = Constants.ConsumerActivityName + ".Stop";

    public AfterProcessMessage(IIncomingPhysicalMessageContext context) 
        => Context = context;

    public IIncomingPhysicalMessageContext Context { get; }
}

Rather than trying to parse and pluck individual pieces of information out of the currently executing context, I just include the entire Context of whatever behavior is executing. I don't have any additional information while processing, but if I did, I'd likely leverage the Activity instead of customizing the event message.

With the events defined, now I can move on to declaring and writing diagnostic events in my middleware.

Producing diagnostic events

Following the diagnostic source user guidelines, each of our middleware behaviors needs to declare a DiagnosticSource:

public class ConsumerDiagnostics : Behavior<IIncomingPhysicalMessageContext>
{
    private static readonly DiagnosticSource _diagnosticListener 
        = new DiagnosticListener(Constants.ConsumerActivityName);

I use the same name for my activity as I do for the listener name. Individual diagnostic events written by that DiagnosticSource will then have that name prepended to the event name, so I'll have NServiceBus.Diagnostics.Receive as my activity name and NServiceBus.Diagnostics.Receive.Start and NServiceBus.Diagnostics.Receive.Stop as the diagnostic event names.

With my listener created, I want to write the start and stop events where I used to start my Activity. However, the usage guidelines recommend only raising a diagnostic event when there are listeners to avoid the overhead. I'll use the StartActivity method as a convenience to start my activity if there is a listener:

if (_diagnosticListener.IsEnabled(BeforeProcessMessage.EventName, context))
{
    _diagnosticListener.StartActivity(activity, new BeforeProcessMessage(context));
}
else
{
    activity.Start();
}

The StartActivity and StopActivity methods on DiagnosticSource will take care of the work of starting/stopping the activity and raising the diagnostic event, named based on the Activity operation name and adding ".Start" or ".Stop" to the end.

Ending the activity now delegates to the diagnostics listener:

private static void StopActivity(Activity activity,
    IOutgoingPhysicalMessageContext context)
{
    _diagnosticListener.StopActivity(activity, new AfterSendMessage(context));
}

With this in place, I've now got diagnostic events firing as part of my middleware, and any interested listeners can pick those events up and do whatever they want!

Right now, I'm only interested in listening to diagnostic events for telemetry purposes, but diagnostic events are also useful for logging, debugging, testing, and anything else where I want to peak "under the hood" of whatever is going on.

In the next post, we'll look at leveraging the OpenTelemetry SDK (in alpha) to create listeners and collectors for these new events.

Building End-to-End Diagnostics and Tracing: Trace Context

Tue, 07 Apr 2020 17:11:23 +0000

Posts in this series:

In the last post, I walked through the overall problem we run into with diagnosing issues in distributed systems - mainly that it can be difficult to determine causality because we don't have that "stack trace" with a single in-process application.

To create a sort of "trace" in a distributed system, we need some way to build breadcrumbs into our communications. When one system communicates with another, and that system calls another, we need some way to link those requests together:

Distributed systems communicating

In a downstream system that experiences a failure, how do we link that failure to the original request, and all between? This is where distributed tracing comes in.

Product companies and OSS filled the void, but there became a problem - each product, OSS or not, had its own way of providing additional context to each request to be able to link them together. The solution to causality is rather simple - we just need some context of the parent system/process that initiated the current request. That context is as simple as providing some unique identifier for the current request to all subsequent requests.

Very recently (February 2020), a new W3C standard exited "draft" status entered the "recommendation", Trace Context. This standard describes mainly:

What is the ID of the current request?
What is the ID of the parent request?

It also allows requests to include some state information, but most important are those identifications. With an ID and parent ID, we can now create a directed acyclic graph, very similar to what we would see in a Git commit history.

Trace Context in .NET Core

In order to flow tracing identifiers through a request pipeline, regardless of the technology of the "in" and "out" request, we need some means of capturing the incoming tracing identifiers (on headers), storing them, and flowing them to outgoing headers. The basic pieces for this flow are:

Incoming requests pull trace identifiers and store in an Activity
Activity.Current includes any additional information for the current activity
Outgoing requests read information from Activity.Current and place on outgoing trace headers.

One of the big pushes for observability in .NET Core 3.0 was to enable this W3C standard. Although it's not turned on by default for backwards compatibility reasons, if you turn it on:

public class Program
{
    public static void Main(string[] args)
    {
        Activity.DefaultIdFormat = ActivityIdFormat.W3C;

        CreateHostBuilder(args).Build().Run();
    }

That will use the W3C standards for identifiers, but we still need to consume and propagate these trace identifiers. Luckily, we can see how ASP.NET Core and HttpClient did this:

There's a lot more going on in that code that we'll get to soon, but first things first, we need middleware for NServiceBus to:

Start an Activity for incoming requests and set its parent ID from the traceparent header
Set the traceparent header for outgoing requests

Luckily for us, NServiceBus has a robust middleware API that makes it easy for us to add these pieces, Behaviors.

Incoming Requests to Activity

The first step in the process for diagnostics is to start an Activity at the beginning of processing a message and stop it at the end. We need to go one step further to add the appropriate parent ID. We can do this with a behavior defined for incoming messages:

public class ConsumerDiagnostics 
    : Behavior<IIncomingPhysicalMessageContext>
{
    public override async Task Invoke(
        IIncomingPhysicalMessageContext context
        Func<Task> next)
    {
        var activity = StartActivity(context);

        try
        {
            await next().ConfigureAwait(false);
        }
        finally
        {
            StopActivity(activity, context);
        }
    }

Behaviors in NServiceBus are similar to ASP.NET Core middleware. You get two parameters, the first being the context of the operation performed, and the second delegate to perform the next action in the chain.

The StartActivity method needs to do two things - start an Activity, and set pull the traceparent header off the incoming message:

private static Activity StartActivity(IIncomingPhysicalMessageContext context)
{
    var activity = new Activity(Constants.ConsumerActivityName);

    if (!context.MessageHeaders.TryGetValue(
        Constants.TraceParentHeaderName, 
        out var requestId))
    {
        context.MessageHeaders.TryGetValue(
            Constants.RequestIdHeaderName, 
            out requestId);
    }

    if (!string.IsNullOrEmpty(requestId))
    {
        // This is the magic 
        activity.SetParentId(requestId);
        
        if (context.MessageHeaders.TryGetValue(
            Constants.TraceStateHeaderName, 
            out var traceState))
        {
            activity.TraceStateString = traceState;
        }
    }

    // The current activity gets an ID with the W3C format
    activity.Start();

    return activity;
}

We first create an activity with a good name, in my case I chose NServiceBus.Diagnostics.Receive. There's not a ton of recommendations about naming activities, but it should be something meaningful to the overall operation that's being performed. Activity names are hierarchical for future purposes, so we want to adhere to some sort of namespacing. The ASP.NET Core name is Microsoft.AspNetCore.Hosting.HttpRequestIn and HttpClient is System.Net.Http.HttpRequestOut.

After creating the Activity, I try to pull the traceparent header value out. I'm also trying to be a good citizen and pull the older, previous request-id header value out. Once i have this, I can set the ParentId on the Activity. Finally, if it exists, I'll pull the tracestate value and stuff it into the Activity. There's some more things in store for distributed tracing related to additional correlation context items, but for now, I'll leave that alone.

Finally, I start the activity, and Activity.Current represents this new activity. Stopping the activity is straightforward - the only thing I really need to care about is setting an end time of the Activity:

private static void StopActivity(Activity activity,
    IIncomingPhysicalMessageContext context)
{
    if (activity.Duration == TimeSpan.Zero)
    {
        activity.SetEndTime(DateTime.UtcNow);
    }

    activity.Stop();
}

Setting an appropriate end time for the activity will mean more later on when we start raising diagnostic events, but we want to make sure the duration of the event is just around calling the next item in the pipeline.

That's incoming requests, what about outgoing ones?

Propagating trace context in outgoing messages

Just like we have incoming behaviors for messages, NServiceBus has outgoing behaviors as well. We just need to reverse the flow from above - set the traceparent header on outgoing messages from the current Actvity:

public class ProducerDiagnostics : Behavior<IOutgoingPhysicalMessageContext>
{
    public override async Task Invoke(
        IOutgoingPhysicalMessageContext context, 
        Func<Task> next)
    {
        var activity = StartActivity(context);

        InjectHeaders(activity, context);

        try
        {
            await next().ConfigureAwait(false);
        }
        finally
        {
            StopActivity(activity, context);
        }
    }

Starting the activity is much simpler now:

private static Activity StartActivity(IOutgoingPhysicalMessageContext context)
{
    var activity = new Activity(Constants.ProducerActivityName);

    activity.Start();

    return activity;
}

But wait, we're not setting the parent ID! For outgoing messages, we don't need to. If there's a current started activity, our activity will automatically have its ParentId set to Activity.Current.Id, so we're good to go without managing all that ourselves.

Next, we need to inject the headers of the current activity into the outgoing request:

private static void InjectHeaders(
    Activity activity,
    IOutgoingPhysicalMessageContext context)
{
    if (activity.IdFormat == ActivityIdFormat.W3C)
    {
        if (!context.Headers.ContainsKey(Constants.TraceParentHeaderName))
        {
            context.Headers[Constants.TraceParentHeaderName] = activity.Id;
            if (activity.TraceStateString != null)
            {
                context.Headers[Constants.TraceStateHeaderName] =
                    activity.TraceStateString;
            }
        }
    }
    else
    {
        if (!context.Headers.ContainsKey(Constants.RequestIdHeaderName))
        {
            context.Headers[Constants.RequestIdHeaderName] = 
                activity.Id;
        }
    }
}

The new request's parent ID will be this activity's ID, and that new parent ID will be consumed by downstream systems as well.

The magic here is Activity.Current, an async local static property that means that anything sharing the same async context will get the same Activity.Current value.

Stopping the activity looks exactly the same as the incoming requests:

private static void StopActivity(
    Activity activity, 
    IOutgoingPhysicalMessageContext context)
{
    if (activity.Duration == TimeSpan.Zero)
    {
        activity.SetEndTime(DateTime.UtcNow);
    }

    activity.Stop();
}

To enable these behaviors, you can use NServiceBus Features to add these behaviors to the processing pipeline automatically:

public class DiagnosticsFeature : Feature
{
    public DiagnosticsFeature()
    {
        EnableByDefault();
    }

    protected override void Setup(FeatureConfigurationContext context)
    {
        context.Pipeline.Register(new ConsumerDiagnostics(), 
            "Parses incoming W3C trace information from incoming messages.");
        context.Pipeline.Register(new ProducerDiagnostics(), 
            "Appends W3C trace information to outgoing messages.");
    }
}

I enable this feature by default, with the future idea that anyone that references this package/assembly will get this behavior opted in. With all of this in place, how does this look in practice?

A Dummy Distributed System

I wanted to simulate all these different kinds of flows, which use a variety of hosts and communication:

Incoming HTTP to Outgoing NServiceBus
Incoming NServiceBus to Outgoing HTTP
Incoming NServiceBus to Outgoing NServiceBus

Incoming HTTP will be a regular ASP.NET Core application and host, and the incoming NServiceBus will be a worker service. I wanted to capture all manners of communication with these two applications:

Dummy distributed system communicating via HTTP and AMQP

My Web Server is a web application with this diagnostics code added, plus using the NServiceBus extension to .NET Core generic hosting. I created a simple API controller that uses the NServiceBus IMessageSession injected to send an AMQP message via RabbitMQ:

[HttpGet]
public async Task<ActionResult> Get(string message)
{
    var command = new SaySomething
    {
        Message = message
    };

    _logger.LogInformation("Sending message {message}", command.Message);

    await _messageSession.Send(command);

    return Accepted();
}

The handler of this message on the worker service makes the HTTP call and a Reply:

public class SaySomethingHandler : IHandleMessages<SaySomething>
{
    private readonly ILogger<SaySomethingHandler> _logger;

    private static readonly HttpClient _httpClient = new HttpClient
    {
        BaseAddress = new Uri("https://localhost:5001")
    };


    public SaySomethingHandler(ILogger<SaySomethingHandler> logger) 
        => _logger = logger;

    public async Task Handle(SaySomething message, IMessageHandlerContext context)
    {
        var content = await _httpClient.GetStringAsync("/weatherforecast/today");

        dynamic json = Deserialize<ExpandoObject>(content);

        var temp = (int)json.temperatureF.GetInt32();

        _logger.LogInformation("Saying {message} and the weather today is {weather}F", message.Message, temp);

        await context.Reply(new SaySomethingResponse
        {
            Message = $"Back at ya {message.Message}"
        });
    }
}

The API endpoint is rather dumb, it's the weather dummy data that I stuck in a database:

[HttpGet("today")]
public async Task<WeatherForecast> GetToday()
{
    var forecastCount = await _dbContext.Forecasts.CountAsync();

    var rng = new Random();

    return await _dbContext.Forecasts.Skip(rng.Next(forecastCount)).FirstAsync();
}

And the Reply handler doesn't do anything fun, but it stops the distributed flow:

public class SaySomethingResponseHandler 
    : IHandleMessages<SaySomethingResponse>
{
    private readonly ILogger<SaySomethingResponseHandler> _logger;

    public SaySomethingResponseHandler(ILogger<SaySomethingResponseHandler> logger)
        => _logger = logger;

    public Task Handle(SaySomethingResponse message, IMessageHandlerContext context)
    {
        _logger.LogInformation("Received {message}", message.Message);

        return Task.CompletedTask;
    }
}

With all the pieces in place, let's trace the flow from the initial request all the way through each receiver and out again.

Tracing the flow

It all starts with initiating the request with the Swagger UI:

Initial request in Swagger UI

Logging the Activity.Current.Id and Activity.Current.ParentId:

Initial request activity ID and parent ID

We see that the current activity ID has a value, but the parent ID does not. This makes sense - the Swagger UI doesn't track activities and does not pass a traceparent header along.

With the message sent, let's look at the message in RabbitMQ to see if it has a traceparent value that matches the above:

RabbitMQ message with matching traceparent header value

It does! Let's now run the whole system end-to-end and watch the activity IDs in our logs:

Aggregate logs for both web and worker service with activity IDs

We can see that all of our activity IDs share the same trace-id fragment, while the parent-id values differ (technical detail, but these link to a span in tracing terms).

With our tracing identifiers correctly propagating, we've laid the groundwork to start putting humpty dumpty together again. In the next post, we'll look at how we can raise diagnostic events so that something can see these traces outside of directly instrumenting our traffic.

Building End-to-End Diagnostics and Tracing: An Intro

Wed, 01 Apr 2020 15:23:15 +0000

As microservices introduced (forced) highly complex distributed systems into organizations, the tools required to operate these architectures needed to evolve as well. What was a simple stack trace in a single in-process monolith became a web of network calls.

In my first large-scale distributed system, well before the term "microservice" was coined, we hit the problem of operating complex distributed systems almost the second the entire system was turned on. When something went wrong, it could take hours to track down the issue. My old talk on Avoiding Microservices Megadisasters goes into one of these stories - that it took 2 weeks just to figure out how a request was stitched together.

Since then, diagnostics and tracing have come a long way. In this series, I want to walk through adding diagnostics and tracing to a library I've used quite a lot over the years - NServiceBus. Based on those techniques, we can add diagnostics and tracing to any network-communicating library or component.

The Overall Problem

In a single in-process application, when something goes wrong, you have a stack trace telling you exactly where in the system an exception occurred. But if you've got distributed systems communication with each other, it's not enough to have a stack trace of a single application. Often, we need to understand causality all the way back out to the original external trigger or event that led to a fault.

The solution to this problem is "distributed tracing". Instead of having a single call stack, we connect multiple call stacks together by introducing some additional tracing metadata between each node.

Over the years, many tools and products arose to fill this niche. I've used a few, and built a few, but with each new tool rose a new means to plug it in.

If I wanted to use Dynatrace, I needed to have Dynatrace plugins to everything I used. If I wanted to use Zipkin, the same. And if those plugins didn't exist for whatever library I was using, I needed to build that myself. Each tool had its own way of providing its tracing context. Zipkin has its own, and NServiceBus has its own, and some don't have anything.

This is where standards come in - to provide a common way of:

Identifying and propagating tracing information
Raising diagnostic event notifications
Reporting diagnostic telemetry information

NServiceBus has a very robust distributed tracing mechanism and reporting tool with ServiceInsight, however, similar to Zipkin/Jaeger/Prometheus etc., it's using proprietary means of doing so and doesn't directly plug in to any other reporting tool or network communication.

The Plan

In order to make any new network component "play nice" with distributed tracing, a few things need to happen:

All incoming network traffic needs to capture tracing information
All outgoing network traffic needs to propagate tracing information
Any interesting diagnostic event needs to be emitted
Diagnostic events raised need to be captured and re-emitted as telemetry

In this series, I'll walk through each of these steps, the standards applied, and middleware needed to connect all the pieces together. In the end, we'll have a complete picture of a distributed system that uses ASP.NET Core, HttpClient, and RabbitMQ, and SQL together in a single picture:

Distributed trace connecting ASP.NET Core, HttpClient, RabbitMQ, and SQL

NServiceBus and .NET Core Generic Host

Mon, 23 Mar 2020 13:19:53 +0000

My current client is using .NET Core 2.x, with plans to upgrade to 3.x next month. As part of that system, we do quite a bit of messaging, with NServiceBus as the tool of choice to help make this easier. To get it working with our .NET Core 2.x applications, we did quite a bit of what I laid out in my Messaging Endpoints in Azure series.

Since then, NServiceBus released first-class support for the .NET Core Generic Host, which underwent a fairly large refactoring in the 2.x to 3.0 timeframe. Andrew Lock's post goes into more detail, but the gist of it is, NServiceBus has first-class support for .NET Core 3.x and later.

What that means for us is hosting NServiceBus inside a .NET Core application couldn't be easier. The NServiceBus.Extensions.Hosting package provides all the integration we need to add a hosted NServiceBus service and integrate with the built-in DI container.

Configuring NServiceBus

With any kind of hosted .NET Core application (Console, ASP.NET Core, Worker), we just need to add the extensions package:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NServiceBus.Extensions.Hosting" Version="1.0.0" />

And add the configuration directly off of the host builder:

Host.CreateDefaultBuilder(args)
    .UseNServiceBus(hostBuilderContext =>
    {
        var endpointConfiguration = new EndpointConfiguration("WebApplication");

        // configure endpoint here

        return endpointConfiguration;
    })
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
    });

Or with a Worker SDK:

<Project Sdk="Microsoft.NET.Sdk.Worker">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.2" />

It's really not much different:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .UseNServiceBus(hostBuilderContext =>
        {
            var endpointConfiguration = new EndpointConfiguration("WorkerService");

            // configure endpoint here

            return endpointConfiguration;
        });

And our endpoint is up and running.

Logging and Serialization

We're not quite there yet, though. The out-of-the-box serialization is XML (which is fine by me), but many folks prefer JSON. Additionally, the logging support inside of NServiceBus is not currently integrated with this package. For serialization, we can use the new System.Text.Json support instead of Newtonsoft.Json.

We'll pull in the community packages from Simon Cropp:

With those two packages in place, we can configure our host's serializer and logging:

Host.CreateDefaultBuilder(args)
    .UseMicrosoftLogFactoryLogging()
    .UseNServiceBus(hostBuilderContext =>
    {
        var endpointConfiguration = new EndpointConfiguration("WorkerService");

        endpointConfiguration.UseSerialization<SystemJsonSerializer>();

We now have integrated logging, hosting, and dependency injection with anything that uses the generic host.

Using the logger

Now in our handlers, we can add dependencies directly on the Microsoft logger, ILogger<T>:

public class SaySomethingHandler : IHandleMessages<SaySomething>
{
    private readonly ILogger<SaySomethingHandler> _logger;

    public SaySomethingHandler(ILogger<SaySomethingHandler> logger) 
        => _logger = logger;

    public Task Handle(SaySomething message, IMessageHandlerContext context)
    {
        _logger.LogInformation("Saying {message}", message.Message);

        return context.Reply(new SaySomethingResponse
        {
            Message = $"Back at ya {message.Message}"
        });
    }
}

And we get an integrated logging experience:

info: NServiceBus.LicenseManager[0]
      Selected active license from C:\Users\jbogard\AppData\Local\ParticularSoftware\license.xml
      License Expiration: 2020-06-16
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\jbogard\source\repos\NsbActivities\WorkerService
info: WorkerService.SaySomethingHandler[0]
      Saying Hello World!

Now with this logging and dependency injection integration, we can use any logger or container that extends the built-in abstractions. My current client (and most) use Serilog, which makes it very easy to plug in to the generic host as well.

With these packages, we'll be able to delete a lot of infrastructure code that wasn't adding any value, which is always a good thing.

Avoid In-Memory Databases for Tests

Wed, 18 Mar 2020 18:25:12 +0000

A controversial GitHub issue came to my attention a couple of weeks ago around ditching the in-memory provider for Entity Framework Core. This seemed like a no-brainer to me - these database providers are far from trivial to maintain, even for in-memory strategies. It's something our teams learned nearly a decade ago, that trying to swap out an in-memory strategy for unit testing simply doesn't provide the value folks may hope for.

It seems rather simple at first - especially in the .NET world and EF Core. EF Core's primary read API is LINQ. LINQ has two flavors - IEnumerable and IQueryable. With IQueryable, an IQueryProvider translates expression trees to...well whatever makes sense for the underlying store. There's a neat trick that you can do, however, as IEnumerable has a method, AsQueryable, to allow complex expression trees to evaluate directly against an in-memory IEnumerable.

Thus, in-memory queryables were born. So why not take advantage of this possibility for unit tests? Why not allow us to swap the implementation of some queryable to the in-memory equivalent, and allow us to write unit tests against in-memory stores?

It all seems so simple, but unfortunately, the devil is in the details.

Simple Ain't Easy

LINQ providers aren't easy. They're not even merely difficult, they're some of the most complex pieces of code you'll see. Why is that?

A LINQ provider is a compiler, a lexer, and a parser, but doesn't own any of those pieces. It's a transpiler, but instead of the output being text, it's API calls. You have to do similar operations as an actual compiler, dealing with ASTs, building your own AST (often), and figuring out how to make a very wide and complex API that is all the IQueryable surface area.

Any ORM maintainer can tell you - the code in the query provider can dwarf the code in the rest of the codebase. This is unlike other ORMs that provide a specialized API or language, such as NHibernate or the MongoDB C# driver.

LINQ surfaces a great amount of flexibility, but that flexibility makes it quite difficult to translate into SQL or any other database API that's already been specifically designed for that database. We're trying to wrap a fit-for-purpose API with a generic-purpose API, and one that can run either in-memory or translated into a database.

On top of that, a good deal of the LINQ surface area can't be translated into SQL, and there are Very Important things that don't translate into LINQ. So you have to extend IQueryable to do fun things like:

using (var context = new BloggingContext())
{
    var blogs = context.Blogs
        .Include(blog => blog.Posts)
            .ThenInclude(post => post.Author)
                .ThenInclude(author => author.Photo)
        .ToList();
}

Yikes! We also have async in the mix, so now we're at the point where the IQueryable isn't remotely the same for in-memory.

But that won't stop us from trying!

Pitfalls of In-Memory

Our teams tried a number of years ago to speed up integration tests by swapping in-memory providers, but we found a number of problems with this approach that led us to abandoning it altogether.

You MUST write a real integration test anyway

First and foremost, an in-memory provider is a pale imitation for the real thing. Even with writing in-memory tests, we still absolutely wrote integration tests against a real database. Those unit tests with in-memory looked exactly like the integration tests, just with a different provider.

Which led us to wonder - if we were writing the tests twice, what was the value of having two tests?

You could write a single test codebase, and run it twice - one with in-memory and one with the real thing, but that has other problems.

You MUST allow raw access to the underlying data API

ORMs allow you to encapsulate your data access, which is good, it allows us to be more productive by focusing on the business problem at hand. But it's also bad because it abstracts your data access, leaving developers to assume that they don't actually need to understand what is going on behind the scenes.

In our projects, we take a pragmatic approach - use the ORM's API when it works, and drop down to the database API when it becomes painful. ORMs these days make it quite easy, such as EF Core's raw SQL capabilities:

var blogs = context.Blogs
    .FromSqlRaw("SELECT * FROM dbo.Blogs")
    .ToList();

There are numerous limitations, many more than with EF6, which is why we often bring in a tool like Dapper to do complex SQL:

var employeeHierarchy = connection.Execute<EmployeeDto>(@"WITH cte_org AS (
    SELECT       
        staff_id, 
        first_name,
        manager_id
        
    FROM       
        sales.staffs
    WHERE manager_id IS NULL
    UNION ALL
    SELECT 
        e.staff_id, 
        e.first_name,
        e.manager_id
    FROM 
        sales.staffs e
        INNER JOIN cte_org o 
            ON o.staff_id = e.manager_id
)
SELECT * FROM cte_org;");

So how do we handle this scenario in our tests? Don't write the unit test (this assumes we're actually writing tests twice)? Somehow exclude it?

What I tend to find is that instead of dropping down to SQL, developers avoid SQL just so that it satisfies the tool. This is unacceptable.

The APIs don't match

The in-memory API of vanilla IQueryProvider doesn't match the LINQ query provider. This means you'll have methods that don't make sense, are no-ops, or even nonsensical for in-memory.

The most obvious example is Include, which instructs the LINQ provider to basically do a join to eagerly fetch some child records. This is to avoid multiple round trips. However, this means nothing to in-memory. You can keep it, remove it, add more, remove more, doesn't matter.

It gets worse on the flip side - when LINQ provides some APIs that aren't supported by the query provider. Since LINQ can run in-memory, it can execute anything on the client side. But when you try to run anything on the server, that won't work:

var blogs = context.Blogs
    .Where(blog => StandardizeUrl(blog.Url).Contains("dotnet"))
    .ToList();

Instead, LINQ providers allow a narrow subset of methods, and even beyond that, a limited set of core .NET methods to translate on the server. But not all obvious methods, and not even all overloads are supported. You don't know this until you actually run the LINQ query against the enormous LINQ provider.

Databases Means Transactions

If I look at a typical integration test we write, we're using both the public and non-public API in a series of transactions to interact with the system under test. Here's a typical example:

[Fact]
public async Task Should_edit_department()
{
    var adminId = await SendAsync(new CreateEdit.Command
    {
        FirstMidName = "George",
        LastName = "Costanza",
        HireDate = DateTime.Today
    });

    var admin2Id = await SendAsync(new CreateEdit.Command
    {
        FirstMidName = "George",
        LastName = "Costanza",
        HireDate = DateTime.Today
    });

    var dept = new Department
    {
        Name = "History",
        InstructorId = adminId,
        Budget = 123m,
        StartDate = DateTime.Today
    };
    await InsertAsync(dept);

    Edit.Command command = null;
    await ExecuteDbContextAsync(async (ctxt, mediator) =>
    {
        var admin2 = await FindAsync<Instructor>(admin2Id);

        command = new Edit.Command
        {
            Id = dept.Id,
            Name = "English",
            Administrator = admin2,
            StartDate = DateTime.Today.AddDays(-1),
            Budget = 456m
        };

        await mediator.Send(command);
    });

    var result = await ExecuteDbContextAsync(db => db.Departments.Where(d => d.Id == dept.Id).Include(d => d.Administrator).SingleOrDefaultAsync());

    result.Name.ShouldBe(command.Name);
    result.Administrator.Id.ShouldBe(command.Administrator.Id);
    result.StartDate.ShouldBe(command.StartDate.GetValueOrDefault());
    result.Budget.ShouldBe(command.Budget.GetValueOrDefault());
}

It's long, but it combines both public APIs (sending commands to create items) and non-public APIs (interacting directly with the DbContext to insert rows), executing an individual command for the test, then finally querying to pull an item out.

In integration tests of long ago, we'd put this entire set of operations in a transaction/unit of work. That's not at all how the application behaves, however, and we'd see many false positives that would only break when each operation was distinct. This is because ORMs use patterns like Unit of Work and Identity Map to determine what to persist and when.

With in-memory providers, there is no "ACID", everything is immediately durable. Each operation is immediately performed, and transactions do nothing! It might seem like a trivial thing, who cares if everything is always immediately durable? The problem is, just like an integration test that uses a single transaction, is that real-life behavior is much different and more complex, and will break in ways you can't predict. Enough false positives, and you wind up distrusting these unit tests.

The database enforces constraints and visibility and isolation levels that these attempts can't, and inevitably, you'll hit problems

But it's working for me!

Great! You're one of the lucky few. Your usage is trivial enough that can fit into the constraints of an in-memory provider. We've tried this (and other in-memory DBs, like SQLite), and it's always failed.

Unfortunately for the EF team, maintaining this provider for public consumption is a cost for them, and a tradeoff. They're coding that instead of coding something else. The question becomes - is the value of a (always flawed) in-memory provider worth the effort for the team?

For me, no, it's not worth negative effects for our team.

Document-Level Pessimistic Concurrency in MongoDB

Mon, 16 Mar 2020 12:28:46 +0000

In the last post, I got quite a few comments on some other ways to approach OCC. One pointed out that I wanted to explore was using a "SELECT...FOR UPDATE", which basically will grant an intent lock on the document in question. With an intent lock, we transition from optimistic concurrency, where we read a document and assume that others won't modify it, but check anyway, to pessimistic concurrency, where we know that our document will be modified by others.

Document-level locks in MongoDB can be granted in single operations or as part of a transaction. Intent locks can be granted as part of a write operation, or at the start of a transaction. Both are slightly different in their approach, one requiring an explicit write, and the other for the entire transaction, so let's look at each approach.

SELECT ... FOR UPDATE

In this approach, we want to trigger an intent exclusive lock (IX) on our first read of the document when we pull it out. The link above describes an approach with the bare API, so we can translate this into the C# version.

The general idea here is that MongoDb itself doesn't support this sort of "intentional" locking, but we can trigger it by updating some field to some new value. If another process has already locked that document, we'll immediately error out. In my case, we can just invent some field, I called it "ETag" to mirror Cosmos DB:

public class Counter
{
    public Guid Id { get; set; }
    public int Version { get; set; }
    public Guid ETag { get; set; }

    public int Value { get; set; }
}

When we first load up the document, we don't just load - we find-and-update that ETag property to some new value, BUT, inside a transaction:

using var session = await client.StartSessionAsync().ConfigureAwait(false);

var transactionOptions = new TransactionOptions(readConcern: ReadConcern.Local, writeConcern: WriteConcern.W1);

session.StartTransaction(transactionOptions);

try
{
    var update = Builders<Counter>.Update.Set(c => c.ETag, Guid.NewGuid());

    var loaded = await collection.FindOneAndUpdateAsync(session, c => c.Id == id, update);

    loaded.Value++;

    result = await collection.ReplaceOneAsync(session, c => c.Id == id, loaded, new UpdateOptions { IsUpsert = false });

    await session.CommitTransactionAsync();
}
catch
{
    await session.AbortTransactionAsync();
    throw;
}

return result;

In that first write operation, or trivial change locks the document for the duration of the transaction. After we perform whatever operation we need to on the document, we write back as normal - no version checking.

When we do this, our throughput drops fairly drastically - since we're locking ahead of time, we disallow any other concurrent write operations. Another concurrent write at the beginning of the operation will simply error out.

In practice, we'd likely want to have some sort of transparent retry mechanism around our operation - especially in scenarios where we're likely to see write collisions. We'd also likely want to introduce some jitter or randomness in our delays, since two operations retrying at the same time will likely collide again.

It can get fairly complicated, which is why you'd only want to introduce this in scenarios where optimistic locking isn't appropriate or viable. In fact, you can see an example of this in action in the NServiceBus MongoDB storage library - it's designed for concurrent operations, and uses pessimistic locking to do so.

Wrapping it up

With optimistic and pessimistic locking solutions possible, we see the flexibility of the MongoDB API. However, I do wish that it were easier and more explicit to call out locking strategies. This might be possible with other APIs on top of the MongoDB client, but as others have pointed out, you're far less likely to "foot-gun" yourself with a client API that encapsulates all of this for us.

Immutability in DTOs?

Thu, 27 Feb 2020 14:11:00 +0000

Something that comes up every so often is the question of whether or not Data Transfer Objects should be immutable - that is, should our design of the classes and types of a DTO enforce immutability.

To answer this question, we first need to look at what purpose a DTO serves. As the name explicitly calls out, it is an object that is used to carry data between processes. In practice, we don't literally transport an object back and forth between processes, and instead there is some form of serialization.

Where the waters get muddied is when types are used to describe message contracts - whether in asynchronous communication in the form of messaging or APIs. Given our usage of DTOs these days, why might we want to enforce immutability?

Benefits of immutability

The primary benefit of immutability is, well, you can't change the object! That makes a lot of sense on the receiving side of a communication - I shouldn't be able to change the message, even it's a deserialized version of that message.

There are some cases where I might want to be able to change a message as it flows through the system. For example, with a Document Message, I often pass the same message between many endpoints, and each endpoint processes the message and adds its own information as it goes along, sometimes with an attached Routing Slip.

This is rarer case, however, and what often winds up happening is I'm not mutating the original message, but the deserialized copy of the message, arriving in the form of the object.

By far the majority of cases are the receiver should not modify the message upon receipt. Sounds great! So why don't we actually do this?

Immutability in a mutable world

The main issue with immutability is that it depends on your language, platform, and even tooling, to support this.

In my systems, the default message content type is application/json, and the senders/receivers are .NET/C# or JavaScript/TypeScript. The main hurdle is that neither of these languages, or the platforms and tooling, support immutability out-of-the-box.

In order to declare an Order message as immutable, it's fairly complex. You have to make sure all write paths are initialized on construction:

public class Order {
  public Order(Customer customer, List<LineItem> lineItems) {
    Customer = customer;
    LineItems = new ReadOnlyCollection<LineItem>(lineItems);
  }
  public Customer Customer { get; }
  public IReadOnlyCollection<LineItem> LineItems { get; }
}
public class Customer {
  public Customer (string firstName, string lastName) {
    // my fingers are tired
  }
  public string FirstName { get; }
  public string LastName { get; }
  // etc
}
public class LineItem {
  public class LineItem(int quantity, decimal total, Product product) {
    // mommy are we there yet
  }
  public decimal Total { get; }
  public int Quantity { get; }
  public Product Product { get; }
}

Because C# doesn't have first-class support for an "immutable" type (like an F# Record Type), it takes a LOT of typing to build out these structures.

When you're done, you'll find that constructing these types is a huge pain. You'll get very long construction declarations:

var order = new Order(
   new Customer(
     "Bob",
     Saget"),
   new {
     new LineItem(10, 100m,
       new Product(
       )
   }
 );

Gross! How do we know what value corresponds to which property? We can add named arguments to make it readable, but those are optional, and still don't correspond to the actual property names (all camelCase). Typically you see no parameter names, making it quite difficult to understand how this all fits together.

Contrast this with a normal C# Object Initializer statement:

var order = new Order {
  Customer = new Customer {
    FirstName = "Bob",
    LastName = "Saget"
  },
  LineItems = new {
    new LineItem {
      Quantity = 10,
      Total = 100m,
      Product = new Product {
        
      }
    }
  }
};

I can see exactly how the entire object is built, one member at a time.

One other major issue with immutability in C# is the support for serializers and deserializers to work is quite a pain. If these tools see something without a setter, that's a problem. If we don't have a no-arg constructor, that's a problem. So we wind up having to bend the tools to be able to handle our types, and very often, with lots of compromises (private, unused setters etc.).

Finally, the nail in the coffin for me, is that when we introduce immutability via constructors, this introduces a breakable contract - a method with fixed arguments. If we add a value to our message, we introduce the possibility that receivers won't be able to properly deserialize.

Given all this, I find it's generally a net negative to attempt immutable DTOs. If the language, framework, and tooling supported it, that would be a different story.

Getting started with Cassandra: Data modeling in the brief

Wed, 05 Feb 2020 20:23:00 +0000

Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.

Introduction

Cassandra distributes data around the cluster via the partition key.

CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));

In the above table the partition key is postal_code and the clustering column isid. The partition key will locate the data on the cluster for us. The clustering column allows us multiple rows per partition key so that we can filter how much data we read per partition. The ‘optimal’ query is one that retrieves data from only one node and not so much data that GC pressure or latency issues result. The following query is breaking that rule and retrieving 2 partitions at once via the IN parameter.

SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code IN ('77002', '77043');

This can be slower than doing two separate queries asynchronously, especially if those partitions are on two different nodes (imagine if there are 1000+ partitions in the IN statement). In summary, the simple rule to stick to is “1 partition per query”.

Partition sizes

A common mistake when data modeling is to jam as much data as possible into a single partition.

This doesn’t distribute the data well and therefore misses the point of a distributed database.
There are practical limits on the performance of partition sizes

Table per query pattern

A common approach to optimize around partition lookup is to create a table per query, and write to all of them on update. The following example has two related tables both to solve two different queries

--query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';
--query by id
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;

You can update both tables at once with a logged batch:

BEGIN BATCH
INSERT INTO my_key.my_table (id, name, address, city, state, postal_code, country, balance) VALUES (7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 'Bordeaux', 'Gironde', '33000', 'France', 56.20);
INSERT INTO my_key.my_table_by_postal_code (postal_code, id, balance) VALUES ('33000', 7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 56.20) ;
APPLY BATCH;

Source of truth

A common design pattern is to have one table act as the authoritative one over data, and if for some reason there is a mismatch or conflict in other tables as long as there is one considered “the source of truth” it makes it easy to fix any conflicts later. This is typically the table that would match what we see in typical relational databases and has all the data needed to generate all related views or indexes for different query methods. Taking the prior example, my_table is the source of truth:

--source of truth table
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;

--based on my_key.my_table and so we can query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';

Next we discuss strategies for keeping tables of related in sync.

Materialized views

Materialized views are a feature that ships with Cassandra but is currently considered rather experimental. If you want to use them anyway:

CREATE MATERIALIZED VIEW my_key.my_table_by_postal_code 
AS SELECT postal_code text, id uuid, balance float
FROM my_key.my_table 
WHERE postal_code IS NOT NULL AND id IS NOT NULL 
PRIMARY KEY(postal_code, id));

Materialized views at least run faster than the comparable BATCH insert pattern, but they have a number of bugs and known issues that are still pending fixes.

Secondary indexes

This are the original server side approach to handling different query patterns but it has a large number of downsides:

rows are read serially one node at time until limit is reached.
a suboptimal storage layout leading to very large partitions if the data distribution of the secondary index is not ideal.

For just those two reasons I think it’s rare that one can use secondary indexes and expect reasonable performance. However, you can make one by hand and just query that data asynchronously to avoid some of the downsides.

CREATE TABLE my_key.my_table_by_postal_code_2i (postal_code text, id uuid, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code_2i WHERE postal_code = '77002';
--retrieve all rows then asynchronously query the resulting ids
SELECT * FROM my_key.my_table WHERE id = ad004ff2-e5cb-4245-94b8-d6acbc22920a;
SELECT * FROM my_key.my_table WHERE id = d30e9c65-17a1-44da-bae0-b7bb742eefd6;
SELECT * FROM my_key.my_table WHERE id = e016ae43-3d4e-4093-b745-8583627eb1fe;

Exercises

Contact List

This is a good basic first use case as one needs to use multiple tables for the same data, but there should not be too many.

requirements

contacts should have first name, last name, address, state/region, country, postal code
lookup by contacts id
retrieve all contacts by a given last name
retrieve counts by zip code

Music Service

Takes the basics from the previous exercise and requires a more involved understanding of the concepts. It will require many tables and some difficult trade-offs on partition sizing. There is no one correct way to do this.

requirements

songs should have album, artist, name, and total likes
The contact list exercise, can be used as a basis for the “users”, users will have no login because we’re trusting people
retrieve all songs by artist
retrieve all songs in an album
retrieve individual song and how many times it’s been liked
retrieve all liked songs for a given user
“like” a song
keep a count of how many times a song has been listened to by all users

IoT Analytics

This will require some extensive time series modeling and takes some of the lessons from the Music Service further. The table(s) used will be informed by the query.

requirements

use the music service data model as a basis, we will be tracking each “registered device” that uses the music service
a given user will have 1-5 devices
log all songs listened to by a given device
retrieve songs listened for a device by day
retrieve songs listened for a device by month
retrieve total listen time for a device by day
retrieve total listen time for a device by month
retrieve artists listened for a device by day
retrieve artists listened for a device by month

Getting started with Cassandra: Load testing Cassandra in brief

Tue, 04 Feb 2020 20:23:00 +0000

An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra.

If you have no data model in mind

Use cassandra stress since it’s around:

first initialize the keyspace with RF3 cassandra-stress "write cl=ONE no-warmup -col size=FIXED(15000) -schema replication(strategy=SimpleStrategy,factor=3)"
second run stress cassandra-stress "mixed n=1000k cl=ONE -col size=FIXED(15000)
repeat as often as you’d like with as many clients as you want.

If you have a specific data model in mind

You can use cassandra-stress, but I suspect you’re going to find your data model isn’t supported (collections for example) or that you don’t have the required PHD to make it work the way you want. There are probably 2 dozen options from here you can use to build your load test, some of the more popular ones are gatling, jmeter, and tlp-stress. My personal favorite for this though, write a small simple python or java program that replicates your use case accurately in your own code, using a faker library to generate your data. This takes more time but you tend to have less surprises in production as it will accurately model your code.

Small python script with python driver

use python3 and virtualenv
python -m venv venv
source venv/bin/activate
read and follow install docs
if you want to skip the docs you can get away with pip install cassandra-driver
install a faker library pip install Faker

import argparse
import uuid
import time
import random
from cassandra.cluster import Cluster
from cassandra.query import BatchStatement
from faker import Faker

parser = argparse.ArgumentParser(description='simple load generator for cassandra')
parser.add_argument('--hosts', default='127.0.0.1',
                    type=str,
                    help='comma separated list of hosts to use for contact points')
parser.add_argument('--port', default=9042, type=int, help='port to connect to')
parser.add_argument('--trans', default=1000000, type=int, help='number of transactions') 
parser.add_argument('--inflight', default=25, type=int, help='number of operations in flight') 
parser.add_argument('--errors', default=-1, type=int, help='number of errors before stopping. default is unlimited') 
args = parser.parse_args()
fake = Faker(['en-US'])
hosts = args.hosts.split(",")
cluster = Cluster(hosts, port=args.port)

try:
    session = cluster.connect()
    print("setup schema");
    session.execute("CREATE KEYSPACE IF NOT EXISTS my_key WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1}")
    session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table (id uuid, name text, address text, state text, zip text, balance int, PRIMARY KEY(id))")
    session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table_by_zip (zip text, id uuid, balance bigint, PRIMARY KEY(zip, id))")
    print("allow schema to replicate throughout the cluster for 30 seconds")
    time.sleep(30)
    print("prepare queries")
    insert = session.prepare("INSERT INTO my_key.my_table (id, name, address, state, zip, balance) VALUES (?, ?, ?, ?, ?, ?)")
    insert_rollup = session.prepare("INSERT INTO my_key.my_table_by_zip (zip, id, balance) VALUES (?, ?, ?)")
    row_lookup = session.prepare("SELECT * FROM my_key.my_table WHERE id = ?")
    rollup = session.prepare("SELECT sum(balance) FROM my_key.my_table_by_zip WHERE zip = ?")
    threads = []
    ids = []
    error_counter = 0
    query = None
    params = []
    ids = []
    
    def get_id():
        items = len(ids)
        if items == 0:
            ## nothing present so return something random
            return uuid.uuid4()
        if items == 1:
            return ids[0]
        return ids[random.randint(0, items -1)]
    print("starting transactions")
    for i in range(args.trans):
        chance = random.randint(1, 100)
        if chance > 0 and chance < 50:
            new_id = uuid.uuid4()
            ids.append(new_id)
            state = fake.state_abbr()
            zip_code = fake.zipcode_in_state(state)
            balance = random.randint(1, 50000)
            query = BatchStatement()
            name = fake.name()
            address = fake.address()
            bound_insert = insert.bind([new_id, fake.name(), fake.address(), state, zip_code, balance])
            query.add(bound_insert)
            bound_insert_rollup = insert_rollup.bind([zip_code, new_id, balance])
            query.add(bound_insert_rollup)
        elif chance > 50 and chance < 75:
            query = row_lookup.bind([get_id()])
        elif chance > 75:
            zip_code = fake.zipcode()
            query = rollup.bind([zip_code])
        threads.append(session.execute_async(query))
        if i % args.inflight == 0:
            for t in threads:
                try:
                    t.result() #we don't care about result so toss it
                except Exception as e:
                    print("unexpected exception %s" % e)
                    if args.errors > 0:
                        error_counter = error_counter + 1
                        if error_counter > args.errors:
                            print("too many errors stopping. Consider raising --errors flag if this happens more quickly than you'd like")
                            break
            threads = []
            print("submitted %i of %i transactions" % (i, args.trans))
finally:
    cluster.shutdown()

Small java program with latest java driver

download java 8
create a command line application in your project technology of choice (I used maven in this example for no particularly good reason)
download a faker lib like this one and the Cassandra java driver from DataStax again using your preferred technology to do so.
run the following code sample somewhere (set your RF and your desired queries and data model)
use different numbers of clients at your cluster until you get enough “saturation” or the server stops responding.

See complete example

package pro.foundev;

import java.lang.RuntimeException;
import java.lang.Thread;
import java.util.Locale;
import java.util.ArrayList;
import java.util.List;
import java.util.function.*;
import java.util.Random;
import java.util.UUID;
import java.util.concurrent.CompletionStage;
import java.net.InetSocketAddress;
import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.CqlSessionBuilder;
import com.datastax.oss.driver.api.core.cql.*;
import com.github.javafaker.Faker;

public class App 
{
    public static void main( String[] args )
    <



Getting started with Cassandra: Setting up a Multi-DC environment
Mon, 03 Feb 2020 20:23:00 +0000
This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.

A new cluster


  In cassandra.yaml set endpoint_snitch: GossipingPropertyFileSnitch, some prefer PropertyFileSnitch for the ease of pushing out one file. GossipingPropertyFileSnitch is harder to get wrong in my experience.
  set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
  finish adding all of your nodes.
  if using authentication,  set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
  nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
  create your new keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above).
  nodetool repair -pr whatever_new_keyspace on each node in the cluster on the new keyspace.


An existing cluster

This is harder and involves more work and more options, but I’m going to discuss the way that gets you into the least amount of trouble operationally.


  make sure none of the drivers you use to connect to cassnadra are using DowngradingConsistencyRetryPolicy, or using the maligned withUsedHostsPerRemoteDc, especially allowRemoteDCsForLocalConsistencyLevel, as this may cause your driver to send requests to the remote data center before it’s populated with data.
  switch endpoint_snitch on each node to GossipingPropertyFileSnitch
  set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
  bootstrap each node in the new data center.
  if using authentication,  set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
  nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
  alter your app keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above),
  nodetool repair -pr whatever_keyspace on each node in the cluster on the new keyspace.


enjoy new data center

how to get data to new dc

Repair approach

Best done with if your repair jobs can’t be missed or stopped, either because you have a process like opscenter or repear running repairs. It also has the advantage of being very easy, and if you’ve already automated repair you’re basically done.


  let repair jobs continue..that’s it!


Rebuild approach

Faster less resource intensive, and if you have enough time to complete it while repair is stopped. Rebuild is easier to ‘resume’ than repair in many ways, so this has a number of advantages.


  run nodetool rebuild on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.
  run nodetool cleanup


YOLO rebuild with repair

This will probably overstream it’s share of data and honestly a lot of folks do this for some reason in practice:


  leave repair jobs running
  run nodetool rebuild on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.
  run nodetool cleanup on each node


Cloud strategies

There are a few valid approaches to this and none of them are wrong IMO.

region == DC, rack == AZ

Will need to get into racks and a lot of people get this wrong and imbalance the racks, but you get the advantage of more intelligent failure modes, with racks mapping to AZs.

AZ..regardless of region == DC

This allows things to be balanced easily, but you have no good option for racks then. However, some people think racks are overated, and I’d say a majority of clusters run with one rack.


MediatR 8.0 Released
Tue, 31 Dec 2019 14:54:19 +0000
This release brings some (minor) breaking changes to the public API. First, we added a non-generic overload to Send on IMediator:
public interface IMediator
{
    Task<TResponse> Send<TResponse>(IRequest<TResponse> request, CancellationToken cancellationToken = default);

+   Task<object> Send(object request, CancellationToken cancellationToken = default);

    Task Publish(object notification, CancellationToken cancellationToken = default);

    Task Publish<TNotification>(TNotification notification, CancellationToken cancellationToken = default)
        where TNotification : INotification;
}

Second, we've modified the Mediator class to include the CancellationToken for the virtual PublishCore method that allows implementors to override the publishing strategy and accept the notification itself:
-        protected virtual async Task PublishCore(IEnumerable<Func<Task>> allHandlers)
+        protected virtual async Task PublishCore(IEnumerable<Func<INotification, CancellationToken, Task>> allHandlers, INotification notification, CancellationToken cancellationToken)
         {
             foreach (var handler in allHandlers)
             {
-                await handler().ConfigureAwait(false);
+                await handler(notification, cancellationToken).ConfigureAwait(false);
             }
         }

Finally, a new feature! We've added some new built-in pipeline behaviors for handling exceptions. You can now handle specific exceptions (similar to an Exception Filter in ASP.NET Core):
public interface IRequestExceptionHandler<in TRequest, TResponse, TException>
    where TException : Exception
{
    Task Handle(TRequest request, 
        TException exception, 
        RequestExceptionHandlerState<TResponse> state, 
        CancellationToken cancellationToken);
}

Alternatively, if you don't want to provide alternate responses based on exceptions, you can provide exception actions:
public interface IRequestExceptionAction<in TRequest, in TException>
    where TException : Exception
{
    Task Execute(TRequest request, TException exception, CancellationToken cancellationToken);
}

These are a bit simpler if you want to just provide some exception logging. I've also updated the MS DI Extensions package, which registers these new behaviors and interfaces. Enjoy!

  



User Secrets in Docker-based .NET Core Worker Applications
Mon, 16 Dec 2019 16:53:16 +0000
As part of the recent Message Endpoints in Azure series, I wanted to check out the new .NET Core 3.0 Worker templates to see how the templates have improved the situation (actually, a lot), but there are still some things missing from the Worker SDK versus the Web SDK.
.NET Core Workers, introduced initially as background services in .NET Core 2.x, are now a top-level dotnet and Visual Studio template:

When you create the worker template, you're also given the option to enable Docker support:

In that series, I connect to my instance of Azure Service Bus in the cloud. However, I don't want to commit the connection string to source control, so I use User Secrets to store the connection string locally. Out-of-the-box, this SDK will include a user secrets ID in the project file, however, if you actually try to use the secrets when running, it won't work!
The first reason is because there was an issue in the Worker MSBuild task pipeline that looked for either the Web SDK or an explicit reference to the User Secrets NuGet package to decide to include the User Secrets in the configuration sources - but that's been fixed (workaround here).
However, it still doesn't work if you run a worker in a Docker container inside Visual Studio. The reason here is because when you run in a Docker container, the Visual Studio tooling does a lot of work to make running the container easier - including mapping a bunch of volumes. On that list is User Secrets - however - user secrets are only mapped for Web SDK!
Running our worker app inside a Docker container during development means we don't have any user secrets - and the solutions aren't great. Environment variables, Docker secrets, all are more annoying than just using the original secrets.
Mapping the volume
Instead of mucking around with custom solutions, ideally we can just map the user secrets volume ourselves. Luckily, there's an easy way to do so - we can specify custom Docker command line arguments:
<PropertyGroup>
  <TargetFramework>netcoreapp3.1</TargetFramework>
  <UserSecretsId>89595f67-a846-41e5-a74e-f876488ea8be</UserSecretsId>
  <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
  <DockerfileRunArguments>-v "$(AppData)/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro"</DockerfileRunArguments>
</PropertyGroup>

The DockerfileRunArguments element we use to pass through our additional volume mapping, which you can verify with a normal Web SDK running under Docker to see what it passes through.
I've opened a GitHub issue with the SDK tools, but in the meantime, we can use this workaround. With this in place, our Worker application can now use User Secrets whether it's running on our host or in a container.

  



Gocode Vim Plugin and Go Modules
Sat, 05 Jan 2019 17:09:26 +0000
I recently purchased Let’s Go from Alex Edwards.  I wanted an end-to-end Golang website tutorial.  It has been great.  Lots learned.

Unfortunately, he is using Go’s modules and the version of the gocode Vim plugin I was using did not support Go modules.

Solution:
Use this fork of the gocode Vim plugin and you’ll get support for Go modules.

I use Vim Plug for my Vim plugins.  Huge fan of Vundle but I like the post-actions feature of Plug.  I just had to change one line of my vimrc and re-run updates.

diff --git a/vimrc b/vimrc
index 3e8edf1..8395705 100644
--- a/vimrc
+++ b/vimrc
@@ -73,7 +73,7 @@ endif
           let editor_name='nvim'
           Plug 'zchee/deoplete-go', { 'do': 'make'}
         endif
-        Plug 'nsf/gocode', { 'rtp': 'vim', 'do': '~/.config/nvim/plugged/gocode/vim/symlink.sh' }
+        Plug 'stamblerre/gocode', { 'rtp': 'vim', 'do': '~/.vim/plugged/gocode/vim/symlink.sh' }
         Plug 'godoctor/godoctor.vim', {'for': 'go'} " Gocode refactoring tool
     " }


That is the line I had to change then run :PlugUpdate! and the new plugin was installed.

I figured all of this out due to this comment by Tommaso Sardelli on Github.  Thank you Tommaso.


Raspberry Pi Kubernetes Cluster - Part 4
Fri, 28 Dec 2018 16:35:23 +0000

    Raspberry Pi Kubenetes Cluster - Part 1

Raspberry Pi Kubenetes Cluster - Part 2

Raspberry Pi Kubenetes Cluster - Part 3

Raspberry Pi Kubenetes Cluster - Part 4

Howdy again.

In this post I’m going to show you how to create a docker image to run on ARM architecture and also how to deploy it and view it.

To start please view my basic flask application called fl8 here

If you’d like to clone and use it:

git clone git@github.com:meridth/fl8.git && cd fl8


ARM docker image
First we need to learn about QEMU

What is QEMU and QEMU installation

QEMU (Quick EMUlator) is an Open-Source hosted hypervisor, i.e. an hypervisor running on a OS just as other computer programs, which performs hardware virtualization. QEMU emulates CPUs of several architectures, e.g. x86, PPC, ARM and SPARC. It allows the execution of non-native target executables emulating the native execution and, as we require in this case, the cross-building process.

Base Docker image that includes QEMU

Please open the Dockerfile.arm and notice the first line: FROM hypriot/rpi-alpine.  This is a base image that includes the target qemu statically linked executable, qemu-arm-static in this case.  I chose hypriot/rpi-alpine because the alpine base images are much smaller than other base images.

Register QEMU in the build agent

To add QEMU in the build agent there is a specific Docker Image performing what we need, so just run in your command line:

docker run --rm --privileged multiarch/qemu-user-static:register --reset


Build image

docker build -f ./Dockerfile.arm -t meridth/rpi-fl8 .


And voila!  You now have an image that will run on Raspberry Pis.

Deployment and Service

/.run-rpi.sh is my script where I run a Kubernetes deployment with 3 replicas and a Kubernetes service.  Please read fl8-rpi-deployment.yml and fl8-rpi-service.yml.  They are only different from the other deployment and service files by labels.  Labels are key/vaule pairs that can be used by selectors later.

The deployment will pull my image from meridth/rpi-fl8 on dockerhub.  If you have uploaded your docker image somewhere you can change the deployment file to pull that image instead.

Viewing application

kubectl get pods


Choose a pod to create the port forwarding ssh tunnel.

kubectl port-forward [pod-name] [app-port]:[app-port]


Example:  kubectl port-forward rpi-fl8-5d84dd8ff6-d9tgz 5010:5010

The final result when you go to http://localhost:5010 in a browser.



Hope this helps someone else.  Cheers.

    Raspberry Pi Kubernetes Cluster - Part 4 was originally published by Jason Meridth at Jason Meridth on December 28, 2018.


Raspberry Pi Kubernetes Cluster - Part 3
Mon, 24 Dec 2018 21:59:23 +0000

    Raspberry Pi Kubenetes Cluster - Part 1

Raspberry Pi Kubenetes Cluster - Part 2

Raspberry Pi Kubenetes Cluster - Part 3

Raspberry Pi Kubenetes Cluster - Part 4

Well, it took me long enough to follow up on my previous posts.  There are reasons.


  The day job has been fun and busy
  Family life has been fun and busy
  I kept hitting annoying errors when trying to get my cluster up and running


The first two reasons are the usual reasons a person doesn’t blog. :)

The last one is what prevented me from blogging sooner.  I had mutliple issues when trying to use rak8s to setup my cluster.  I’m a big fan of Ansible and I do not like running scripts over and over.  I did read K8S on Raspbian Lite from top to bottom and realized automation would make this much better.



The issues I experienced:

apt-get update would not work

I started with the vanilla Raspbian lite image to run on my nodes and had MANY MANY issues with running apt-get update and apt-get upgrade.  The mirrors would disconnect often and just stall.  This doesn’t help my attempted usage of rak8s which does both on the cluster.yml run (which I’ll talk about later).

rak8s changes needed to run HypriotOS and kubernetes 1.13.1

Clone the repo locally and I’ll walk you through what I changed to get rak8s working for me and HypriotOS.

Change the following files:


  ansible.cfg
    
      change user from pi to pirate
    
  
  roles/kubeadm/tasks/main.yml
    
      add ignore_errors: True to Disable Swap task
      I have an open PR for this here
    
  
  group_vars/all.yml
    
      Change kubernetes_package_version to "1.13.1-00"
      Change kubernetes_version to "v1.13.1"
    
  


After you make those changes you can run ansible-playbook cluster.yml as the rak8s documentation suggests.  Please note this is after you edit inventory and copy ssh keys to raspberry pis.

Flannel networking issue once nodes are up

After I get all of the nodes up I noticed the master node was marked ast NotReady and when I ran kubectl describe node raks8000 I saw the following error:


  KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized


This error is known in kubernetes > 1.12 and flannel v0.10.0.  It is mentioned in this issue.  The fix is specifically mentioned here.  It is to run the following command

kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

After readin the issue it seems the fix will be in the next version of flannel and will be backported to v0.10.0.

A running cluster


    Raspberry Pi Kubernetes Cluster - Part 3 was originally published by Jason Meridth at Jason Meridth on December 24, 2018.


MVP how minimal
Thu, 20 Dec 2018 20:00:00 +0000
MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas.
In recent history I’ve seen up close two extreme constrasting examples of MVP:


  Mega Minimal: website and db, mostly manual on the backend
  Mega Mega: provisioning system, dynamic tuning of systems via ML, automated operations, monitoring a few others I’m leaving out.


Feedback

If we’re evaluating which approach gives us more feedback, Mega Minimal MVP is gonna win hands down here. Some will
counter they don’t want to give people a bad impression with a limited product and that’s fair, but it’s better than no impression (the dreaded never shipped MVP). The Mega Mega
MVP I referenced took months to demo. only had one of those checkboxes setup and wasn’t ever demod again. So we can categorical say that failed 
at getting any feedback.

Whereas the Mega Minimal MVP, got enough feedback and users for the founders to realize that wasn’t a business for them.
Better than after hiring a huge team and sinking a million plus into dev efforts for sure. Not the happy ending I’m sure you all were expecting, but I view that as mission accomplished.

Core Value


  Mega Minimal, they only focused on a single feature, executed well enough that people gave them some positive feedback, but not enough to justify automating everything.
  Mega Mega. I’m not sure anyone who talked about the product saw the same core value, and there were several rewrites and shifts along the way.


Advantage Mega Minimal again

What about entrants into a crowded field

Well that is harder and the MVP tends to be less minimal, because the baseline expectations are just much higher. I still lean towards Mega Minimal having a better chance at getting users, since there is a non zero chance the Mega Mega MVP will never get finished.
I still think the exercise in focusing on core value that makes your product not a me too, and even considering how you can find a niche in a crowded field
instead of just being “better”, and your MVP can be that niche differentiator.

Internal users

Sometimes a good middle ground is considering getting lots of internal users if you’re really worried about bad experiences. This has it’s
it’s definite downsides however, and you may not get diverse enough opinions. But it does give you some feedback while saving some face or bad experiences. I often think of the example of EC2 that was heavily used by Amazon, before being released to the world.
That was a luxury Amazon had, where their customer base and their user base happened to be very similar, and they had bigger scale needs than any of their early customers, so the early internal feedback loop was a very strong signal.

Summary

In the end however you want to approach MVPs is up to you, and if you find success with a meatier MVP than I have please don’t let me push you away from what works. But if you are having trouble shipping and are getting pushed all the time to add one more feature to that MVP before releasing it, consider stepping back and asking is this really core value 
for the product? Do you already have your core value? if so, consider just releasing it.


Surprise Go is ok for me now
Thu, 13 Dec 2018 20:23:00 +0000
I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve.

About 7 months ago I wrote about all the things I didn’t really care for in Go and now I either no longer am so bothered by it or things have improved.

Go Modules so far is a huge improvement over Dep and Glide for dependency management. It’s easy to setup, performant and eliminates the GOPATH silliness. I haven’t tried it yet with some of the goofier libraries that gave me problems in the past (k8s api for example) so the jury is out on that, but again pretty impressed. I now longer have to check in vendor to speed up builds. Lesson use Go Modules.

I pretty much stopped using channels for everything but shutdown signals and that fits my preferences pretty well, I use mutex and semaphores for my multithreaded code and feel no guilt about it. This cut out a lot of pain for me, and with the excellent race detector I feel really comfortable writing multi-threaded in Go now. Lesson, don’t use channels much.

Lack of generics still sometimes sucks but I usually implement some crappy casting with dynamic types if I need that. I’ve sorta made my piece with just writing more code, and am no longer so hung up. Lesson relax.

Error handling I’m still struggling with, I thought about using one of the error Wrap() libraries but an official one is in draft spec now, so I’ll wait on that. I now tend to have less nesting of functions as a result, this probably means longer functions than I like, but my code looks more “normal” now. This is a trade off I’m ok with. Lesson relax more.

I see the main virtue of Go now that it is very popular in the infrastructure space where I am and so it’s becoming the common tongue (largely replacing Python for those sorts of tasks). For this, honestly it’s about right. It’s easy to rip out command line tools and deploy binaries for every platform with no runtime install.

The community’s conservative attitude I sort of view as a feature now, in that there isn’t a bunch of different options that are popular and there is no arguing over what file format is used. This drove me up the wall initially, but I appreciate how much less time I spend on these things now.

So now I suspect Go will be my “last” programming language. It’s not the one I would have chosen, but where I am at in my career, where most of my dev work is automation and tooling it fits the bill pretty well.

Also equally important most of the people working with me didn’t have full time careers as developers or spend their time reading “Domain Driven Design” (amazing book) so adding in a bunched of nuanced stuff that maybe technically optimal but assumes the reader grasps all that nuance isn’t a good tradeoff for me.

So I think I sorta get it now. I’ll never be a cheerleader for the language but it definitely solves my problems well enough.


Collaboration vs. Critique
Fri, 18 May 2018 17:00:00 +0000
While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams.  The people aspect of software development, more often than not, tends to be the most difficult part of software engineering.  Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs.  The nature of software development tends to follow an empirical process model rather than a defined process model.  That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset.  Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation.  This means that team members are often required to work together to determine how to proceed.  Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation.  One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique.

To help illustrate what I’ve seen happen countless times both in catch-up design sessions and code reviews, consider the following two scenarios:

Scenario 1
Tom and Sally are both developers on a team maintaining a large-scale application.  Tom takes the next task in the development queue which happens to have some complex processes that will need to be addressed.  Being the good development team that they are, both Tom and Sally are aware of the requirements of the application (i.e. how the app needs to work from the user’s perspective), but they have deferred design-level discussions until the time of implementation.  After Tom gets into the process a little, seeing that the problem is non-trivial, he pings Sally to help him brainstorm different approaches to solving the problem.  Tom and Sally have been working together for over a year and have become accustomed to these sort of ad-hoc design sessions.  As they begin discussing the problem, they each start tossing ideas out on the proverbial table resulting in multiple approaches to compare and contrast.  The nature of the discussion is such that neither Tom nor Sally are embarrassed or offended when the other points out flaws in the design because there’s a sense of safety in their mutual understanding that this is a brainstorming session and that neither have thought in depth about the solutions being set froth yet.  Tom throws out a couple of ideas, but ends up shooting them down himself as he uses Sally as a sounding board for the ideas.  Sally does the same, but toward the end of the conversation suggests a slight alteration to one of Tom’s initial suggestions that they think may make it work after all.  They end the session both with a sense that they’ve worked together to arrive at the best solution.

Scenario 2
Bill and Jake are developers on another team.  They tend to work in a more siloed fashion, but they do rely upon one another for help from time to time and they are required to do code reviews prior to their code being merged into the main branch of development.  Bill takes the next task in the development queue and spends the better part of an afternoon working out a solution with a basic working skeleton of the direction he’s going.  The next day he decides that it might be good to have Jake take a look at the design to make him aware of the direction.  Seeing where Bill’s design misses a few opportunities to make the implementation more adaptable to changes in the future, Jake points out where he would have done things differently.  Bill acknowledges that Jake’s suggestions would be better and would have probably been just as easy to implement from the beginning, but inwardly he’s a bit disappointed that Jake didn’t like his design as-is and that he has to do some rework.  In the end, Bill is left with a feeling of critique rather than collaboration.

Whether it’s a high-level UML diagram or working code, how one person tends to perceive feedback on the ideas comprising a potential solution has everything to do with timing. It can be the exact same feedback they would have received either way, but when the feedback occurs often makes a difference between whether it’s perceived as collaboration or critique.  It’s all about when the conversation happens.


Testing Button Click in React with Jest
Mon, 07 May 2018 17:01:59 +0000
When building React applications you will most likely find yourself using Jest as your testing framework.  Jest has some really, really cool features built in.  But when you use Enzyme you can take your testing to the nest level.
One really cool feature is the ability to test click events via Enzyme to ensure your code responds as expected.
Before we get started you are going to want to make sure you have Jest and Enzyme installed in your application.

Installing Enzyme
Installing Jest

Sample code under test

What I would like to be able to do is pull the button out of my component and test the onClick event handler.

  
// Make sure you have your imports setup correctly
import React from 'react';
import { shallow } from 'enzyme';


it('When active link clicked, will push correct filter message', () => {
    let passedFilterType = '';
    const handleOnTotalsFilter = (filterType) => {
        passedFilterType = filterType;
    };
    const accounts = {};
    const wrapper = shallow(<MyComponent accounts={accounts} filterHeader="" onTotalsFilter={handleOnTotalsFilter} />);

    const button = wrapper.find('#archived-button');
    button.simulate('click');
    expect(passedFilterType).toBe(TotalsFilterType.archived);
});


Lets take a look at the test above

First we are going to create a callback (click handler) to catch the bubbled up values.
We use Enzyme to create our component MyComponent
We use the .find() on our wrapped component to find our <Button /> by id
After we get our button we can call .simulate(‘click’) which will act as a user clicking the button.
We can assert that the expected value bubbles up.

As you can see, simulating a click event of a rendered component is very straight forward, yet very powerful.
Till next time,


Lessons from a year of Golang
Mon, 07 May 2018 13:16:00 +0000
I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive.

This isn’t meant to evaluate if you should pick up Go or tell you what you should think of it, this is strictly meant to help people out that are new to the language but experienced in Java, Python, Ruby, C#, etc and have read some basic Go getting started guide.

Dependency management

This is probably the feature most frequently talked about by newcomers to Go and with some justification, as dependency management been a rapidly shifting area that’s nothing like what experienced Java, C#, Ruby or Python developers are used to.

I’ll cut to the chase the default tool now is Dep all other tools I’ve used such as Glide or Godep they’re now deprecated in favor of Dep, and while Dep has advanced rapidly there are some problems you’ll eventually run into (or I did):


  Dep hangs randomly and is slow, which is supposedly network traffic but it happens to everyone I know with tons of bandwidth. Regardless, I’d like an option to supply a timeout and report an error.
  Versions and transitive depency conflicts can be a real breaking issue in Go still. So without shading or it’s equivalent two package depending on different versions of a given package can break your build, there are a number or proposals to fix this but we’re not there yet.
  Dep has some goofy ways it resolves transitive dependencies and you may have to add explicit references to them in your Gopkg.toml file. You can see an example here under Updating dependencies – golang/dep.


My advice


  Avoid hangs by checking in your dependencies directly into your source repository and just using the dependency tool (dep, godep, glide it doesn’t matter) for downloading dependencies.
  Minimize transitive dependencies by keeping stuff small and using patterns like microservices when your dependency tree conflicts.


GOPATH

Something that takes some adjustment is you check out all your source code in one directory with one path (by default ~/go/src ) and include the path to the source tree to where you check out. Example:


  I want to use a package I found on github called jim/awesomeness
  I have to go to ~/go/src and mkdir -p github.com/jim
  cd into that and clone the package.
  When I reference the package in my source file it’ll be literally importing github.com/jim/awesomeness


A better guide to GOPATH and packages is here.

My advice

Don’t fight it, it’s actually not so bad once you embrace it.

Code structure

This is a hot topic and there are a few standards for the right way to structure you code from projects that do “file per class” to giant files with general concept names (think like types.go and net.go). Also if you’re used to using a lot of sub package you’re gonna to have issues with not being able to compile if for example you have two sub packages reference one another.

My Advice

In the end I was reasonably ok with something like the following:


  myproject/bin for generated executables
  myproject/cmd for command line code
  myproject/pkg for code related to the package


Now whatever you do is fine, this was just a common idiom I saw, but it wasn’t remotely all projects. I also had some luck with just jamming everything into the top level of the package and keeping packages small (and making new packages for common code that is used in several places in the code base). If I ever return to using Go for any reason I will probably just jam everything into the top level directory.

Debugging

No debugger! There are some projects attempting to add one but Rob Pike finds them a crutch.

My Advice

Lots of unit tests and print statements.

No generics

Sorta self explanatory and it causes you a lot of pain when you’re used to reaching for these.

My advice

Look at the code generation support which uses pragmas, this is not exactly the same as having generics but if you have some code that has a lot of boiler plate without them this is a valid alternative.  See this official Go Blog post for more details.

If you don’t want to use generation you really only have reflection left as a valid tool, which comes with all of it’s lack of speed and type safety.

Cross compiling

If you have certain features or dependencies you may find you cannot take advantage of one of Go’s better features cross compilation.

I ran into this when using the confluent-go/kafka library which depends on the C librdkafka library. It basically meant I had to do all my development in a Linux VM because almost all our packages relied on this.

My Advice

Avoid C dependencies at all costs.

Error handling

Go error handling is not exception base but return based, and it’s got a lot of common idioms around it:

myValue, err := doThing()
if err != nil {
	return -1, fmt.Errorf(“unable to doThing %v”, err)
  	}


Needless to say this can get very wordy when dealing with deeply nested exceptions or when you’re interacting a lot with external systems.  It is definitely a mind shift if you’re used to the throwing exceptions wherever and have one single place to catch all exceptions where they’re handled appropriately.

My Advice

I’ll be honest I never totally made my peace with this. I had good training from experienced opensource contributors to major Go projects, read all the right blog posts, definitely felt like I’d heard enough from the community on why the current state of Go error handling was great in their opinions, but the lack of stack traces was a deal breaker for me.

On the positive side, I found Dave Cheney’s advice on error handling to be the most practical and he wrote a package containing a lot of that advice, we found it invaluable as it provided those stack traces we all missed but you had to remember to use it.

Summary

A lot of people really love Go and are very productive with it, I just was never one of those people and that’s ok. However, I think the advice in this post is reasonably sound and uncontroversial. So, if you find yourself needing to write some code it Go, give this guide a quick perusal and you’ll waste a lot less time than I did getting productive in developing applications in Go.


Raspberry Pi Kubernetes Cluster - Part 2
Thu, 03 May 2018 02:13:07 +0000

    Howdy again.

Alright, my 8 port switch showed up so I was able to connect my raspberry 3B+ boards to my home network.  I plugged it in with 6 1ft CAT5 cables I had in my catch-all box that all of us nerds have.  I’d highly suggest flexible CAT 6 cables instead if you can get them, like here.   I ordered them and they showed up before I finished this post, so I am using the CAT6 cables.



The IP addresses they will receive initialy from my home router via DHCP can be determined with nmap.  Lets imagine my subnet is 192.168.1.0/24.

Once I got them on the network I did the following:



Install Raspberrian OS On SD Cards

You can get the Raspberry Pi Stretch Lite OS from here.



Then use the Etcher tool to install it to each of the 6 SD cards.



IMPORTANT

Before putting the cards into the Raspberry Pis you need to add a ssh folder to the root of the SD cards.  This will allow you to ssh to each Raspberry Pi with default credentials (username: pi and password raspberry).  Example: ssh pi@raspberry_pi_ip where raspberry_ip is obtained from the nmap command above.

Next post will be setting up kubernetes.  Thank you for reading.

Cheers.

    Raspberry Pi Kubernetes Cluster - Part 2 was originally published by Jason Meridth at Jason Meridth on May 02, 2018.


Multi-Environment Deployments with React
Tue, 10 Apr 2018 12:54:17 +0000
If you are using Create-React-App to scaffold your react application there is built in support for changing environment variables based on the NODE_ENV values, this is done by using .env files.  In short this process works by having a .env, .env.production, .env.development set of files.  When you run/build your application CRA will set the NODE_ENV value to either development or production and based on these values the correct .env file will be used.
This works great, when you have a simple deploy setup. But many times in enterprise level applications you need support for more than just 2 environments, many times it is 3-4 environments.  Common logic would suggest that you can accomplish this via the built in mechanism by having additional .env files and changing the NODE_ENV value to the value you care about.  However, CRA does not support this with doing an eject, which will eject all the default conventions and leave it to you to configure your React application.  Maybe this is a good idea, but in my case ejecting was not something I wanted to do.
Because I did not want to do an eject I needed to find another solution, and after a fair amount of searching I found a solution that seems to work for me and my needs and is about the amount of effort as I wanted Hart Hoover and it finally motivated me to purchase his “grocery list” and do this finally.  I’ve been using Minikube for local Kubernetes testing but it doesn’t give you multi-host testing abilities.  I’ve also been wanting to get deeper into my Raspberry Pi knowledge.  Lots of learning and winning.

The items I bought were:


  Six Raspberry Pi 3 Model B+ Motherboards
  Six SanDisk Ultra 32GB microSDHC UHS-I Card with Adapter, Grey/Red, Standard Packaging (SDSQUNC-032G-GN6MA)
  One Sabrent 6-Pack 22AWG Premium 3ft Micro USB Cables High Speed USB 2.0 A Male to Micro B Sync and Charge Cables Black CB-UM63
  One AmazonBasics 6-Port USB Wall Charger (60-Watt) - Black
  One GeauxRobot Raspberry Pi 3 Model B 6-layer Dog Bone Stack Clear Case Box Enclosure also for Pi 2B B+ A+ B A
  One Black Box 8-Port Switch


Here is the tweet when it all arrived:

I blame @hhoover ;). I will be building my @kubernetesio cluster once the 6pi case shows up next Wednesday. The extra pi is to upgrade my @RetroPieProject. Touch screen is an addition I want to try. Side project here I come. pic.twitter.com/EebIKbsCeH
— Jason Meridth (@jmeridth) March 31, 2018



I spent this morning finally putting it together.

Here is me getting started on the “dogbone case” to hold all of the Raspberry Pis:



The bottom and one layer above:



And the rest:









Different angles completed:





And connect the power:



Next post will be on getting the 6 sandisk cards ready and putting them in and watching the Raspberry Pis boot up and get a green light.  Stay tuned.

Cheers.

    Raspberry Pi Kubernetes Cluster - Part 1 was originally published by Jason Meridth at Jason Meridth on April 07, 2018.


Building AWS Infrastructure with Terraform: S3 Bucket Creation
Fri, 06 Apr 2018 14:28:49 +0000
If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure.  In the AWS word you can use the native CloudFormation solution, but honestly I find this painful and the docs very lacking.  Personally, I prefer Terraform by Hashicorp.  In my experience the simplicity and easy of use, not to mention the stellar documentation make this the product of choice.
This is the initial post in what I hope to be a series of post about how to use Terraform to setup/build AWS Infrastructure.
Terrform Documentation on S3 Creation -> Here
In this post I will cover 2 things

Basic bucket setup
Bucket setup as Static website

Setting up a basic bucket we can use the following

resource "aws_s3_bucket" "my-bucket" {
    bucket = "my-bucket"
    acl    = "private"

    tags {
        Any_Tag_Name = "Tag value for tracking"
    }    
}


When looking at the example above the only 2 values that are required are bucket and acl.
I have added the use of Tags to show you can add custom tags to your bucket
Another way to setup an S3 bucket is to act as a Static Web Host.   Setting this up takes a bit more configuration, but not a ton.

resource "aws_s3_bucket" "my-website-bucket" {
    bucket = "my-website-bucket"
    acl    = "public-read"

    website {
        index_document = "index.html"
        error_document = "index.html"
    }    

    policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-website-bucket/*"
        }
    ]
}
    POLICY
    tags {
          Any_Tag_Name = "Tag value for tracking"
    }
}


The example above has 2 things that need to be pointed out.

The website settings.  Make sure you setup the correct pages here for index/error

The Policy settings.  Here I am using just basic policy.  You can of course setup any policy here you want/need.
As you can see, setting up S3 buckets is very simple and straight forward.
*** Reminder: S3 bucket names MUST be globally unique ***
Till next time,


SSH - Too Many Authentication Failures
Wed, 28 Mar 2018 05:00:00 +0000

    Problem
I started seeing this error recently and had brain farted on why.

Received disconnect from 123.123.132.132: Too many authentication failures for hostname

After a bit of googling it came back to me.  This is because I’ve loaded too many keys into my ssh-agent locally (ssh-add).  Why did you do that?  Well, because it is easier than specifying the IdentityFile on the cli when trying to connect.  But there is a threshhold.  This is set by the ssh host by the MaxAuthTries setting in /etc/ssh/sshd_config.  The default is 6.

Solution 1
Clean up the keys in your ssh-agent.

ssh-add -l lists all the keys you have in your ssh-agent
ssh-add -d key deletes the key from your ssh-agent

Solution 2
You can solve this on the command line like this:

ssh -o IdentitiesOnly=yes -i ~/.ssh/example_rsa foo.example.com

What is IdentitiesOnly?  Explained in Solution 3 below.

Solution 3 (best)
Specifiy, explicitly, which key goes to which host(s) in your .ssh/config file.

You need to configure which key (“IdentityFile”) goes with which domain (or host). You also want to handle the case when the specified key doesn’t work, which would usually be because the public key isn’t in ~/.ssh/authorized_keys on the server. The default is for SSH to then try any other keys it has access to, which takes us back to too many attempts. Setting “IdentitiesOnly” to “yes” tells SSH to only try the specified key and, if that fails, fall through to password authentication (presuming the server allows it).

Your ~/.ssh/config would look like:

Host *.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myhost
Host secure.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/mysecurehost_rsa
Host *.myotherhost.domain
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myotherhost_rsa


Host is the host the key can connect to
IdentitiesOnly means only to try this specific key to connect, no others
IdentityFile is the path to the key

You can try multiple keys if needed

Host *.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myhost_rsa
  IdentityFile ~/.ssh/myhost_dsa


Hope this helps someone else.

Cheers!

    SSH - Too Many Authentication Failures was originally published by Jason Meridth at Jason Meridth on March 28, 2018.


Clear DNS Cache In Chrome
Tue, 27 Mar 2018 20:42:00 +0000

    I’m blogging this because I keep forgetting how to do it.  Yeah, yeah, I can google it.  I run this blog so I know it is always available…..anywho.

Go To:

chrome://net-internals/#dns


Click “Clear host cache” button



Hope this helps someone else.

Cheers.

    Clear DNS Cache In Chrome was originally published by Jason Meridth at Jason Meridth on March 27, 2018.


Create Docker Container from Errored Container
Mon, 26 Mar 2018 03:31:00 +0000

    When I’m trying to “dockerize” an applciation I usually have to work through some wonkiness.

To diagnose a container that has errored out, I, obviously, look at the logs via docker logs -f [container_name].  That is sometimes helpful.  It will, at minimum tell me where I need to focus on the new container I’m going to create.



Pre-requisites to being able to build a diagnosis container:


  You need to use CMD, not ENTRYPOINT in the Dockerfile
    
      with CMD you’ll be able to start a shell, with ENTRYPOINT your diagnosis container will just keep trying to run that
    
  


To create a diagnosis container, do the following:


  Check your failed container ID by docker ps -a
  Create docker image form the container with docker commit
    
      example: docker commit -m "diagnosis" [failed container id]
    
  
  Check the newly create docker image ID by docker images
  docker run -it [new container image id] sh
    
      this takes you into a container immediately after the error occurred.
    
  


Hope this helps someone else.

Cheers.

    Create Docker Container from Errored Container was originally published by Jason Meridth at Jason Meridth on March 25, 2018.


Log Early, Log Often… Saved my butt today
Wed, 21 Mar 2018 13:16:03 +0000
In a prior posting (AWS Lambda:Log Early Log often, Log EVERYTHING) I wrote about the virtues and value about having really in depth logging, especially when working with cloud services.  Well today this logging saved my ASS a ton of detective work.
Little Background

I have a background job (Lambda that is called on a schedule) to create/update data cache in a DynamoDB table.  Basically this job will pull data from one data source and attempt to push it as create/update/delete to our Dynamo table.
Today when I was running our application I noticed things were not loading right, in fact I had javascript errors because of null reference errors.  I knew that the issue had to be in our data, but was not sure what was wrong.  If I had not had a ton of logging (debug and info) I would have had to run our code locally and step though/debug code for hundreds of items of data.
However, because of in depth logging I was able to quickly go to CloudWatch and filter on a few key words and narrow hundreds/thousands of log entries down to 5.  Once I had these 5 entries I was able to expand a few of those entries and found the error within seconds.
Total time to find the error was less than 5 minutes and I never opened a code editor or stepped into code.
The moral of this story, because I log everything, including data (no PII of course) I was able to quickly find the source of the error.  Now to fix the code….
Till next time,


AWS Lambda: Log early, Log often, Log EVERYTHING
Tue, 06 Mar 2018 14:00:58 +0000
In the world of building client/server applications logs are important.  They are helpful when trying to see what is going on in your application.  I have always held the belief  that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code.
But lets be honest, in most cases when building client/server applications logs are an afterthought.  Often this is because you can pretty easily (in most cases) debug your application and step through the code.
When building a serverless applications with technologies like AWS Lambda functions (holds true for Azure Functions as well) your logging game really needs to step up.
The reason for this is that you cannot really debug your Lambda in the wild (you can to some degree locally with AWS SAM or the Serverless framework).  Because of this you need produce detailed enough logs to allow you to easily determine the WHAT and WHERE.
When I build my serverless functions I have a few guidelines I follow

Info Log calls to methods, output argument data (make sure no PII/PHI)
Error Log any failures (in try/catch or .catch for promises)
Debug Log any critical decision points
Info Log exit calls at top level methods

I also like to setup a simple and consistent format for my logs.  The example I follow for my Lambda logs is as seen below

timestamp: [logLevel] : [Class.Method] - message {data points} 


I have found that if I follow these general guidelines the pain of determine failure points in serverless environments is heavily reduced.
Till next time,


Sinon Error: Attempted to wrap undefined property ‘XYZ as function
Tue, 27 Feb 2018 13:45:29 +0000
I ran into a fun little error recently when working on a ReactJs application.  In my application I was using SinonJs to setup some spies on a method, I wanted to capture the input arguments for verification.  However, when I ran my test I received the following error.
Attempted to wrap undefined property handlOnAccountFilter as function
My method under test is setup as such

handleOnAccountFilter = (filterModel) => {
    // logic here
}


I was using the above syntax is the proposed class property feature, which will automatically bind the this context of the class to my method.
My sinon spy is setup as such

let handleOnAccountFilterSpy = null;
beforeEach(() => {
    handleOnAccountFilterSpy = sinon.spy(AccountsListingPage.prototype, 'handleOnAccountFilter');
});

afterEach(() => {
     handleOnAccountFilterSpy.restore();
});


Everything looked right, but I was still getting this error.  It turns out that this error is due in part in the way that the Class Property feature implements the handlOnAccountFilter.  When you use this feature the method/property is added to the class as an instance method/property, not as a prototype method/property.  This means that sinon is not able to gain access to it prior to creating an instance of the class.
To solve my issue I had to make a change in the implementation to the following

handleOnAccountFilter(filterModel) {
    // logic here
}


After make the above change I needed to determine how I wanted to bind this to my method (Cory show 5 ways to do this here).  I chose to bind this inside the constructor as below

constructor(props){
    super(props);

    this.handleOnAccountFilter = this.handleOnAccountFilter.bind(this);
}


I am not a huge fan of having to do this (pun intended), but oh well.  This solved my issues.
Till next time


Ensuring componentDidMount is not called in Unit Tests
Thu, 22 Feb 2018 19:45:53 +0000
If you are building a ReactJs you will often times implement componentDidMount on your components.  This is very handy at runtime, but can pose an issue for unit tests.
If you are building tests for your React app you are very likely using enzyme to create instances of your component.  The issue is that when enzyme creates the component it invokes the lifecyle methods, like componentDidMount.  Sometimes we do not want this to be called, but how to suppress this?
I have found 2 different ways to suppress/mock componentDidMount.
Method one is to redefine componentDidMount on your component for your tests.  This could have interesting side effects so use with caution.

    describe('UsefullNameHere', () => {
        
        beforeAll(() => {
            YourComponent.prototype.componentDidMount = () => {
                // can omit or add custom logic
            };
        });
    });


Basically above I am just redefining the componentDidMount method on my component.  This works and allows you to have custom logic.  Be aware that when doing above you will have changed the implementation for your component for the lifetime of your test session.
Another solution is to use a mocking framework like SinonJs.  With Sinon you can stub out the componentDidMount implementation as seen below

    describe('UsefullNameHere', () => {
        
        let componentDidMountStub = null;
        beforeAll(() => {
            componentDidMountStub = sinon.stub(YourComponent.prototype, 'componentDidMount').callsFake(function() {
                 // can omit or add custom logic
            });
        });

        afterAll(() => {
            componentDidMountStub.restore();
        });
    });


Above I am using .stub to redefine the method.  I also added .callsFake() but this can be omitted if you just want to ignore the call.  You will want to make sure you restore your stub via the afterAll, otherwise you will have stubbed out the call for the lifetime of your test session.
Till next time,


Los Techies Welcomes Derik Whittaker
Wed, 21 Feb 2018 11:00:00 +0000
Los Techies would like to introduce, and extend a welcome to Derik Whittaker.
Derik is a C# MVP, member of the AspInsiders group, community speaker, and Pluralsight author.
Derik was previously a contributor at CodeBetter.com.  Welcome, Derik!


Ditch the Repository Pattern Already
Tue, 20 Feb 2018 21:00:00 +0000
One pattern that still seems particularly common among .Net developers is the Repository pattern. 
I began using this pattern with NHibernate around 2006 and only abandoned its use a few years ago.

I had read several articles over the years advocating abandoning the Repository
pattern in favor of other suggested approaches which served as a pebble in my
shoe for a few years, but there were a few design principles whose application
seemed to keep motivating me to use the pattern.  It wasn’t until a change of
tooling and a shift in thinking about how these principles should be applied
that I finally felt comfortable ditching the use of repositories, so I thought
I’d recount my journey to provide some food for thought for those who still
feel compelled to use the pattern.

Mental Obstacle 1: Testing Isolation

What I remember being the biggest barrier to moving away from the use of repositories was writing
tests for components which interacted with the database.  About a year or so
before I actually abandoned use of the pattern, I remember trying to stub out a
class derived from Entity Framework’s DbContext after reading an
anti-repository blog post.  I don’t remember the details now, but I remember it
being painful and even exploring use of a 3rd-party library designed to help
write tests for components dependent upon Entity Framework.  I gave up after a while,
concluding it just wasn’t worth the effort.  It wasn’t as if my previous
approach was pain-free, as at that point I was accustomed to stubbing out
particularly complex repository method calls, but as with many things we often
don’t notice friction to which we’ve become accustomed for one reason or
another.  I had assumed that doing all that work to stub out my repositories
was what I should be doing.

Another principle that I picked up from somewhere (maybe the big xUnit Test Patterns book? … I don’t remember) that seemed to keep me bound to my repositories was
that you shouldn’t write tests that depend upon dependencies you
don’t own.  I believed at the time that I should
be writing tests for Application Layer services (which later morphed into
discrete dispatched command handlers) and the idea of stubbing out either
NHIbernate or Entity Framework violated my sensibilities.

Mental Obstacle 2: The Dependency Inversion Principle Adherence

The Dependency Inversion Principle seems to be a source of confusion for many which
stems in part from the similarity of wording with the practice of
Dependency Injection
 as well as from the fact that the pattern’s formal definition
reflects the platform from whence the principle was conceived (i.e. C++).  One
might say that the abstract definition of the Dependency Inversion Principle
was too dependent upon the details of its origin (ba dum tss).  I’ve written
about the principle a few times (perhaps my most succinct being
this Stack Overflow answer),
but put simply, the Dependency Inversion Principle has at its
primary goal the decoupling of the portions of your application which define policy
from the portions which define implementation.  That is to say, this
principle seeks to keep the portions of your application which govern what your
application does (e.g. workflow, business logic, etc.) from being tightly
coupled to the portions of your application which govern the low level details
of how it gets done (e.g. persistence to an Sql Server database, use of Redis
for caching, etc.).

A good example of a violation of this principle, which I recall from my NHibernate
days, was that once upon a time NHibernate was tightly coupled to log4net. 
This was later corrected, but at one time the NHibernate assembly had a hard
dependency on log4net.  You could use a different logging library for your own
code if you wanted, and you could use binding redirects to use a different
version of log4net if you wanted, but at one time if you had a dependency on
NHibernate then you had to deploy the log4net library.  I think this went
unnoticed by many due to the fact that most developers who used NHibernate also
used log4net.

When I first learned about the principle, I immediately recognized that it seemed to
have limited advertized value for most business applications in light of what
Udi Dahan labeled
The Fallacy Of ReUse.  That is to say, properly understood, the Dependency
Inversion Principle has as its primary goal the reuse of components and keeping
those components decoupled from dependencies which would keep them from being
easily reused with other implementation components, but your application and
business logic isn’t something that is likely to ever be reused in a different
context.  The take away from that is basically that the advertized value of
adhering to the Dependency Inversion Principle is really more applicable to
libraries like NHibernate, Automapper, etc. and not so much to that workflow
your team built for Acme Inc.’s distribution system.  Nevertheless, the
Dependency Inversion Principle had a practical value of implementing an
architecture style Jeffrey Palermo labeled 
the Onion Architecture.
Specifically, in contrast to 
traditional 3-layered architecture models
where UI, Business, and Data Access layers precluded using
something like
Data Access Logic Components
to encapsulate an ORM to map data directly to entities within
the Business Layer, inverting the dependencies between the Business Layer and
the Data Access layer provided the ability for the application to interact with
the database while also seemingly abstracting away the details of the
data access technology used.

While I always saw the fallacy in strictly trying to apply the Dependency Inversion
Principle to invert the implementation details of how I got my data from my
application layer so that I’d someday be able to use the application in a
completely different context, it seemed the academically astute and in vogue
way of doing Domain-driven Design at the time, seemed consistent with the GoF’s
advice to program to an interface rather than an implementation, and provided
an easier way to write isolation tests than trying to partially stub out ORM
types.

The Catalyst

For the longest time, I resisted using Entity Framework.  I had become fairly
proficient at using NHibernate and I just saw it as plain stupid to use a
framework that was years behind NHibernate in features and maturity, especially
when it had such a steep learning curve.  A combination of things happened,
though.  A lot of the NHibernate supporters (like many within the Alt.Net
crowd) moved on to other platforms like Ruby and Node; anything with
Microsoft’s name on it eventually seems to gain market share whether it’s
better or not; and Entity Framework eventually did seem to mostly catch up with
NHibernate in features, and surpassed it in some areas. So, eventually I found
it impossible to avoid using Entity Framework which led to me trying to apply
the same patterns I’d used before with this newer-to-me framework.

To be honest, everything mostly worked, especially for the really simple stuff. 
Eventually, though, I began to see little ways I had to modify my abstraction
to accommodate differences in how Entity Framework did things from how
NHibernate did things.  What I discovered was that, while my repositories
allowed my application code to be physically decoupled from the ORM, the way I
was using the repositories was in small ways semantically coupled to the
framework.  I wish I had kept some sort of record every time I ran into
something, as the only real thing I can recall now were motivations with
certain design approaches to expose the SaveChanges method for

Unit of Work implementations
I don’t want to make more of the semantic coupling argument
against repositories than it’s worth, but observing little places where
my abstractions were leaking,
combined with the pebble in my shoe of developers who I felt
were far better than me were saying I shouldn’t use them lead me to begin
rethinking things.

More Effective Testing Strategies

It was actually a few years before I stopped using repositories that I stopped
stubbing out repositories.  Around 2010, I learned that you can use Test-Driven
Development to achieve 100% test coverage for the code for which you’re
responsible, but when you plug your code in for the first time with that team
that wasn’t designing to the same specification and not writing any tests at
all that things may not work.  It was then that I got turned on to Acceptance
Test Driven Development.  What I found was that writing high-level subcutaneous
tests (i.e. skipping the UI layer, but otherwise end-to-end) was overall
easier, was possible to align with acceptance criteria contained within a user
story, provided more assurance that everything worked as a whole, and was
easier to get teams on board with.  Later on, I surmised that I really
shouldn’t have been writing isolation tests for components which, for the most
part, are just specialized facades anyway.  All an isolation test for a facade
really says is “did I delegate this operation correctly” and if you’re not
careful you can end up just writing a whole bunch of tests that basically just
validate whether you correctly configured your mocking library.

So, by the time I started rethinking my use of repositories, I had long since stopped
using them for test isolation.

Taking the Plunge
It was actually about a year after I had become convinced that
repositories were unnecessary, useless abstractions that I started working with
a new codebase I had the opportunity to steer.  Once I eliminated them from the
equation, everything got so much simpler.   Having been repository-free for
about two years now, I think I’d have a hard time joining a team that had an
affinity for them.

Conclusion
If you’re still using repositories and you don’t have some
other hangup you still need to get over like writing unit tests for your
controllers or application services then give the repository-free lifestyle a
try.  I bet you’ll love it.


Using Manual Mocks to test the AWS SDK with Jest
Tue, 20 Feb 2018 13:56:45 +0000
Anytime you build Node applications it is highly suggested that your cover your code with tests.  When your code interacts with 3rd party API’s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests not unit tests.
If you are using Jest, one solution is utilize the built in support for manual mocks.  I have found the usage of manual mocks invaluable while testing 3rd party API’s such as the AWS.  Keep in mind just because I am using manual mocks this will remove the need for using libraries like SinonJs (a JavaScript framework for creating stubs/mocks/spies).
The way that manual mocks work in Jest is as follows (from the Jest website’s documentation).
Manual mocks are defined by writing a module in a __mocks__/ subdirectory immediately adjacent to the module. For example, to mock a module called user in the models directory, create a file called user.js and put it in the models/__mocks__ directory. Note that the __mocks__ folder is case-sensitive, so naming the directory __MOCKS__ will break on some systems. If the module you are mocking is a node module (eg: fs), the mock should be placed in the __mocks__ directory adjacent to node_modules (unless you configured roots to point to a folder other than the project root).
In my case I want to mock out the usage of the AWS-SDK for Node.
To do this I created a __mocks__ folder at the root of my solution.  I then created a aws-sdk.js file inside this folder.
Now that I have my mocks folder created with a aws-sdk.js file I am able to consume my manual mock in my jest test by simply referencing the aws-sdk via a require('aws-sdk') command.

const AWS = require('./aws-sdk');


With declaration of AWS above my code is able to a use the NPM package during normal usage, or my aws-sdk.js mock when running under the Jest context.
Below is a small sample of the code I have inside my aws-sdk.js file for my manual mock.

const stubs = require('./aws-stubs');
const AWS = {};

// This here is to allow/prevent runtime errors if you are using 
//   AWS.config to do some runtime configuration of the library.
// If you do not need any runtime configuration you can omit this.
AWS.config = {
    setPromisesDependency: (arg) => {}
};

AWS.S3 = function() {

}

// Because I care about using the S3 service's which are part of the SDK
//  I need to setup the correct identifier.
// 
AWS.S3.prototype = {
    ...AWS.S3.prototype,

    // Stub for the listObjectsV2 method in the sdk
    listObjectsV2(params){
        const stubPromise = new Promise((resolve, reject) => {
            // pulling in stub data from an external file to remove the noise
            // from this file.  See the top line for how to pull this in
            resolve(stubs.listObjects);
        });

        return {
            promise: () => {
                return stubPromise;
            }
        }
    }
};

// Export my AWS function so it can be referenced via requires
module.exports = AWS; 


A few things to point out in the code above.

I chose to use the promises implementation of the listObjectsV2.  Because of this I need to return a promise method as my result on my listObjectsV2 function.  I am sure there are other ways to accomplish this, but this worked and is pretty easy.
My function is returning stub data, but this data is described in a separate file called aws-stubs.js which sites along side of my aws-sdk.js file.  I went this route to remove the noise of having the stub data inside my aws-adk file.  You can see a full example of this here.

Now that I have everything setup my tests will no longer attempt to hit the actually aws-sdk, but when running in non-test mode they will.
Till next time,


Configure Visual Studio Code to debug Jest Tests
Fri, 16 Feb 2018 21:33:03 +0000
If you have not given Visual Studio Code a spin you really should, especially if  you are doing web/javascript/Node development.
One super awesome feature of VS Code is the ability to easily configure the ability to debug your Jest (should work just fine with other JavaScript testing frameworks) tests.  I have found that most of the time I do not need to actually step into the debugger when writing tests, but there are times that using console.log is just too much friction and I want to step into the debugger.
So how do we configure VS Code?
First you  will need to install the Jest-Cli NPM package (I am assuming you already have Jest setup to run your tests, if you do not please read the Getting-Started docs).  If you fail to do this step you will get the following error in Code when you try to run the debugger.

After you have Jest-Cli installed you will need to configure VS Code for debugging.  To do this open up the configuration by clicking Debug -> Open Configurations.  This will open up a file called launch.json.
Once launch.json is open add the following configuration

        {
            "name": "Jest Tests",
            "type": "node",
            "request": "launch",
            "program": "${workspaceRoot}/node_modules/jest-cli/bin/jest.js",
            "stopOnEntry": false,
            "args": ["--runInBand"],
            "cwd": "${workspaceRoot}",
            "preLaunchTask": null,
            "runtimeExecutable": null,
            "runtimeArgs": [
                "--nolazy"
            ],
            "env": {
                "NODE_ENV": "development"
            },
            "console": "internalConsole",
            "sourceMaps": false,
            "outFiles": []
        }


Here is a gist of a working launch.json file.
After you save the file you are almost ready to start your debugging.
Before you can debug you will want to open the debug menu (the bug icon on the left toolbar).   This will show a drop down menu with different configurations.  Make sure ‘Jest Test’ is selected.

If you have this setup correctly you should be able to set breakpoints and hit F5.
Till next time,


Going Async with Node AWS SDK with Express
Tue, 13 Feb 2018 13:00:30 +0000
When building applications in Node/Express you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary.  The ‘old school’ way was to use call backs, which often led to callback hell.  Than came along Promises which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything.  Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for async/await was introduced and this really has cleaned up and enhanced the readability of your code.
Having the ability to use async/await is great, and is supported out of the box w/ Express.  But what do you do when you using a library which still wants to use promises or callbacks? The case in point for this article is AWS Node SDK.
By default if you read through the AWS SDK documentation the examples lead you to believe that you need to use callbacks when implementing the SDK.  Well this can really lead to some nasty code in the world of Node/Express.  However, as of v2.3.0 of the AWS SDK there is support for Promises.  This is much cleaner than using callbacks, but still poses a bit of an issue if you want to use async/await in your Express routes.
However, with a bit of work you can get your promise based AWS calls to play nicely with your async/await based Express routes.  Lets take a look at how we can accomplish this.
Before you get started I am going to make a few assumptions.

You already have a Node/Express application setup
You already have the AWS SDK for Node installed, if not read here

The first thing we are going to need to do is add reference to our AWS SDK and configure it to use promises.

const AWS = require('aws-sdk');
AWS.config.setPromisesDependency(null);


After we have our SDK configured we can implement our route handler.  In my example here I am placing all the logic inside my handler.  In a real code base I would suggest better deconstruction of this code into smaller parts.

const express = require('express');
const router = express.Router();
const s3 = new AWS.S3();

router.get('/myRoute', async (req, res) => {

    const controller = new sitesController();
    const params = req.params;
    const params = { 
       Bucket: "bucket_name_here"
    };

    let results = {};
    var listPromise = s3.listObjects(params).promise();
    listPromise.then((data) => {
        results = data;
    });

    await Promise.all([listPromise]);

    res.json({data: results })
})

module.exports = router;


Lets review the code above and call out a few important items.
The first thing to notice is the addition of the async keyword in my route handler.  This is what allows us to use async/await in Node/Express.
The next thing to look at is how I am calling the s3.listObjects.  Notice I am NOT providing a callback to the method, but instead I am chaining with .promise().  This is what instructs the SDK to use promises vs callbacks.  Once I have my callback I chain a ‘then’ in order to handle my response.
The last thing to pay attention to is the line with await Promise.All([listPromise]); This is the magic forces our route handler to not return prior to the resolution of all of our Promises.  Without this your call would exit prior to the listObjects call completing.
Finally, we are simply returning our data from the listObjects call via res.json call.
That’s it, pretty straight forward, once you learn that the AWS SDK supports something other than callbacks.
Till next time,


Unable To Access Mysql With Root and No Password After New Install On Ubuntu
Wed, 31 Jan 2018 00:13:00 +0000

    This bit me in the rear end again today.  Had to reinstall mysql-server-5.7 for other reasons.

You just installed mysql-server locally for your development environment on a recent version of Ubuntu (I have 17.10 artful installed).  You did it with a blank password for root user.  You type mysql -u root and you see Access denied for user 'root'@'localhost'.



Issue:
Because you chose to not have a password for the root user, the auth_plugin for my MySQL defaulted to auth_socket.  That means if you type sudo mysql -u root you will get in.  If you don’t, then this is NOT the fix for you.

Solution:
Change the auth_plugin to mysql_native_password so that you can use the root user in the database.

$ sudo mysql -u root

mysql> USE mysql;
mysql> UPDATE user SET plugin='mysql_native_password' WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> exit;

$ sudo systemctl restart mysql
$ sudo systemctl status mysql


NB
ALWAYS set a password for mysql-server in staging/production.

Cheers.

    Unable To Access Mysql With Root and No Password After New Install On Ubuntu was originally published by Jason Meridth at Jason Meridth on January 30, 2018.


New Job
Mon, 08 Jan 2018 18:13:00 +0000

    Well, it is a new year and I’ve started a new job.  I am now a Senior Software Engineer at True Link Financial.



After interviewing with the co-founders Kai and Claire and their team, I knew I wanted to work here.

TL;DR: True Link:  We give elderly and disable (really, anyone) back their financial freedom where they may not usually have it.

Longer Version:  Imagine you have an elderly family member who may start showing signs of dimensia.  You can give them a True Link card and administer their card.  You link it to their bank account or another source of funding and you can set limitations on when, where and how the card can be used.  The family member feels freedom by not having to continually ask for money but is also protected by scammers and non-friendly people (yep, they exist).

The customer service team, the marketing team, the product team, the engineering team and everyone else at True Link are amazing.

For any nerd readers, the tech stack is currently Rails, React, AWS, Ansible.  We’ll be introducing Docker and Kubernetes soon hopefully, but always ensuring the right tools for the right job.

Looking forward to 2018.

Cheers.


    New Job was originally published by Jason Meridth at Jason Meridth on January 08, 2018.