Los Techies

A Lap Around ActivitySource and ActivityListener in .NET 5

Mon, 04 Jan 2021 20:40:16 +0000

Part of the new DiagnosticSource API are new ways of "listening" in to activities with the addition of the ActivitySource and ActivityListener APIs. These are intended to replace the DiagnosticSource and DiagnosticListener APIs. However, the latter two types aren't deprecated, and aren't being removed from the existing usages. However, ActivitySource/Listener represent a pretty big leap forward in usability and performance over the old APIs.

I've shown a little bit on the ActivitySource side of things, but there's the other side - listening to activity events. This is where the new API starts to show its benefits. The previous DiagnosticListener API used nested observables without much insight other than the events. You had one "callback" for when a new DiagnosticSource was available, then another for individual events:

using var sub = DiagnosticListener.AllListeners.Subscribe(listener =>
{
    Console.WriteLine($"Listener name {listener.Name}");

    listener.Subscribe(kvp => Console.WriteLine($"Received event {kvp.Key}:{kvp.Value}"));
});

The events didn't have any semantic meaning. The "key" for the event was whatever you wanted. The "value" was object, and could again be anything. Some events used semantic names for "Start" and "Stop" events, some did not. Some events had some context object passed in through the Value, some were anonymous types that required reflection (gross!) to get values out.

In the new world, the API is separated into the individual concerns when instrumenting:

Should I listen to these activities?
Should I sample this activity?
Notify me when an activity starts.
Notify when an activity stops.

With a DiagnosticListener, the DiagnosticListener.AllListeners property is an IObservable<DiagnosticListener>, from which you only have the Name to make a decision to listen to events.

The new API combines all of these into one single object:

public sealed class ActivityListener : IDisposable
{
    public Action<Activity>? ActivityStarted { get; set; }
    public Action<Activity>? ActivityStopped { get; set; }

    public Func<ActivitySource, bool>? ShouldListenTo { get; set; }

    public SampleActivity<string>? SampleUsingParentId { get; set; }

    public SampleActivity<ActivityContext>? Sample { get; set; }

    public void Dispose() => ActivitySource.DetachListener(this);
}

The SampleActivity type is a delegate:

public delegate ActivitySamplingResult SampleActivity<T>(ref ActivityCreationOptions<T> options);

This means we can have different sampling decisions based on how much detail we have. On Dispose, something nice, our ActivityListener automatically disconnects from the ActivitySource.

Attaching an ActivityListener is similar to DiagnosticListener, except now we simply call a method to add our listener:

ActivitySource.AddActivityListener(listener);

We now get explicit callbacks for Start/Stop, as well as for Sampling. Instrumentation libraries will use the Sample property to decide the level of sampling for this activity - some of which will eventually make it to trace context headers.

In my typical use case, I'm using sampling to do integration testing, so I'll typically have my listener set to ActivitySamplingResult.AllData. With the DiagnosticListener, there were no different levels of listening. Someone was either listening, or not, with no levels in between.

Next, we get distinct ActivityStarted and ActivityStopped callbacks, where we get the entire Activity instead of a KeyValuePair<string, object?>, much richer and focused API:

using var listener = new ActivityListener
{
    ShouldListenTo = _ => true,
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData,
    ActivityStarted = activity => Console.WriteLine($"{activity.ParentId}:{activity.Id} - Start"),
    ActivityStopped = activity => Console.WriteLine($"{activity.ParentId}:{activity.Id} - Stop")
};

ActivitySource.AddActivityListener(listener);

One thing we don't have now is callbacks for generic events as we could do with the DiagnosticListener from before. It turns out all this information is now on our Activity:

Links
Tags
Events
Baggage

Rather than having a mystery meat object of the KeyValuePair from before, our Activity now has much more information (thanks to the OpenTelemetry folks for standardizing this information).

But what about existing users of DiagnosticListener? Well it turns out we can still get access to Activity start/stop events because Activity has a default Source that gets initialized with a blank Name. Although DiagnosticListener will not "forward" events to an ActivitySource, it's certainly possible.

While most folks will never really need to touch the Diagnostics API, the improvements with .NET 5 and following OpenTelemetry concepts mean the future is bright for telemetry and instrumentation - even if you only turn AppInsights on and forget about it.

Increasing Trace Cardinality with Activity Tags and Baggage

Wed, 16 Dec 2020 21:03:17 +0000

One of the first "oh no" moments for folks new to distributed tracing is the needle in the haystack problem. Someone reports an error, you go look for traces in your tool (Azure Monitor or whatever), and because there are thousands of traces, you can't easily figure out which is your trace that you want to find. Fundamentally, this is the challenge of cardinality. We want a high enough cardinality of data to be able to effectively find our individual trace when searching.

OpenTelemetry and the Activity API give us two main ways to add additional information to spans/traces:

Attributes (tags)
Baggage

In the Activity API, attributes are "Tags" and Baggage is still "Baggage", since these names predate OpenTelemetry.

The key difference between these two are propagation. Both are key-value pairs of information, but only Baggage propagates to subsequent traces. This means that intraprocess communication needs a means to propagate - and that's exactly what the W3C Baggage standard describes.

We do have to be careful about baggage, however, as it will accumulate. Anything added to baggage will show up in all child spans.

Tags and Baggage in Activities

With the activity API, it's quite straightforward to add tags and baggage to the current activity. Suppose we want to include some operation ID as part of our trace with a tag:

[HttpGet]
public async Task<ActionResult<Guid>> Get(string message)
{
    var command = new SaySomething
    {
        Message = message,
        Id = Guid.NewGuid()
    };

    Activity.Current?.AddTag("cart.operation.id", command.Id.ToString());

The AddTag and AddBaggage methods let us add these key/value pairs of data. If we're using OpenTelemetry, our tags will automatically show up in a trace:

This tag can be searched on, making sure we only find the trace we're interested in:

Searching tag value in Zipkin

If you're already using logging contexts, this is probably quite familiar to you.

While tags are great for including additional information about a single span, they don't propagate, so you might have a harder time correlating multiple sets of data together, such as "find all queries executed for cart 123". You might only know the cart at one specific span, but not all the way down in all the related activities.

For that, we can use Baggage:

[HttpGet]
public async Task<ActionResult<Guid>> Get(string message)
{
    var command = new SaySomething
    {
        Message = message,
        Id = Guid.NewGuid()
    };
    
    Activity.Current?.AddBaggage("cart.operation.id", command.Id.ToString());

For both interprocess and intraprocess Activities, the Baggage will propagate. Here we can see baggage propagate through the headers in RabbitMQ:

Baggage and trace context in headers in RabbitMQ message

Unfortunately, baggage won't automatically show up in our traces, we'll have to do something special to get those to show up.

Reporting baggage in telemetry data

While this context information gets passed through all of our spans, it won't necessarily show up in our tracing tools. This is because attributes are the primary reporting mechanism for information for traces, and baggage is a larger concept that can be used by logs, traces, and metrics to enrich each of those. It might seem counterintuitive that baggage is not automatically reported, but it's simply because it's a broader concept intended to be consumed by other observability pillars.

For us, if we want to just automatically include all baggage in tags as they are recorded, we can do so by registering a simple ActivityListener at startup:

public static void Main(string[] args)
{
    var listener = new ActivityListener
    {
        ShouldListenTo = _ => true,
        ActivityStopped = activity =>
        {
            foreach (var (key, value) in activity.Baggage)
            {
                activity.AddTag(key, value);
            }
        }
    };
    ActivitySource.AddActivityListener(listener);

    CreateHostBuilder(args).Build().Run();
}

With this in each of my applications, I can ensure that all baggage gets shipped as tags out to my tracing system:

Baggage showing up in span for database interaction

Above, I can see my baggage I set way up in a Controller made it all the way down to a MongoDB call. It's crossed several process boundaries to get there:

Trace highlighted with span far from baggage origination

Typically, we'll also include this baggage context in our structured logs with Serilog:

class BaggageEnricher : ILogEventEnricher
{
    public void Enrich(LogEvent logEvent, 
        ILogEventPropertyFactory propertyFactory)
    {
        if (Activity.Current == null)
            return;

        foreach (var (key, value) in Activity.Current.Baggage)
        {
            logEvent.AddPropertyIfAbsent(propertyFactory.CreateProperty(key, value));
        }

    }
}

With baggage, we get a bit of overhead since it will piggyback everywhere. However, we can start to leverage our baggage to include contextual information that we find valuable in logs, traces, and metrics. Common data might be:

Business identifiers (cart ID etc)
Workflow/batch identifiers
Session identifiers
Machine information
User information

You'll have to, of course, follow privacy laws for some of this information (maybe don't log tax numbers), so care is needed here. In practice, it's been invaluable to take some business information (say, a cart ID), and see all traces related to it, not just information only available in a single trace.

By including high cardinality data in our logs and traces, we can far more quickly locate what we're looking for, instead of resorting to small time windows as I've had to in the past.

.Net Project Builds with Node Package Manager

Thu, 10 Dec 2020 07:00:00 +0000

A few years ago, I wrote an article entitled Separation of Concerns: Application Builds & Continuous Integration wherein I discussed the benefits of separating project builds from CI/CD concerns by creating a local build script which lives with your project. Not long after writing that article, I was turned on to what I’ve come to believe is one of the easiest tools I’ve encountered for managing .Net project builds thus far: npm.

Most development platforms provide a native task-based build technology. Microsoft’s tooling for these needs is MSBuild: a command-line tool whose build files double as Visual Studio’s project and solution definition files. I used MSBuild briefly for scripting custom build concerns for a couple of years, but found it to be awkward and cumbersome. Around 2007, I abandoned use of MSBuild for creating builds and began using Rake. While it had the downside of requiring a bit of knowledge of Ruby, it was a popular choice among those willing to look outside of the Microsoft camp for tooling and had community support for working with .Net builds through the Albacore library. I’ve used a few different technologies since, but about 5 years ago I saw a demonstration of the use of npm for building .Net projects at a conference and I was immediately sold. When used well, it really is the easiest and most terse way to script a custom build for the .Net platform I’ve encountered.

“So what’s special about npm?” you might ask. The primary appeal of using npm for building applications is that it’s easy to use. Essentially, it’s just an orchestration of shell commands.

Tasks

With other build tools, you’re often required to know a specific language in addition to learning special constructs peculiar to the build tool to create build tasks. In contrast, npm’s expected package.json file simply defines an array of shell command scripts:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "echo Clean the project.",
    "restore": "echo Restore dependencies.",
    "compile": "echo Compile the project.",
    "test": "echo Run the tests.",
    "dist": "echo Create a distribution."
  },
  "author": "Some author",
  "license": "ISC"
}

As with other build tools, NPM provides the ability to define dependencies between build tasks. This is done using pre- and post- lifecycle scripts. Simply, any task issued by NPM will first execute a script by the same name with a prefix of “pre” when present and will subsequently execute a script by the same name with a prefix of “post” when present. For example:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "echo Clean the project.",
    "prerestore": "npm run clean",
    "restore": "echo Restore dependencies.",
    "precompile": "npm run restore",
    "compile": "echo Compile the project.",
    "pretest": "npm run compile",
    "test": "echo Run the tests.",
    "prebuild": "npm run test",
    "build": "echo Publish a distribution."
  },
  "author": "Some author",
  "license": "ISC"
}

Based on the above package.json file, issuing “npm run build” will result in running the tasks of clean, restore, compile, test, and build in that order by virtue of each declaring an appropriate dependency.

Given you’re okay with limiting a fully-specified dependency chain where a subset of the build can be initiated at any stage (e.g. running “npm run test” and triggering clean, restore, and compile first) , the above orchestration can be simplified by installing the npm-run-all node dependency and defining a single pre- lifetime script for the main build target:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "echo Clean the project.",
    "restore": "echo Restore dependencies.",
    "compile": "echo Compile the project.",
    "test": "echo Run the tests.",
    "prebuild": "npm-run-all clean restore compile test",
    "build": "echo Publish a distribution."
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

In this example, issuing “npm run build” will result in the prebuild script executing npm-run-all with the parameters: clean, restore, compile and test which it will execute in the order listed.

Variables

Aside from understanding how to utilize the pre- and post- lifecycle scripts to denote task dependencies, the only other thing you really need to know is how to work with variables.

Node’s npm command facilitates the definition of variables by command-line parameters as well as declaring package variables. When npm executes, each of the properties declared within the package.json are flattened and prefixed with “npm_package_”. For example, the standard “version” property can be used as part of a dotnet build to denote a project version by referencing ${npm_package_version}:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "configuration": "Release",
  "scripts": {
    "build": "dotnet build ./src/*.sln /p:Version=${npm_package_version}"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

Command-line parameters can also be passed to npm and are similarly prefixed with “npm_config_” with any dashes (“-”) replaced with underscores (“_”). For example, the previous version setting could be passed to dotnet.exe in the following version of package.json by issuing the below command:

```npm run build --product-version=2.0.0```

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "configuration": "Release",
  "scripts": {
    "build": "dotnet build ./src/*.sln /p:Version=${npm_config_product_version}"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

(Note: the parameter –version is an npm parameter for printing the version of npm being executed and therefore can’t be used as a script parameter.)

The only other important thing to understand about the use of variables with npm is that the method of dereferencing is dependent upon the shell used. When using npm on Windows, the default shell is cmd.exe. If using the default shell on Windows, the version parameter would need to be deference as %npm_config_product_version%:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "configuration": "Release",
  "scripts": {
    "build": "dotnet build ./src/*.sln /p:Version=%npm_config_product_version%"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

Until recently, I used a node package named “cross-env” which allows you to normalize how you dereference variables regardless of platform, but for several reasons including cross-env being placed in maintenance mode, the added dependency overhead, syntax noise, and support for advanced variable expansion cases such as default values, I’d recommend any cross-platform execution be supported by just standardizing on a single shell (e.g. “Bash”). With the introduction of Windows Subsystem for Linux and the virtual ubiquity of git for version control, most developer Windows systems already contain the bash shell. To configure npm to use bash at the project level, just create a file named .npmrc at the package root containing the following line:

script-shell=bash

Using Node Packages

While not necessary, there are many CLI node packages that can be easily leveraged for aiding in authoring your builds. For example, a package named “rimraf”, which functions like Linux’s “rm -rf” command, is a utility you can use to implement a clean script for recursively deleting any temporary build folders created as part of previous builds. In the following package.json build, a package target builds a NuGet package which it outputs to a dist folder in the package root. The rimraf command is used to delete this temp folder as part of the build script’s dependencies:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "rimraf dist",
    "prebuild": "npm run clean",
    "build": "dotnet pack ./src/ExampleLibrary/ExampleLibrary.csproj -o dist /p:Version=${npm_package_version}"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5",
    "rimraf": "^3.0.2"
  

Building End-to-End Diagnostics: ActivitySource and OpenTelemetry 1.0
Tue, 08 Dec 2020 20:01:16 +0000
Posts in this series:
An Intro
Trace Context
Diagnostic Events
OpenTelemetry Integration
Activity and Span Correlation
Visualization with Exporters
User-Defined Context with Correlation Context
ActivitySource and OpenTelemetry 1.0
It's a few months, and quite a lot has changed in the tracing landscape in .NET. As OpenTelemetry marches towards 1.0, the .NET team had to make a decision. Given there is a lot of code out there using the existing tracing/telemetry APIs in both the .NET codebase and various SDKs, how should .NET 5 best support the OpenTelemetry specification for:
Logging
Tracing
Metrics
The tracing specification is solid at this point (but not yet "1.0"), and the tracing specification is not just the W3C standards on the wire, but the span APIs themselves.
Since the Activity API already exists, and it's already being used, it was far easier to support a "Span" concept as part of an Activity and close the gaps in functionality.
The second major addition in the .NET 5 release was the addition of the ActivitySource/ActivityListener APIs, which make it quite a bit simpler to raise and listen to events for Activity start/stop.
The overall plan was to convert from DiagnosticListener -based activity interaction to the new ActivitySource API. But before we do that, let's look at the current state of our packages.
Current Packages
Before I made any changes, you would see the code that started the activity be fairly light, something like:
_diagnosticListener.OnActivityImport(activity, context);

if (_diagnosticListener.IsEnabled(StartActivityName, context))
{
    _diagnosticListener.StartActivity(activity, context);
}
else
{
    activity.Start();
}
 The code starting the Activity would need to do all the work to see if the diagnostic listener was enabled, and if it were, then use the listener to start the activity. You could then pass in a context object that could be...anything. The code then receiving the listener event would then do whatever it wanted, and in the case of OpenTelemetry, it would enrich the event with additional tags, create a Span, then record it.
That left my two packages as:
Xyz.Extensions.Diagnostics - raise a diagnostic listener event
Xyz.Extensions.OpenTelemetry - listen to the diagnostic listener event, add tags, and enlist with OpenTelemetry infrastructure
In the "new" world, we really want to treat our generated Activity as our first-class "Span", and not something that some other package needs to muck around with. This means a substantial change for our packages:
Xyz.Extensions.Diagnostics - create ActivitySource, add tags, start/stop event through ActivitySource
Xyz.Extensions.OpenTelemetry - ??? maybe nothing?
I'll come back to the second package, but the big change is moving all the tag creation code from the OpenTelemetry package over to the Diagnostics one.
This is a good thing, because now we don't have to rely on some external package to add interesting telemetry information to our Activity, we've already added it!
Converting to ActivitySource
Instead of a DiagnosticListener, we'll want to use an ActivitySource, and for this part, I'll follow the guidance on this API from OpenTelemetry. We create an ActivitySource that we'll share for all activities in our application:
internal static class NServiceBusActivitySource
{
    private static readonly AssemblyName AssemblyName 
        = typeof(NServiceBusActivitySource).Assembly.GetName();
    internal static readonly ActivitySource ActivitySource 
        = new (AssemblyName.Name, AssemblyName.Version.ToString());
}
Per conventions, the activity source name should be the name of the assembly creating the activities. That makes it much easier to "discover" activities, you don't have to expose a constant or search through source code to discern the name.
In our main methods for dealing with the Activity, we can take advantage of Activity implementing IDisposable to automatically stop the activity:
public override async Task Invoke(
    IIncomingPhysicalMessageContext context,
    Func<Task> next)
{
    using (StartActivity(context))
    {
        await next().ConfigureAwait(false);

        if (_diagnosticListener.IsEnabled(EventName))
        {
            _diagnosticListener.Write(EventName, context);
        }
    }
}
I've kept a DiagnosticListener here for backward compatibility purposes. The StartActivity method takes a lot of the header business we did earlier, with one slight variation - using the renamed W3C  Baggage spec:
if (context.MessageHeaders.TryGetValue(Headers.BaggageHeaderName, out var baggageValue)
   || context.MessageHeaders.TryGetValue(Headers.CorrelationContextHeaderName, out baggageValue))
{
    var baggage = baggageValue.Split(',');
    if (baggage.Length > 0)
    {
        foreach (var item in baggage)
        {
            if (NameValueHeaderValue.TryParse(item, out var baggageItem))
            {
                baggageItems.Add(new KeyValuePair<string, string?>(baggageItem.Name, HttpUtility.UrlDecode(baggageItem.Value)));
            }
        }
    }
}
Finally, starting our activity is a little different. We use the ActivitySource to start, passing along the parentId if we found it in the incoming headers:
var activity = parentId == null
    ? NServiceBusActivitySource.ActivitySource.StartActivity(
        ActivityNames.IncomingPhysicalMessage, 
        ActivityKind.Consumer)
    : NServiceBusActivitySource.ActivitySource.StartActivity(
        ActivityNames.IncomingPhysicalMessage, 
        ActivityKind.Consumer, 
        parentId);

if (activity == null)
{
    return activity;
}
Now instead of checking the diagnosticListener to see if any one is listening, StartActivity returns null when there are no listeners. We can simply return.
If there is someone listening, we can enrich our Activity with data:
activity.TraceStateString = traceStateString;

_activityEnricher.Enrich(activity, context);

foreach (var baggageItem in baggageItems)
{
    activity.AddBaggage(baggageItem.Key, baggageItem.Value);
}

return activity;
I've encapsulated all of the context reading of headers and setting of tags in a separate ActivityEnricher object:
public void Enrich(Activity activity, IIncomingPhysicalMessageContext context)
{
    var destinationName = _settings.LogicalAddress();
    const string operationName = "process";
    activity.DisplayName = $"{destinationName} {operationName}";

    activity.AddTag("messaging.message_id", context.Message.MessageId);
    activity.AddTag("messaging.operation", operationName);
    activity.AddTag("messaging.destination", destinationName);
    activity.AddTag("messaging.message_payload_size_bytes", context.Message.Body.Length.ToString());
I've included everything I could match on the OpenTelemetry semantic conventions.
There was one extra piece, however, which was configuring for "extra" information that I didn't want to turn on by default.
Configuring Span Attributes
In my original OpenTelemetry integration, I included a way to record the message contents as part of your trace. However, now that the Activity needs to include all the relevant information, I needed to have a way to configure the tags when it's actually started and stopped.
To do so, I integrated with the configuration of the extension point of this feature in NServiceBus. I first created a class to hold all of the configuration settings (just one for now):
public class InstrumentationOptions
{
    public bool CaptureMessageBody { get; set; }
}
Next, I had my Activity enrichment class take the settings as a constructor argument:
internal class SettingsActivityEnricher : IActivityEnricher
{
    private readonly ReadOnlySettings _settings;
    private readonly InstrumentationOptions _options;

    public SettingsActivityEnricher(ReadOnlySettings settings)
    {
        _settings = settings;
        _options = settings.Get<InstrumentationOptions>();
    }
Now when I enrich the Activity, I'll also look at these settings:
if (activity.IsAllDataRequested 
    && _options.CaptureMessageBody)
{
    activity.AddTag("messaging.message_payload", Encoding.UTF8.GetString(context.Message.Body));
}
Then when I register the NServiceBus behavior, I'll look for these settings:
public DiagnosticsFeature()
{
    Defaults(settings => settings.SetDefault<InstrumentationOptions>(new InstrumentationOptions
    {
        CaptureMessageBody = false
    }));
    EnableByDefault();
}

protected override void Setup(FeatureConfigurationContext context)
{
    var activityEnricher = new SettingsActivityEnricher(context.Settings);

    context.Pipeline.Register(new IncomingPhysicalMessageDiagnostics(activityEnricher), "Parses incoming W3C trace information from incoming messages.");
    context.Pipeline.Register(new OutgoingPhysicalMessageDiagnostics(activityEnricher), "Appends W3C trace information to outgoing messages.");
All this code is very specific to NServiceBus, so for any custom diagnostics, they'll need to plug in to whatever extensibility model that exists for the middleware they're building. For MongoDB for example, I had a direct constructor:
var clientSettings = MongoClientSettings.FromUrl(mongoUrl);
var options = new InstrumentationOptions { CaptureCommandText = true };
clientSettings.ClusterConfigurator = cb => cb.Subscribe(new DiagnosticsActivityEventSubscriber(options));
var mongoClient = new MongoClient(clientSettings);
With all this in place, now I just needed to integrate with OpenTelemetry (though this is really optional).
OpenTelemetry Integration
Now that OpenTelemetry and System.Diagnostics.DiagnosticSource are more aligned, registering and listening to activities in OpenTelemetry is as simple as registering a source:
services.AddOpenTelemetryTracing(builder =>  builder
    .AddAspNetCoreInstrumentation()
    .AddSqlClientInstrumentation(opt => opt.SetTextCommandContent = true)
    .AddSource("NServiceBus.Extensions.Diagnostics")
    .AddZipkinExporter(o =>
    {
        o.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
        o.ServiceName = Program.EndpointName;
    })
Or, to make things a little easier, I created a tiny little package, the old NServiceBus.Extensions.Diagnostics.OpenTelemetry one that wraps that one line:
public static class TracerProviderBuilderExtensions
{
    public static TracerProviderBuilder AddNServiceBusInstrumentation(this TracerProviderBuilder builder)
        => builder.AddSource("NServiceBus.Extensions.Diagnostics");
}
It's not required, but at this point, this is all we need to bridge ActivitySource-enabled telemetry with OpenTelemetry.
So that's all for now - OpenTelemetry marches to 1.0, and once it does, I'll update my end-to-end example (it's on the RC now).

  



Conventional Options
Fri, 20 Nov 2020 07:00:00 +0000
I’ve really enjoyed working with the Microsoft Configuration libraries introduced with .Net Core approximately 5 years ago.  The older XML-based API was quite a pain to work with, so the ConfigurationBuilder and associated types provided a long overdue need for the platform.

I had long since adopted a practice of creating discrete configuration classes populated and registered with a DI container over direct use of the ConfigurationManager class within components, so I was pleased to see the platform nudge developers in this direction through the introduction of the IOptions type.

A few aspects surrounded the prescribed use of the IOptions type of which I wasn't particularly fond were needing to inject IOptions rather than the actual options type, taking a dependency upon the Microsoft.Extensions.Options package from my library packages, and the cermony of binding the options to the IConfiguration instance.  To address these concerns, I wrote some extension methods which took care of binding the type to my configuration by convention (i.e. binding a type with a suffix of Options to a section corresponding to the option type's prefix) and registering it with the container.

I’ve recently released a new version of these extensions supporting several of the most popular containers as an open source library.  You can find the project here.

The following are the steps for using these extensions:

Step 1
Install ConventionalOptions for the target DI container:

$> nuget install ConventionalOptions.DependencyInjection


Step 2
Add Microsoft’s Options feature and register option types:

  services.AddOptions();
  services.RegisterOptionsFromAssemblies(Configuration, Assembly.GetExecutingAssembly());


Step 3
Create an Options class with the desired properties:

    public class OrderServiceOptions
    {
        public string StringProperty { get; set; }
        public int IntProperty { get; set; }
    }


Step 4
Provide a corresponding configuration section matching the prefix of the Options class (e.g. in appsettings.json):

{
  "OrderService": {
    "StringProperty": "Some value",
    "IntProperty": 42
  }
}


Step 5
Inject the options into types resolved from the container:

    public class OrderService
    {
        public OrderService(OrderServiceOptions options)
        {
            // ... use options
        }
    }


Currently ConventionalOptions works with Microsoft’s DI Container, Autofac, Lamar, Ninject, and StructureMap.

Enjoy!


Vertical Slice Example Updated to .NET 5
Thu, 19 Nov 2020 20:22:38 +0000
With the release of .NET 5, I wanted to update my Vertical Slice code example (Contoso University) to .NET 5, as well as leverage all the C# 9 goodness. The migration itself was very simple, but updating the dependencies along the way proved to be a bit more of a challenge.
Updating the runtime and dependencies
The first step was migrating the target framework to .NET 5. This is about as simple as it gets:
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
-    <TargetFramework>netcoreapp3.1</TargetFramework>
+    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

</Project>
Next, I needed to update the dependencies. This application hadn't had its dependencies for a few months, so most of the changes were around that. Otherwise, any Microsoft.* dependency updated to a 5.0 version:
  <ItemGroup>
    <PackageReference Include="AutoMapper" Version="10.1.1" />
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="8.1.0" />
-   <PackageReference Include="DelegateDecompiler.EntityFrameworkCore" Version="0.28.0" />
-   <PackageReference Include="FluentValidation.AspNetCore" Version="9.2.0" />
-   <PackageReference Include="HtmlTags" Version="8.0.0" />
+   <PackageReference Include="DelegateDecompiler.EntityFrameworkCore5" Version="0.28.2" />
+   <PackageReference Include="FluentValidation.AspNetCore" Version="9.3.0" />
+   <PackageReference Include="HtmlTags" Version="8.1.1" />
    <PackageReference Include="MediatR" Version="9.0.0" />
    <PackageReference Include="MediatR.Extensions.Microsoft.DependencyInjection" Version="9.0.0" />
    <PackageReference Include="MiniProfiler.AspNetCore.Mvc" Version="4.2.1" />
    <PackageReference Include="MiniProfiler.EntityFrameworkCore" Version="4.2.1" />
-   <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="3.1.9" />
-   <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="3.1.9">
+   <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="5.0.0" />
+   <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="5.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
-   <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="3.1.9" />
-   <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="3.1.4" />
+   <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="5.0.0" />
+   <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="5.0.0" />
  </ItemGroup>
This was as simple as updating my packages to the latest version. From here, I compiled, ran the build, and ran the app. Everything worked as expected.
Some folks are put off by the fact that 5.0 is not "LTS" but that really shouldn't stop you from using it - you can always upgrade to the next LTS when it comes out.
One package I did need to change was the DelegateDecompiler.EntityFrameworkCore one, and this led to my last post and finally a new package that explicitly targets EF Core 5. This is because EF Core changed an API that broke the existing package, so  it was easier to create a new package than make the multi-targeting more complicated. With all the packages and runtime update, I next wanted to see how C# 9 might improve things.
Updating to C# 9
The runtime upgrade on a vanilla ASP.NET Core 3.1 application brings with it lots of performance improvements, but I was really looking forward to the C# 9 improvements (which many technically work on lower versions, but that's a different story).
C# 9 has a lot major and minor improvements, but a few I was really interested in:
Records
Init-only setters
Pattern-matching
Static lambda functions
First up are records. I wasn't quite sure if records were supported everywhere in my application, but I wanted to look at them for DTOs, which were all the request/response objects used with MediatR:
Request and responses with handlers
Incoming request objects use ASP.NET Core data binding, which does support records, and outgoing objects use AutoMapper with ProjectTo, so I needed to understand if EF Core 5 supported record types for LINQ projections.
It should however, as I've tested LINQ expressions and compilation and reflection, and they all still work with record types. Converting my DTOs to record types was simple, but I did need to decide between positional vs. nominal record types. I decided to go with nominal as I found it easier to read:
public record Query : IRequest<Command>
{
    public int? Id { get; init; }
}

public record Command : IRequest
{
    public int Id { get; init; }
    public string LastName { get; init; }

    [Display(Name = "First Name")]
    public string FirstMidName { get; init; }

    public DateTime? EnrollmentDate { get; init; }
}
I changed class to record and set to init, and now I've got an "immutable" record type. From there, I needed to fix some compile errors which were mainly around mutating DTOs. That wound up making the code easier to understand/follow, because instead of creating the DTO and mutating several times, I could either gather all the data I needed initially, or as part of the initialization expression:
var results = await students
    .ProjectTo<Model>(_configuration)
    .PaginatedListAsync(pageNumber, pageSize);

var model = new Result
{
    CurrentSort = message.SortOrder,
    NameSortParm = string.IsNullOrEmpty(message.SortOrder) ? "name_desc" : "",
    DateSortParm = message.SortOrder == "Date" ? "date_desc" : "Date",
    CurrentFilter = searchString,
    SearchString = searchString,
    Results = results
};      
Initially this code would instantiate the Result object, mutate it a few places, then return it. I found the places that mutated the DTOs to be much more confusing, which I'm sure the functional folks are scoffing at.
Less impactful were the pattern matching with switch expressions, but it did provide some improvement:
//before
students = message.SortOrder switch
{
    case "name_desc":
        students = students.OrderByDescending(s => s.LastName);
        break;
    case "Date":
        students = students.OrderBy(s => s.EnrollmentDate);
        break;
    case "date_desc":
        students = students.OrderByDescending(s => s.EnrollmentDate);
        break;
    default: // Name ascending 
        students = students.OrderBy(s => s.LastName);
        break;
}

//after
students = message.SortOrder switch
    "name_desc" => students.OrderByDescending(s => s.LastName),
    "Date" => students.OrderBy(s => s.EnrollmentDate),
    "date_desc" => students.OrderByDescending(s => s.EnrollmentDate),
    _ => students.OrderBy(s => s.LastName)
}
It's small, but removes a lot of the noise of the code. Finally, I tried to use the static lambdas as much as I could when the lambda didn't capture any closure variables (or, shouldn't).
All in all, a very easy migration. Next, I'll be updating my OSS to C# 9 where I can (without using breaking features).

  



Mind Your Strings with .NET 5.0
Tue, 27 Oct 2020 14:05:20 +0000
In the process of upgrading a library to support .NET 5.0, I ran into a rather bizarre test failure. Simply by adding a new target of net5.0 in the library and unit tests, single test started to fail. It was absolutely baffling to me, leading my to think that I may have somehow broken strings. Instead, I got to learn waaaaaay more about strings, Unicode, and comparison strategies than I ever really cared to.
But for those migrating to .NET 5.0, you'll want to mind your string methods quite closely.
The Faulty Assertion
It started off with a test failure:
[Test]
public void TestDetailOneGroupTwoClassesSupported()
{
    //SETUP
    MasterEnvironment.ResetLogging();
    var classLog1 = new ClassLog(@"TestGroup01UnitTestGroup\Test01MyUnitTest1");
    MasterEnvironment.AddClassLog(classLog1);
    example2Supported.ForEach(classLog1.MethodLogs.Add);
    var classLog2 = new ClassLog(@"TestGroup01UnitTestGroup\Test01MyUnitTest2");
    MasterEnvironment.AddClassLog(classLog2);
    example2Supported.ForEach(classLog2.MethodLogs.Add);

    //ATTEMPT
    var markup = MasterEnvironment.ResultsAsMarkup(OutputVersions.Detail);

    //VERIFY
    markup.ShouldStartWith("Detail");
    markup.ShouldContain("Group: Unit Test Group");
    markup.ShouldContain("\n#### [My Unit Test1](");
    markup.ShouldContain("\n#### [My Unit Test2]("); // <- this blew up
    markup.ShouldContain("\n- Supported\n  * Good1 (line 1)\n  * Good2 (line 2)");
}
The next to last assertion blew up on .NET 5.0 but worked on all other platforms tested (.NET Core 2.0, 3.0, 3.1, .NET 4.5). I first verified that the value to test was exactly the same (it was), then looked at the assertion. This library uses NUnit, and eventually makes a comparison using string.IndexOf(value, StringComparison.CurrentCulture). As I was debugging, I put the values into the watch window and got some rather strange results:
IndexOf not matching Contains
Well that's a bit disconcerting - I ask "do you contain this value". "Yes". "Where". "I dunno". How can this be? I went back to .NET Core 3.1, and saw that the two methods were consistent, it said the value was contained AND it found the right place.
So I didn't break strings, but SOMETHING was different here. I opened a GitHub issue to understand why this behavior seemingly broke.
Unicode Strikes Again
Underneath the covers, IndexOf and Contains do subtly different things, it turns out. While IndexOf is an ordinal search, Contains is a linguistic search, which, depends on what kind of linguistic searching strategy I would use. NUnit using CurrentCulture means it's searching linguistically (the default). But why would the result change based on just upgrading the target platform? Why do I see such different behavior of strings based on the platform?



Method
netcoreapp3.1
net5.0




actual.Contains(expected)
True
True


actual.IndexOf(expected)
1475
-1


actual.Contains(expected, StringComparison.CurrentCulture)
True
False


actual.IndexOf(expected, StringComparison.CurrentCulture)
1475
-1


actual.Contains(expected, StringComparison.Ordinal)
True
True


actual.IndexOf(expected, StringComparison.Ordinal)
1475
1475


actual.Contains(expected, StringComparison.InvariantCulture)
True
False


actual.IndexOf(expected, StringComparison.InvariantCulture)
1475
-1



It turns out my string I was searching was....weird. It contained quite a few instances of Lorem Ipsum\n\r\ndolor sit amet, where it's mixing different line ending styles, then searching \ndolor. That would split the \r\n value into two.
So what changed? It turns out for globalization services, the Windows default is Natural Language Support (NLS). But for Unix-based platform, that standard is International Components for Unicode (ICU). The switch in .NET 5.0 is to move towards this more widely-accepted standard, ICU. ICU will be part of Windows and can be consumed using a NuGet package, but in my code, the behavior was always different between Unix and Windows. If I ran .NET Core 3.1 on Unix, I'd get the same result as .NET 5.0.
So how to fix this?
Back on the table above, we can see that an Ordinal comparison results in consistent behavior, and searching behavior I actually want. I changed my test to use Ordinal and everything passed.
But what should you do? For a start do NOT rely on any string methods that do not explicitly pass in a StringComparison value. Unless you can readily pull up the underlying source code, it's not obvious what the default string comparison value is, and this default is not consistent across string methods.
From there, I recommend following the Microsoft guidelines for string usage. If you're upgrading to .NET 5.0, be aware of ICU and how to turn it off if it's a problem.
For more reading, check out the original GitHub issue, and you'll learn all about exciting words such as "grapheme".

  



MediatR 9.0 Released
Fri, 09 Oct 2020 13:05:01 +0000
A small release, from the release notes:
This release contains a small, but breaking change. In order to provide a simpler interface, the IMediator interface is now split into two interfaces, a sender and publisher:
public interface ISender
{
    Task<TResponse> Send<TResponse>(IRequest<TResponse> request, CancellationToken cancellationToken = default);

    Task<object?> Send(object request, CancellationToken cancellationToken = default);
}

public interface IPublisher
{
    Task Publish(object notification, CancellationToken cancellationToken = default);

    Task Publish<TNotification>(TNotification notification, CancellationToken cancellationToken = default)
        where TNotification : INotification;
}

public interface IMediator : ISender, IPublisher
{
}
The main motivation here is that sending should be a top-level concern of an application, but publishing can happen anywhere. This interface segregation should help catch design errors, where should never send requests from anywhere inside a request handler.
I've also updated the MediatR.Extensions.Microsoft.DependencyInjection package to 9.0 as well. Enjoy!

  



Constrained Open Generics Support Merged in .NET Core DI Container
Tue, 25 Aug 2020 13:51:00 +0000
Continuing the trend of extremely long blog titles...years ago I opened a pull request for supporting constrained open generics in the built-in Microsoft.Extensions.DependencyInjection container. And another. And another....and another. But I did not give up hope - after porting through a couple repo moves and consolidations, this feature is now merged and will get released with .NET 5 (huzzah).
So what is this mouthful going to get you? Well, for users of MediatR, quite a lot. Something I see quite a lot when building generic-oriented code are declarations of IFoo<T> where you have a method accepting that T. For example, Fluent Validation defines an IValidator<T> interface that is (roughly): 
public interface IValidator<in T> {
    ValidationResult Validate(T instance);
}
Notice the contravariance here - we can use more derived types in the T parameter. I often do this in validation when I have many different screens that share some common trait, or validation:
public class UserInfoValidator : IValidator<IContainUserInfo> {
    public ValidationResult Validate(IContainUserInfo instance) {
    }
}
All DTOs that then want this common validation only need to implement the IContainUserInfo interface.
This is all well and good, but there is a slight problem here. My type parameter has lost its original type information, because I've closed the IValidator interface, so if I want to plug in additional dependencies based on that type, it's gone.
Constraining instead of closing the generic type preserves the generic parameter:
public class UserInfoValidator<T> : IValidator<T>
    where T : IContainUserInfo
{
    public ValidationResult Validate(T instance) {
        // I can still use T's members of IContainUserInfo
    }
}
What's important here is that because I still have the type parameter, I can now include dependencies here that still can be based on the type T. For example, I could do something like:
public class UserInfoValidator<T> : IValidator<T>
    where T : IContainUserInfo
{
    public UserInfoValidator(IEnumerable<IPermission<T>> permissions)
        => _permissions = permissions;
In this example, I depend on another collection of permission dependencies to validate my user information. However, because my generic type is still open, at runtime, only the IPermission<T> instances that satisfy T will be satisfied and supplied.
There's a gap in .NET to make this sort of check much easier, and I've opened an API proposal to help. Please vote if this proposal interests you!
Constrained generics are a powerful tool to apply cross-cutting concerns to generics in dependency injection, giving us a "DI-enabled" sort of pattern matching. The pull request that merged provides a simple fix (don't blow up).
What Took So Long?
Great question, especially since the change is so small (adding a try-catch), but it exposed a challenge that the MS team has to worry about. The built-in container is a conforming container - that is, it provides some default behaviors that ASP.NET Core requires, and if you want to replace that container, you need to conform to those behaviors.
This is a challenge for the DI container library authors, who have built their own libraries over the years, independently, and now need to conform to the needs and design of the MS.Ext.DI container. Conversely, external library authors (such as myself) want to build on top of the default DI container without needing to provide a bunch of library-specific extensions (MediatR.Autofac, MediatR.Lamar, MediatR.TinyIoc etc).
But what happens if you want to depend on a feature that:
Exists in most or all popular containers
Does not exist in the conforming container (MS.Ext.DI)
Is not needed by ASP.NET Core (the primary user)
That's where constrained open generics sat. It's not needed by the primary users (various .NET hosts) but is needed, or wanted, by a large number of people that use these .NET hosts built on top of the conforming container.
So, it's a risk - so what I wound up needing to do was:
Show that the majority (or all) of the containers support this behavior
Document this behavior explicitly with a set of specification tests
These specification tests run against the built-in container plus 8 popular ones (Autofac, Unity, Lamar etc.)
With this in place, I could show the expected behavior of this feature in DI containers, and prove that yes indeed, all the listed containers do support this feature.
And with that, and a little try..catch, this feature will drop in .NET 5.

  



Diagnostics and Instrumentation Packages Targeting Open Telemetry Beta for MongoDB and NServiceBus Published
Thu, 06 Aug 2020 19:47:16 +0000
That's...a long title. I've published release packages for System.Diagnostics support for NServiceBus and MongoDB:
NServiceBus.Extensions.Diagnostics
MongoDB.Driver.Core.Extensions.DiagnosticSources
And published new beta packages targeting the beta release of OpenTelemetry:
NServiceBus.Extensions.Diagnostics.OpenTelemetry
MongoDB.Driver.Core.Extensions.OpenTelemetry
The Diagnostics packages are separate from the OpenTelemetry ones and only reference System.Diagnostics package. The OpenTelemetry ones now use OpenTelemetry beta, which switches to solely using Activity for recording OpenTelemetry spans, instead of mapping to a Span.

  



End-to-End Integration Testing with NServiceBus: How It Works
Tue, 14 Jul 2020 20:16:56 +0000
In my last post, I walked through setting up end-to-end integration testing with NServiceBus, and how we can use it to black box test message endpoints similar to how the ASP.NET Core integration testing works. In this post, I want to walk through how it all works underneath the covers.
The major challenge in testing an async messaging flow is observability - how do we know when the work is started or completed? Our tests are written in an imperative, top-down, "Arrange-Act-Assert" style but the system under test (SUT) is intentionally asynchronous.
In my end-to-end diagnostics series, I walked through adding observability to our message endpoints, and it turns out, we can leverage this same mechanism to observe our SUT in our integration tests.
What is done?
In order to know what we need to test, we need to understand what it means for the "Act" step in a test to complete successfully. In a normal synchronous interaction, the "Act" is guaranteed to complete when the method returns - whether this call is "awaited" or not, it's still blocking until the response is returned.
With async messaging, our calling code only blocks while until the message is "ack"d:
var firstMessage = new FirstMessage {Message = "Hello World"};

var session = _factory.Services.GetService<IMessageSession>();

await _session.SendLocal(firstMessage);

// OK now what?
We could do something silly like an await Task.Delay(something??) but that's not terribly efficient. Those task delays easily add up, and we could have finished far earlier but just not known it.
In an end-to-end message endpoint scenario, there are some top-level indications my work is complete:
When a specific message is processed
When a specific message is sent
Depending on my scenario, when one of these conditions is met, it's now safe to make my assertions of the messages or the handler's expected side effects.
When we have an external source of signals we're waiting to receive, we don't want to wait forever. We'll still want a timeout - a maximum amount of time we wait for those observable conditions to be met, otherwise we'll just timeout.
For those that have used various BDD or QA testing tools, this is pretty common for UI tests. We wait for the browser to make its calls and update things, and wait for some expected UI state to move on, but not wait forever. It's not much different, except instead of waiting for some DOM element to show up, we're waiting for messages.
Observing Messages
To be notified for messages sent or processed, we can first look at our diagnostic events we created in the end-to-end tracing posts. Our "Act" code needs to look for specific messages sent/received. Unfortunately, the same diagnostic event we used for telemetry is a bit lower level than we want - for example, the messages are transport-oriented, with byte[] messages bodies and the like.
Luckily, NServiceBus has an additional pipeline hook for logical messages sent/received, and this context object has more relevant information we'd want in our tests. We can then expose more diagnostic events at the logical message level, instead of physical. First, a behavior for processing messages:
public class IncomingLogicalMessageDiagnostics : Behavior<IIncomingLogicalMessageContext>
{
    private readonly DiagnosticListener _diagnosticListener;
    private const string EventName = ActivityNames.IncomingLogicalMessage + ".Processed";

    public IncomingLogicalMessageDiagnostics(DiagnosticListener diagnosticListener)
        => _diagnosticListener = diagnosticListener;

    public IncomingLogicalMessageDiagnostics()
        : this(new DiagnosticListener(ActivityNames.IncomingLogicalMessage))
    {
    }

    public override async Task Invoke(IIncomingLogicalMessageContext context, Func<Task> next)
    {
        await next().ConfigureAwait(false);

        if (_diagnosticListener.IsEnabled(EventName))
        {
            _diagnosticListener.Write(EventName, context);
        }
    }
}
In this example, I'm not using Activity information, I'm simply raising a single diagnostic event since the physical message behavior already does all of the telemetry work. We await the next step in the pipeline, and raise the event. This ensures our diagnostic event only raises when the message is successfully handled.
The parameter we include in the diagnostic event, the IIncomingLogicalMessageContext, includes the deserialized message and its type information. That way if we want to observe a specific message type with specific data, we can.
Next, we create a similar pipeline behavior for outgoing logical messages:
public class OutgoingLogicalMessageDiagnostics : Behavior<IOutgoingLogicalMessageContext>
{
    private readonly DiagnosticListener _diagnosticListener;
    private const string EventName = ActivityNames.OutgoingLogicalMessage + ".Sent";

    public OutgoingLogicalMessageDiagnostics(DiagnosticListener diagnosticListener)
        => _diagnosticListener = diagnosticListener;

    public OutgoingLogicalMessageDiagnostics()
        : this(new DiagnosticListener(ActivityNames.OutgoingLogicalMessage)) { }

    public override async Task Invoke(IOutgoingLogicalMessageContext context, Func<Task> next)
    {
        await next().ConfigureAwait(false);

        if (_diagnosticListener.IsEnabled(EventName))
        {
            _diagnosticListener.Write(EventName, context);
        }
    }
}
Almost exactly the same! With our diagnostic events defined and set up in behaviors (this is in the NServiceBus.Extensions.Diagnostics package), we can listen in on these events in our integration test.
Observing Diagnostics Events in a Test
I boiled the "wait for an event" method to a single method in a fixture:
private static async Task<ObservedMessageContexts> ExecuteAndWait<TMessageContext>(
    Func<Task> testAction,
    Func<TMessageContext, bool> predicate,
    TimeSpan? timeout = null)
    where TMessageContext : IPipelineContext
{
This method takes a generic parameter - an IPipelineContext, which is either the incoming or outgoing logical message contexts. I need the test action to perform as a Func<Task> as I will need to call this after setting up the event observation. Finally, I need to know when to "stop" observing, with a predicate around the pipeline context information and a timeout.
The return value is a collection of all the observed message contexts - so the "Assert" step can look at everything sent/received.
First, I want to default a timeout and initialize some collectors:
timeout = Debugger.IsAttached
    ? (TimeSpan?)null
    : timeout ?? TimeSpan.FromSeconds(10);

var incomingMessageContexts = new List<IIncomingLogicalMessageContext>();
var outgoingMessageContexts = new List<IOutgoingLogicalMessageContext>();
var obs = Observable.Empty<object>();
The Activity API is a two-level set of observables. The first is when the initial diagnostic listener starts up, and the second is when diagnostic events are published. It's a little confusing to be sure, but not terrible. We want to first subscribe to all listeners when the two listener names we care about are published:
using var allListenerSubscription = DiagnosticListener.AllListeners
    .Subscribe(listener =>
    {
        switch (listener.Name)
        {
            case ActivityNames.IncomingLogicalMessage:
                // Subscribe to the incoming listener

                break;
            case ActivityNames.OutgoingLogicalMessage:
                // Subscribe to the outgoing listener

                break;
        }
    };
When we run our application, our Subscribe method will be called for all diagnostic listeners starting up - the ASP.NET Core ones, HttpClient ones, SqlClient ones etc. but we only care about the NServiceBus logical message listeners.
Once we have this set up, we need to do 2 things inside those each switch block:
Capture the observable so that we can await it based on our predicate later
Capture all contexts in those lists
For each, we can rely on Reactive Extensions via the System.Reactive.Core library. Here's the incoming logical message block:
var incomingObs = listener
    .Select(e => e.Value)
    .Cast<IIncomingLogicalMessageContext>();

incomingObs.Subscribe(incomingMessageContexts.Add);

if (typeof(TMessageContext) ==  typeof(IIncomingLogicalMessageContext))
{
    obs = obs.Merge(incomingObs);
}

break;
The listener passed in is an IObservable<KeyValuePair<string, object>> where the object is that message context object. The first thing we can do is use LINQ to treat this observable as IObservable<*message context type>. Next, we add a single subscription action to simply capture the IIncomingLogicalMessageContext values on the list we created earlier.
Next, because I can test multiple hosts and endpoints, we Merge the different listener observables into our single obs variable we declared earlier, as our diagnostic listener object is created per endpoint.
The outgoing message switch block looks almost identical:
var outgoingObs = listener
    .Select(e => e.Value)
    .Cast<IOutgoingLogicalMessageContext>();

outgoingObs.Subscribe(outgoingMessageContexts.Add);

if (typeof(TMessageContext) == typeof(IOutgoingLogicalMessageContext))
{
    obs = obs.Merge(outgoingObs);
}

break;
The idea is we're only listening to incoming or outgoing messages, so I use that generic parameter to understand which one we care about.
Now that we've got our captured and merged observable, we can construct or final observable that includes the predicate and timeout, if exists (it won't if we're debugging):
var finalObs = obs.Cast<TMessageContext>().TakeUntil(predicate);
if (timeout != null)
{
    finalObs = finalObs.Timeout(timeout.Value);
}
With our observable in place, we can execute our test action and await for our observable to complete (either by satisfying our predicate or timing out):
await testAction();

// Force the observable to complete
await finalObs;

return new ObservedMessageContexts(
    incomingMessageContexts, 
    outgoingMessageContexts);
Once the final observable sequence completes, we can return back to our test the list of observed incoming/outgoing message contexts we capture earlier.
With this in place, our tests have a simple filter they can pass in based on their logical message type, with all the sent/received messages to assert against:
[Fact]
public async Task Should_execute_orchestration_saga()
{
    var client = _fixture.WebAppHost.CreateClient();

    var message = Guid.NewGuid().ToString();

    var response =
        await ExecuteAndWaitForHandled<SaySomethingResponse>(() =>
            client.GetAsync($"saysomething?message={message}"), 
            TimeSpan.FromSeconds(30));

    var saySomethingResponses = response.ReceivedMessages.OfType<SaySomethingResponse>().ToList();
    saySomethingResponses.Count.ShouldBe(1);
    saySomethingResponses[0].Message.ShouldContain(message);
}
It took a little bit to get the observables to work correctly, but once in, we can easily add end-to-end integration testing for complex messaging scenarios. Observability has a long tail of payoffs, as I found with integration testing.

  



End-to-End Integration Testing with NServiceBus
Wed, 08 Jul 2020 16:43:12 +0000
One of the interesting side effects of adding diagnostic events to infrastructure is that you can now "listen in" to what's going on in your applications for black box testing. This can be especially useful in scenarios where you're building on top of a framework that includes a lot of built-in behavior, such as ASP.NET Core and NServiceBus.
In ASP.NET Core, you can write tests directly against your controllers/handlers/pages, but that won't execute the entire request pipeline, only a little snippet. Similarly, you can write a test against an NServiceBus handler, but there's a lot more behavior going on around it.
To get an idea that the entire application can execute successfully, you need to actually run the application, which can be a huge challenge for message-oriented architectures that are by nature asynchronous.
To help with this, I created the NServiceBus.Extensions.IntegrationTesting project that leverages the diagnostics events I blogged about to listen in to messages sent/received.
Usage
I created a rather complex scenario in that series, in which an API call would result in either an orchestration or choreography-based workflow, depending on the API call I used. Eventually, this worklow terminates, but the only way I could test this entire flow was to run the entire system.
If I want to test the entire end-to-end system, I can start with the integration testing capabilities with ASP.NET Core. This package creates a special test host and test HTTP client that executes the entire pipeline, but all in-memory.
I want to do similar with NServiceBus, except using the Learning Transport instead of a "real" transport. That way, I don't have to worry about provisioning queues in test environments.
The NServiceBus integration test project includes two extensions:
A helper method to configure your endpoint for integration testing
A fixture class that observes the diagnostic events around some supplied action
In our Xunit tests, we create a derived WebApplicationFactory that includes the test setup for our NServiceBus host:
public class TestFactory : WebApplicationFactory<HostApplicationFactoryTests>
{
    public EndpointFixture EndpointFixture { get; }

    public TestFactory() => EndpointFixture = new EndpointFixture();

    protected override IHostBuilder CreateHostBuilder() =>
        Host.CreateDefaultBuilder()
            .UseNServiceBus(ctxt =>
            {
                var endpoint = new EndpointConfiguration("HostApplicationFactoryTests");

                endpoint.ConfigureTestEndpoint();

                return endpoint;
            })
            .ConfigureWebHostDefaults(b => b.Configure(app => {}));
In the above example, we're using a Worker Service that doesn't run a web host. However, the WebApplicationFactory still expects one, so we create an empty web application. The ConfigureTestEndpoint method configures our endpoint to run in a test mode (turn off auditing, retries, etc.)
Finally, in our test class, we add our fixture:
public class HostApplicationFactoryTests 
    : IClassFixture<HostApplicationFactoryTests.TestFactory>
{
    private readonly TestFactory _factory;

    public HostApplicationFactoryTests(TestFactory factory) => _factory = factory;
Now in our test, we execute our "Send" and wait for a message to be handled that we expect:
[Fact]
public async Task Can_send_and_wait()
{
    var firstMessage = new FirstMessage {Message = "Hello World"};

    var session = _factory.Services.GetService<IMessageSession>();

    var result = await _factory
        .EndpointFixture
        .ExecuteAndWaitForHandled<FinalMessage>(() => session.SendLocal(firstMessage));

    result.IncomingMessageContexts.Count.ShouldBe(3);
    result.OutgoingMessageContexts.Count.ShouldBe(3);

    result.ReceivedMessages.ShouldNotBeEmpty();

    var message = result.ReceivedMessages.OfType<FinalMessage>().Single();

    message.Message.ShouldBe(firstMessage.Message);
}
This simple workflow is 3 different message handlers in a row, and kick things off with an initial message. However, we can get more complicated and create a fixture that includes multiple different test hosts:
public class SystemFixture : IDisposable
{
    public WebAppFactory WebAppHost { get; }

    public WebApplicationFactory<Program> WorkerHost { get; }

    public ChildWorkerServiceFactory ChildWorkerHost { get; }

    public EndpointFixture EndpointFixture { get; }
Our integration test can now send an API call via the web host, and it kick off the messages to execute the entire saga until it's complete:
[Fact]
public async Task Should_execute_orchestration_saga()
{
    var client = _fixture.WebAppHost.CreateClient();

    var message = Guid.NewGuid().ToString();

    var response =
        await _fixture.EndpointFixture.ExecuteAndWaitForHandled<SaySomethingResponse>(() =>
            client.GetAsync($"saysomething?message={message}"), 
            TimeSpan.FromSeconds(30));

    var saySomethingResponses = response.ReceivedMessages.OfType<SaySomethingResponse>().ToList();
    saySomethingResponses.Count.ShouldBe(1);
    saySomethingResponses[0].Message.ShouldContain(message);
}
This snippet executes the entire workflow:
Orchestration workflow
It kicks off the test with the initial POST, then waits until the final REPLY to complete the test. Each application runs in a test host, listening to messages, sending messages, making API calls etc. I just need to know what "Done" should be, so I wait for the SaySomethingResponse message to be handled.
The EndpointFixture class also lets me listen for messages sent, in case "done" is some message that gets sent but not handled by these systems.
I find these kinds of integration tests especially useful for process manager/saga scenarios, where I'm pushing over the first domino, and I want to test that the last domino falls successfully. I still expect to have tests all along the way, but I want to have something that tests the entire scenario end-to-end in an environment that runs the entire stack.
In the next post, I'll walk through the code in the integration tests extensions project to show how I used diagnostics events and Reactive Extensions to "know" when to stop.

  



Becoming a Technical Fellow at Headspring, and New Beginnings
Tue, 23 Jun 2020 19:46:43 +0000
My journey at Headspring started for me at a local developer meetup, the "Austin Agile Lunch" group, way back in 2007. I had started blogging and getting involved in the community, attending the local .NET user group, code camp, and most influentially, the first ALT.NET open space conference. At this lunch, Jeffrey Palermo, who had started at Headspring that year, asked the innocuous question "Do you know anyone looking for a job?" And the naive me answered "No, but if I know of anyone, I'll be glad to point them your way!" Jeffrey, always direct, tried again, "Do you know anyone named JIMMY looking for a job?"
I had been quite unhappy at my current gig (as anyone who's seen my talks about it can attest), but wasn't quite sure what was next. I had done a cash-flow challenged startup, a product company, and Big Corp IT, all in the span of the first 4 years of my career. Although Headspring was only 5 people at the time, it offered something I couldn't get anywhere else. An opportunity to drive for excellence in an environment where everyone else had a similar passion, and an ability to affect positive change at a level I couldn't dream of before.
Within a year or so after I started, thanks to support and encouragement from coworkers and the opportunities I now had, I received my first Microsoft MVP award, co-authored my first book, published my first OSS project used in production, and wrote dozens of blog posts of everything I was working on, all in trying to live up to a promise I made to myself to pass on what I had learned as I had indebted myself to so many others who shared their own learnings.
To pay it forward.
This culminated in Headspring recognizing me as the first-ever Technical Fellow, an honor I never set out to receive, and I'm even more indebted to others to repay.
And now, 13 years later, my Headspring journey shifts (again).
For some years, I looked at going independent as the next logical step for my career. However, I love working at Headspring, the people, the clients, and the work, and could not imaging giving that up, for almost any opportunity. I recognized that although I could ask for the flexibility of an independent consultant, I would not ask for, nor accept, that sort of special treatment. As it became clear that I could go independent, I approached the Headspring leadership to look at possibilities of being independent while also at Headspring.
Thanks to the great support of Dustin and rest of the Headspring leadership, we were able to come to a mutually beneficial agreement. While I'm continuing my role as the Chief Architect at Headspring, I've stepped back as a full-time employee and staying on as an independent contractor (Jimmy Bogard Consulting LLC, I'm not going to win any awards for creativity).
My role as Chief Architect I hope (and plan) to continue for years to come, as I believe in Headspring's vision and future, and we've signed a partnership agreement to that effect. What changes for me is my day-to-day projects and clients may or may not be strictly Headspring projects and clients, or any projects at all (maybe become a true brisket master?). My hope is to be able to have the flexibility to consult on the kinds of opportunities that are only feasible as an independent consultant, while still helping to fulfill the vision and mission of Headspring that I've helped to define and shape for over a decade.
As my grandfather was fond of saying, "Onward and upward".

  



Building End-to-End Diagnostics: User-Defined Context with Correlation Context
Mon, 22 Jun 2020 13:06:04 +0000
Posts in this series:
An Intro
Trace Context
Diagnostic Events
OpenTelemetry Integration
Activity and Span Correlation
Visualization with Exporters
User-Defined Context with Correlation Context
Source Code
With a brief detour to push out some NuGet packages, I wanted to pick up with a common issue folks run into once they start implementing distributed tracing. And that problem is one of locality - trying to find one trace among many, the needle in the haystack. In the past, I've resorted to combing through logs, using timestamps as a very poor means of trying to diagnose an issue.
We can decorate our logs with context information to make searching easier, but when it comes to our traces, how can we triangulate something a user did with a workflow and the associated traces?
This is where the emerging Correlation Context standard comes in to play - the ability for application code to add arbitrary information to traces - and critically - have that information flow through subsequent spans.
With distributed trace spans, you can add information to a span so that it eventually shows up in a collector/exporter. This is possible today with the System.Diagnostics.Activity API through tags:
Activity.Current.AddTag("user.id", user.Id);
But that information does not flow through to any subsequent spans within a process, or to subsequent processes. It exists for a single Activity/span, then it's gone.
This is where Correlation Context comes in. Trace context spec defines a "tracestate" header for vendor-specific trace information to propagate, and Correlation Context allows application code to add application-specific trace information to propagate.
Propagating Correlation Context in NServiceBus
ASP.NET Core will automatically parse correlation context header information, and places this in Activity.Baggage:
string[] baggage = headers.GetCommaSeparatedValues(HeaderNames.CorrelationContext);
if (baggage.Length > 0)
{
    foreach (var item in baggage)
    {
        if (NameValueHeaderValue.TryParse(item, out var baggageItem))
        {
            activity.AddBaggage(baggageItem.Name.ToString(), HttpUtility.UrlDecode(baggageItem.Value.ToString()));
        }
    }
}
Baggage, unlike Tags, will flow through to child Activities. When an Activity starts, the parent Activity.Current will flow its ParentId through to the new Activity. When you access an Activity's Baggage, the implementation pulls the current baggage, its Parent's baggage, and every parent up the chain.
With NServiceBus, we want to also parse incoming baggage, and propagate outgoing baggage through the Correlation-Context header. And although this header is still in draft mode with the W3C, it already has implementation support in ASP.NET Core 3.0.
To parse incoming headers, we can do the same operation that ASP.NET Core does, in our code that parses the incoming traceparent, we can look for the Correlation-Context header and place the values in Activity.Baggage:
if (context.MessageHeaders.TryGetValue(Headers.CorrelationContextHeaderName, out var correlationContext))
{
    var baggage = correlationContext.Split(',');
    if (baggage.Length > 0)
    {
        foreach (var item in baggage)
        {
            if (NameValueHeaderValue.TryParse(item, out var baggageItem))
            {
                activity.AddBaggage(baggageItem.Name, HttpUtility.UrlDecode(baggageItem.Value));
            }
        }
    }
}
Now that we have correlation context in our Baggage, any other child activities will have this baggage, too.
The last piece is propagation, in our original outgoing NServiceBus behavior that propagates traceparent:
if (!context.Headers.ContainsKey(Headers.CorrelationContextHeaderName))
{
    var baggageItems = activity.Baggage.Select(item => $"{item.Key}={item.Value}");
    var headerValue = string.Join(",", baggageItems);
    if (!string.IsNullOrEmpty(headerValue))
    {
        context.Headers[Headers.CorrelationContextHeaderName] = headerValue;
    }
}
Will now propagate the baggage out through the Correlation-Context header. With the incoming and outgoing header behaviors in place, any service can drop some data into baggage and have it propagate to all downstream services.
So all done, right? Well, not quite, as even though we added information to the Activity.Baggage, it doesn't necessarily mean that those values get exported to our tracing tools. Unfortunately today, OpenTelemetry exporters only consider the Tags portion of an Activity for exporting. This will be opt-in in the future, but for now, we'll need to manually copy our Baggage to Tags during the export process (or in child activities).
In the next post, we'll walk through exactly that - creating a "breadcrumb" Activity that piggybacks on events of other activities.

  



Diagnostics and Instrumentation Packages for MongoDB and NServiceBus Published
Wed, 17 Jun 2020 14:25:00 +0000
As part of the end-to-end diagnostics and tracing blog series, I had an end goal of eventually publishing out NuGet packages for diagnostics (Activity and DiagnosticSource). I'm happy to announce that I've released 4 packages to NuGet:
NServiceBus.Extensions.Diagnostics
NServiceBus.Extensions.Diagnostics.OpenTelemetry
MongoDB.Driver.Core.Extensions.DiagnosticSources
MongoDB.Driver.Core.Extensions.OpenTelemetry
These packages add:
Activity and DiagnosticsSource support
OpenTelemetry adapters
W3C Trace Context and Correlation Context support for NServiceBus
With the diagnostics packages, you can write your own DiagnosticListeners to subscribe to diagnostics Activity events. The OpenTelemetry packages listen to these Activity events and adapt them to OpenTelemetry spans. The various existing exporters can output these spans to Zipkin, Jaeger, Application Insights, and many other observability tools.
I've updated the ReadMe's for each repo to include instructions for how to use:
Mongo.Driver.Core.Extensions.DiagnosticSources
NServiceBus.Extensions.Diagnostics
With these packages, you can easily configure OpenTelemetry to include MongoDB and NServiceBus support:
services.AddOpenTelemetry(builder => 
    builder
        .UseZipkin(o =>
        {
            o.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
            o.ServiceName = EndpointName;
        })
        .AddNServiceBusAdapter(opt => opt.CaptureMessageBody = true)
        .AddMongoDBAdapter(opt => opt.CaptureCommandText = true)
        .AddRequestAdapter()
        .AddDependencyAdapter(configureSqlAdapterOptions: opt => opt.CaptureTextCommandContent = true);
);
Enjoy!

  



Picking a Web Microframework
Wed, 27 May 2020 00:23:00 +0000
I’ve had to use this at work the last couple of weeks. We had a “home grown” framework for a new application 
we’re working and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization
anyway to do routes, etc).

However, being that the group I was working with is pretty “anti framework” I had to settle on something that was light weight, integrated
with jetty and allowed us to work the way that was comfortable for us as team (also it had to work with Scala).

Microframeworks

The team had shown a lot of disdain for Play (which I had actually quite a lot when I last was leading a JVM based tech stack) and Spring Boot as being too heavy weight, so these
were definitely out.

Fortunately, in the JVM world there is a big push back now on heavy web frameworks so meant I had lots of choices for “non frameworks” but could
still do some basic security, routing, authentication but not hurt the existing team’s productivity.

There are probably 3 dozen microframeworks to choose from with varying degrees of value but the two that seemed to easiest to start with today were:


  Scalatra
  Javalin
  Quarkus


My Attempt with Quarkus

Quarkus has a really great getting started story but it’s harder to get started on an existing project with it, it was super trivial to add, and after a couple of days of figuring out the magic incantation I just decided to punt on it.
I think because of it’s popularity in the Cloud Native space (which we’re trying to target), the backing of Red Hat, and the pluggable nature of the stack there are a lot of reasons to want this to work. 
In the end because of the timeline it didn’t make the cut. But it may come back.

My Attempt with Javalin

Javalin despite being a less popular project than Quarkus it is getting some buzz. It also looks like it just slides into the team’s existing Servlet code base. I wanted this to work very badly but stopped before 
I even started because of this issue so this was out despite being on paper a really execellent framework.

My Attempt with Scalatra

Scalatra has been around for a number of years and is inspired by Sinatra which I used quite a bit in my Ruby years. 
This took a few minutes to get going just following their standalone directions and then some more to successful convert the routes
and account for learning curves with routes.

Some notes:


  The routing API and parameters etc are very nice to work with IMO.
  It was very easy to get json by default support setup.
  Metrics were very easy to wire up.
  Swagger integration was pretty rough, while it looks good on paper I could not get an example to show up, and it is unable to handle case classes or enums which we use.
  Benchmark performance when I’ve looked around the web was pretty bad, I’ve not done enough to figure out if this is real or not. I’ve seen first hand a lot of benchmarking are just wrong.
  Integration with JUnit has been rough and I cannot seem to get the correct port to fire, I suspect I have to stop using the @Test annotation is all (which I’m not enjoying).
  Http/2 support is still lacking despite being available in the version of Jetty they’re on, I’ve read a few places that an issue is keeping web sockets working but either way there is no official support in the project yet.


Conclusion

I think we’re going to stick with Scalatra for the time being as it is a muture framework that works well for our current goals. However, the lack of http/2 support maybe a deal breaker in the medium term.


Building End-to-End Diagnostics: Visualization with Exporters
Tue, 19 May 2020 14:46:42 +0000
Posts in this series:
An Intro
Trace Context
Diagnostic Events
OpenTelemetry Integration
Activity and Span Correlation
Visualization with Exporters
Source Code
In the last post, we looked at surfacing our diagnostics events from the Activity and DiagnosticSource APIs on through to OpenTelemetry Spans.
Side note - the .NET team is working to close the gap between the OTel "Span" concept and the Activity API. The end goal is that the Activity class represents an OTel "Span". 
With all of these pieces in place, we can now export these traces to...well, something! The benefit of this common OTel API means that it should be much easier to export your traces to some collector that can then store, index, and visualize your traces. But before we look at visualization, let's build a couple of complex but contrived scenarios for a more real-world scenario distributed system.
Orchestration Sample
I wanted to try to use as many different distributed components as possible that I typically see in my client work, in an arrangement of different conversation patterns, so I wanted an example system that did:
Request/Response with HTTP
Request/Response with AMQP
Pub/Sub with AMQP
Orchestration and Choreography with AMQP
Database interaction with SQL and MongoDB
There's probably someone winning a distributed systems bingo at this point. My sample application will have 3 different endpoints:
Web Application (receives the first request)
Worker Application (Worker SDK)
Child Worker Application (another Worker SDK)
Because I'm lazy, I'll kick things off with the Swagger UI, and initiate 2 kinds of interactions:
Process manager using orchestration
Process manager using choreography
First, let's look at an orchestration conversation:
Conversation using orchestration with HTTP and AMQP
The code is a bit much, but let's walk through the interactions:
HTTP POST to Web Application
AMQP Send to Worker Application
Worker Application receives message and starts process manager; AMQP Send to itself
Worker Application receives message, makes HTTP POST to Web Application
Web Application makes SQL call
Worker Application receives HTTP reply, makes AMQP Reply
Worker Application process manager receives reply, makes AMQP Send to Child Application
Child Application receives message, makes MongoDB call and makes AMQP Reply to Worker Application
Worker Application process manager receives reply, marks process as complete and makes AMQP Reply to process manager initiator
Web Application receives message and conversation completes
WHEW. Now this conversation does resemble ones I've built, but it is a bit contrived as I don't usually have these back-and-forth communications. But it does show the different kinds of communications in a complex conversation.
If something were to go awry in this conversation, we wouldn't have much recourse to fix anything without distributed tracing.
Exporting Traces with OpenTelemetry
Configuring our applications to export their traces and spans is easy, first we add the appropriate exporter packages:
<PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="0.2.0-alpha.283" />
<PackageReference Include="OpenTelemetry.Exporter.Zipkin" Version="0.2.0-alpha.283" />
Once we have our packages referenced, we can configure them with the rest of our OpenTelemetry setup:
services.AddOpenTelemetry(builder => builder
    .UseZipkin(o =>
    {
        o.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
        o.ServiceName = Program.EndpointName;
    })
    .UseJaeger(c =>
    {
        c.AgentHost = "localhost";
        c.AgentPort = 6831;
        c.ServiceName = Program.EndpointName;
    })
    .AddNServiceBusAdapter()
    .AddRequestAdapter()
    .AddDependencyAdapter(configureSqlAdapterOptions: 
        opt => opt.CaptureTextCommandContent = true));
Each call to an exporter needs some configuration on how to connect to it. Zipkin and Jaeger will need a URI, while Application Insights would need our instrumentation key.
That's it! Once we've configured our exporters, our application will export them seamlessly as it handles requests. Let's look at traces for our orchestration-style communication (and we should see all the steps above), first in Zipkin:
Zipkin trace of orchestration conversation
And next in Jaeger:
Jaeger trace of orchestration conversation
In each of these traces, we can follow the conversation all the way from the initial HTTP POST all the way down to the final AMQP "reply" back to the process manager originator. And because this is an orchestration-style process, the execution times are completely linear - no process hands off to another until it completes.
Without our integration into NServiceBus, we wouldn't have this connected conversation - only disconnected individual traces for each HTTP request. With OpenTelemetry, we've got a complete view of our entire conversation, with each step recorded along the way.
Let's contrast this with a choreography-style conversation to see how that might look.
Choreography Sample
In our orchestration sample, we used commands and replies to dictate a flow for our conversation. With choreography, our communication is more event-oriented, and multiple steps can execute in parallel. Our choreography conversation looks like:
Conversation using choreography with HTTP and AMQP
This conversation is a bit different than the orchestration conversation because we're using events, rather than commands, to coordinate an activity. This allows some processes to execute in parallel. The worker application and child application now handle the published event simultaneously, and publish an event when their step completes.
Finally, our process manager subscribes to the "complete" event from both processes, and when both events are received, the process manager publishes its final event back to the Web Application.
We can see that there are fewer messages in this approach, as each application uses an event to fire, as opposed to the two messages involved in request/response. With our choreography application set up, let's look to see how this conversation manifests in Zipkin:
Zipkin trace of choreography conversation
And in Jaeger:
Jaeger trace for choreography conversation
In each of these traces, we see the interaction kick off with the Web Application publishing an event. Next, both the Child Worker Service and Worker Service execute in parallel, making requests and doing work. We also see the Child Worker Service publishing an event to the Worker Service process manager, but it is not until the second event that the process manager receives that the final publish and receive from the Web Application happens.
Looking at each of these traces, we can see how the conversation patterns differ in their communication style and processing.
With OpenTelemetry, we've made it simple to add end-to-end distributed tracing to our system. None of our application code is aware of OpenTelemetry even existing, and with the upcoming changes to the Activity API, adding distributed tracing will be made even easier.
In the next (and last) post, I'll share some future direction of OpenTelemetry, as well as links to the sample extensions made into released NuGet packages.

  



Building End-to-End Diagnostics: Activity and Span Correlation
Wed, 13 May 2020 14:23:28 +0000
Posts in this series:
An Intro
Trace Context
Diagnostic Events
OpenTelemetry Integration
Activity and Span Correlation
Source Code
In the last post, we looked at hooking up our diagnostics events (and Activities) to OpenTelemetry, where our main task was creating the appropriate span attributes based on the OpenTelemetry Span conventions. When everything runs, we get lovely correlated events based on our W3C tracecontext headers flowing through, as our Activity maintains a link to its parent, regardless of where that parent activity came from.
The Activity to Span connection works great, as long as we can maintain a link between the OTel "Span" and the Activity Start/Stop events themselves, and we're effectively able to start and stop the right activity when something interesting happens in our observed code.
Sometimes, it's not so easy to correlate the "start" and "stop" events for a given observed event. Consider our NServiceBus consumer activity logic:
public override async Task Invoke(
    IIncomingPhysicalMessageContext context,
    Func<Task> next)
{
    var activity = StartActivity(context);

    try
    {
        await next().ConfigureAwait(false);
    }
    finally
    {
        StopActivity(activity, context);
    }
}
For the Start and Stop activities, we have the reference to the activity we started in that activity variable. We can do this because the infrastructure hook for consuming a message wraps the entire operation, and we're passed a continuation (that next parameter). But what happens if the infrastructure hooks we have for "start" and "stop" are physically separate? How do we know which Activity to stop once the observed event completes?
I ran into this exact situation with the MongoDB .NET Driver, which does have a mechanism to raise internal diagnostic events, but your handling of these events is physically separate in your code.
Correlating Events
The main interface to listen to internal diagnostic events in the MongoDB .NET Driver is the IEventSubscriber interface:
public interface IEventSubscriber
{
    bool TryGetEventHandler<TEvent>(out Action<TEvent> handler);
}
We can register subscribers, but we only get this one method that gets called based on different TEvent types. That means that we don't get a single method for opening a connection or issuing a command. Those are separate calls with separate callbacks.
First, let's get some boilerplate to handle these events. There's a helper class, ReflectionEventSubscriber, that lets us define methods with a signature that accepts the event objects and the subscriber uses reflection to find these events. The events we're looking to handle are CommandStartedEvent, CommandSucceededEvent, and CommandFailedEvent. Our event subscriber implementation can wrap this reflection-based event subscriber:
public class DiagnosticsActivityEventSubscriber : IEventSubscriber
{
    private readonly ReflectionEventSubscriber _subscriber;

    public DiagnosticsActivityEventSubscriber() =>
        _subscriber = new ReflectionEventSubscriber(this, 
            bindingFlags: BindingFlags.Instance | BindingFlags.NonPublic);

    public bool TryGetEventHandler<TEvent>(out Action<TEvent> handler) 
        => _subscriber.TryGetEventHandler(out handler);

    private void Handle(CommandStartedEvent @event)
    {
        // Start activity
    }

    private void Handle(CommandSucceededEvent @event)
    {
        // Stop activity
    }

    private void Handle(CommandFailedEvent @event)
    {
        // Stop activity
    }
}
As we can see above, the methods to handle the start/stop activities are physically separate, but we need to share our started Activity between these! We can store the started activity in a field, but when the stop events come in, we need to make sure we stop the right activity.
To find the "right" activity, we need some way to correlate the information in the CommandStartedEvent object and the CommandSucceededEvent/ CommandFailedEvent objects. Luckily, the Mongo drivers for other languages are quite similar to .NET, and some of those other drivers have OpenTelemetry implementations. From that code, we can see that these events have a RequestId that uniquely identifies this request against the MongoDB server. This could work!
Implementing our Event Subscriber
First, we need the plumbing of our diagnostic source in our DiagnosticsActivityEventSubscriber class to raise the diagnostic events:
public class DiagnosticsActivityEventSubscriber : IEventSubscriber
{
    public const string ActivityName = "MongoDB.Driver.Core.Events.Command";

    private static readonly DiagnosticSource _diagnosticListener
        = new DiagnosticListener(ActivityName);
Next, we somewhere to place our started Activity objects, correlated by the RequestId property. The most straightforward solution would be a ConcurrentDictionary for this value:
private readonly ConcurrentDictionary<int, Activity> _activityMap
    = new ConcurrentDictionary<int, Activity>();
Now, when we start our activity, we follow the normal steps we did in our NServiceBus example, with the added step of adding the started activity to our dictionary:
private void Handle(CommandStartedEvent @event)
{
    var activity = new Activity(ActivityName);

    if (_diagnosticListener.IsEnabled(CommandStarted.EventName, @event))
    {
        _diagnosticListener.StartActivity(activity, new CommandStarted(@event));
    }
    else
    {
        activity.Start();
    }

    _activityMap.TryAdd(@event.RequestId, activity);
}
With the event started and stored locally, we can stop our activity based on pulling the activity back out of the local dictionary:
private void Handle(CommandSucceededEvent @event)
{
    if (_activityMap.TryRemove(@event.RequestId, out var activity))
    {
        _diagnosticListener.StopActivity(activity, new CommandSucceeded(@event));
    }
}

private void Handle(CommandFailedEvent @event)
{
    if (_activityMap.TryRemove(@event.RequestId, out var activity))
    {
        _diagnosticListener.StopActivity(activity, new CommandFailed(@event));
    }
}
Initially, I tried to use Activity.Current to pull out the activity, but that won't always be correct if the async context hasn't flowed through to my event handler (it did not). Instead, with this correlated dictionary, I can ensure I stop the correct activities and my eventual OTel spans will correlate as well.
OpenTelemetry Integration
On the OTel side, we have a similar issue where the "Current Span" assumes a common execution context. But since context isn't shared, we have to implement our ListenerHandler in a similar fashion, but this time, correlate a TelemetrySpan:
internal class CommandListener : ListenerHandler
{
    public CommandListener(string sourceName, Tracer tracer) 
        : base(sourceName, tracer)
    {
    }

    private readonly ConcurrentDictionary<int, TelemetrySpan> _spanMap
        = new ConcurrentDictionary<int, TelemetrySpan>();
Now when we receive the OnStartActivity call, we have to perform a similar operation to store our TelemetrySpan based on that OperationId:
public override void OnStartActivity(Activity activity, object payload)
{
    if (!(payload is CommandStarted message))
    {
        AdapterEventSource.Log.NullPayload("CommandListener.OnStartActivity");
        return;
    }

    Tracer.StartActiveSpanFromActivity($"mongodb.{message.Event.CommandName}",
        activity, 
        SpanKind.Client, 
        out var span);

    SetSpanAttributes(span, message);

    _spanMap.TryAdd(message.Event.RequestId, span);
}
From here, stopping the activity will mean, instead of us accessing the CurrentSpan, pulling it out of our dictionary:
public override void OnStopActivity(Activity activity, object payload)
{
    if (!(payload is CommandSucceeded message))
    {
        AdapterEventSource.Log.NullPayload("CommandListener.OnStopActivity");
        return;
    }

    if (_spanMap.TryRemove(message.Event.RequestId, out var span))
    {
        span.End();
    }
}

public override void OnException(Activity activity, object payload)
{
    if (!(payload is CommandFailed message))
    {
        AdapterEventSource.Log.NullPayload("CommandListener.OnExceptionActivity");
        return;
    }

    if (_spanMap.TryRemove(message.Event.RequestId, out var span))
    {
        span.Status = Status.Unknown.WithDescription(message.Event.Failure.Message);
        SetSpanAttributes(span, message);
        span.End();
    }
}
Although it took a bit more work to store the Activity and TelemetrySpan locally, doing so ensured that we correlated the correct instances for eventual publishing to our tracing adapters. If we only went with the Current properties, we'd be stopping the wrong activities and spans, resulting in a very wonky looking request graph.
The last piece is registering my subscriber with MongoDB, but it's quite specific to that driver so you can check out the source code to see how it's all registered and hooked up.
In the next (and last) post, I'll walk through my sample application and see how these adapters will be hooked up and used to visualize complicated traces in Zipkin, Jaeger, and Application Insights.

  



Getting started with Cassandra: Data modeling in the brief
Wed, 05 Feb 2020 20:23:00 +0000
Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.

Introduction

Cassandra distributes data around the cluster via the partition key.

CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));


In the above table the partition key is postal_code and the clustering column isid. The partition key will locate the data on the cluster for us. The clustering column allows us multiple rows per partition key so that we can filter how much data we read per partition.
The ‘optimal’ query is one that retrieves data from only one node and not so much data that GC pressure or latency issues result. The following query is breaking that rule and retrieving 2 partitions at once via the IN parameter.

SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code IN ('77002', '77043');


This can be slower than doing two separate queries asynchronously, especially if those partitions are on two different nodes (imagine if there are 1000+ partitions in the IN statement). In summary, the simple rule to stick to is “1 partition per query”.

Partition sizes

A common mistake when data modeling is to jam as much data as possible into a single partition.


  This doesn’t distribute the data well and therefore misses the point of a distributed database.
  There are practical limits on the performance of partition sizes


Table per query pattern

A common approach to optimize around partition lookup is to create a table per query, and write to all of them on update. The following example has two related tables both to solve two different queries

--query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';
--query by id
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;


You can update both tables at once with a logged batch:

BEGIN BATCH
INSERT INTO my_key.my_table (id, name, address, city, state, postal_code, country, balance) VALUES (7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 'Bordeaux', 'Gironde', '33000', 'France', 56.20);
INSERT INTO my_key.my_table_by_postal_code (postal_code, id, balance) VALUES ('33000', 7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 56.20) ;
APPLY BATCH;


Source of truth

A common design pattern is to have one table act as the authoritative one over data, and if for some reason there is a mismatch or conflict in other tables as long as there is one considered “the source of truth” it makes it easy to fix any conflicts later. This is typically the table that would match what we see in typical relational databases and has all the data needed to generate all related views or indexes for different query methods. Taking the prior example, my_table is the source of truth:

--source of truth table
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;

--based on my_key.my_table and so we can query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';


Next we discuss strategies for keeping tables of related in sync.

Materialized views

Materialized views are a feature that ships with Cassandra but is currently considered rather experimental. If you want to use them anyway:

CREATE MATERIALIZED VIEW my_key.my_table_by_postal_code 
AS SELECT postal_code text, id uuid, balance float
FROM my_key.my_table 
WHERE postal_code IS NOT NULL AND id IS NOT NULL 
PRIMARY KEY(postal_code, id));


Materialized views at least run faster than the comparable BATCH insert pattern, but they have a number of bugs and known issues that are still pending fixes.

Secondary indexes

This are the original server side approach to handling different query patterns but it has a large number of downsides:


  rows are read serially one node at time until limit is reached.
  a suboptimal storage layout leading to very large partitions if the data distribution of the secondary index is not ideal.


For just those two reasons I think it’s rare that one can use secondary indexes and expect reasonable performance. However, you can make one by hand and just query that data asynchronously to avoid some of the downsides.

CREATE TABLE my_key.my_table_by_postal_code_2i (postal_code text, id uuid, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code_2i WHERE postal_code = '77002';
--retrieve all rows then asynchronously query the resulting ids
SELECT * FROM my_key.my_table WHERE id = ad004ff2-e5cb-4245-94b8-d6acbc22920a;
SELECT * FROM my_key.my_table WHERE id = d30e9c65-17a1-44da-bae0-b7bb742eefd6;
SELECT * FROM my_key.my_table WHERE id = e016ae43-3d4e-4093-b745-8583627eb1fe;


Exercises

Contact List

This is a good basic first use case as one needs to use multiple tables for the same data, but there should not be too many.

requirements


  contacts should have first name, last name, address, state/region, country, postal code
  lookup by contacts id
  retrieve all contacts by a given last name
  retrieve counts by zip code


Music Service

Takes the basics from the previous exercise and requires a more involved understanding of the concepts. It will require many tables and some difficult trade-offs on partition sizing. There is no one correct way to do this.

requirements


  songs should have album, artist, name, and total likes
  The contact list exercise, can be used as a basis for the “users”, users will have no login because we’re trusting people
  retrieve all songs by artist
  retrieve all songs in an album
  retrieve individual song and how many times it’s been liked
  retrieve all liked songs for a given user
  “like” a song
  keep a count of how many times a song has been listened to by all users


IoT Analytics

This will require some extensive time series modeling and takes some of the lessons from the Music Service further. The table(s) used will be informed by the query.

requirements


  use the music service data model as a basis, we will be tracking each “registered device” that uses the music service
  a given user will have 1-5 devices
  log all songs listened to by a given device
  retrieve songs listened for a device by day
  retrieve songs listened for a device by month
  retrieve total listen time for a device by day
  retrieve total listen time for a device by month
  retrieve artists listened for a device by day
  retrieve artists listened for a device by month



Getting started with Cassandra: Load testing Cassandra in brief
Tue, 04 Feb 2020 20:23:00 +0000
An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra.

If you have no data model in mind

Use cassandra stress since it’s around:


  first initialize the keyspace with RF3 cassandra-stress "write cl=ONE no-warmup -col size=FIXED(15000) -schema replication(strategy=SimpleStrategy,factor=3)"
  second run stress cassandra-stress "mixed n=1000k cl=ONE -col size=FIXED(15000)
  repeat as often as you’d like with as many clients as you want.


If you have a specific data model in mind

You can use cassandra-stress, but I suspect you’re going to find your data model isn’t supported (collections for example) or that you don’t have the required PHD to make it work the way you want. There are probably 2 dozen options from here you can use to build your load test, some of the more popular ones are gatling, jmeter, and tlp-stress. My personal favorite for this though, write a small simple python or java program that replicates your use case accurately in your own code, using a faker library to generate your data. This takes more time but you tend to have less surprises in production as it will accurately model your code.

Small python script with python driver


  use python3 and virtualenv
  python -m venv venv
  source venv/bin/activate
  read and follow install docs
  if you want to skip the docs you can get away with pip install cassandra-driver
  install a faker library pip install Faker


import argparse
import uuid
import time
import random
from cassandra.cluster import Cluster
from cassandra.query import BatchStatement
from faker import Faker

parser = argparse.ArgumentParser(description='simple load generator for cassandra')
parser.add_argument('--hosts', default='127.0.0.1',
                    type=str,
                    help='comma separated list of hosts to use for contact points')
parser.add_argument('--port', default=9042, type=int, help='port to connect to')
parser.add_argument('--trans', default=1000000, type=int, help='number of transactions') 
parser.add_argument('--inflight', default=25, type=int, help='number of operations in flight') 
parser.add_argument('--errors', default=-1, type=int, help='number of errors before stopping. default is unlimited') 
args = parser.parse_args()
fake = Faker(['en-US'])
hosts = args.hosts.split(",")
cluster = Cluster(hosts, port=args.port)

try:
    session = cluster.connect()
    print("setup schema");
    session.execute("CREATE KEYSPACE IF NOT EXISTS my_key WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1}")
    session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table (id uuid, name text, address text, state text, zip text, balance int, PRIMARY KEY(id))")
    session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table_by_zip (zip text, id uuid, balance bigint, PRIMARY KEY(zip, id))")
    print("allow schema to replicate throughout the cluster for 30 seconds")
    time.sleep(30)
    print("prepare queries")
    insert = session.prepare("INSERT INTO my_key.my_table (id, name, address, state, zip, balance) VALUES (?, ?, ?, ?, ?, ?)")
    insert_rollup = session.prepare("INSERT INTO my_key.my_table_by_zip (zip, id, balance) VALUES (?, ?, ?)")
    row_lookup = session.prepare("SELECT * FROM my_key.my_table WHERE id = ?")
    rollup = session.prepare("SELECT sum(balance) FROM my_key.my_table_by_zip WHERE zip = ?")
    threads = []
    ids = []
    error_counter = 0
    query = None
    params = []
    ids = []
    
    def get_id():
        items = len(ids)
        if items == 0:
            ## nothing present so return something random
            return uuid.uuid4()
        if items == 1:
            return ids[0]
        return ids[random.randint(0, items -1)]
    print("starting transactions")
    for i in range(args.trans):
        chance = random.randint(1, 100)
        if chance > 0 and chance < 50:
            new_id = uuid.uuid4()
            ids.append(new_id)
            state = fake.state_abbr()
            zip_code = fake.zipcode_in_state(state)
            balance = random.randint(1, 50000)
            query = BatchStatement()
            name = fake.name()
            address = fake.address()
            bound_insert = insert.bind([new_id, fake.name(), fake.address(), state, zip_code, balance])
            query.add(bound_insert)
            bound_insert_rollup = insert_rollup.bind([zip_code, new_id, balance])
            query.add(bound_insert_rollup)
        elif chance > 50 and chance < 75:
            query = row_lookup.bind([get_id()])
        elif chance > 75:
            zip_code = fake.zipcode()
            query = rollup.bind([zip_code])
        threads.append(session.execute_async(query))
        if i % args.inflight == 0:
            for t in threads:
                try:
                    t.result() #we don't care about result so toss it
                except Exception as e:
                    print("unexpected exception %s" % e)
                    if args.errors > 0:
                        error_counter = error_counter + 1
                        if error_counter > args.errors:
                            print("too many errors stopping. Consider raising --errors flag if this happens more quickly than you'd like")
                            break
            threads = []
            print("submitted %i of %i transactions" % (i, args.trans))
finally:
    cluster.shutdown()


Small java program with latest java driver


  download java 8
  create a command line application in your project technology of choice (I used maven in this example for no particularly good reason)
  download a faker lib like this one and the Cassandra java driver from DataStax again using your preferred technology to do so.
  run the following code sample somewhere (set your RF and your desired queries and data model)
  use different numbers of clients at your cluster until you get enough “saturation” or the server stops responding.


See complete example
package pro.foundev;

import java.lang.RuntimeException;
import java.lang.Thread;
import java.util.Locale;
import java.util.ArrayList;
import java.util.List;
import java.util.function.*;
import java.util.Random;
import java.util.UUID;
import java.util.concurrent.CompletionStage;
import java.net.InetSocketAddress;
import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.CqlSessionBuilder;
import com.datastax.oss.driver.api.core.cql.*;
import com.github.javafaker.Faker;

public class App 
{
    public static void main( String[] args )
    <


Getting started with Cassandra: Setting up a Multi-DC environment
Mon, 03 Feb 2020 20:23:00 +0000
This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.

A new cluster


  In cassandra.yaml set endpoint_snitch: GossipingPropertyFileSnitch, some prefer PropertyFileSnitch for the ease of pushing out one file. GossipingPropertyFileSnitch is harder to get wrong in my experience.
  set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
  finish adding all of your nodes.
  if using authentication,  set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
  nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
  create your new keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above).
  nodetool repair -pr whatever_new_keyspace on each node in the cluster on the new keyspace.


An existing cluster

This is harder and involves more work and more options, but I’m going to discuss the way that gets you into the least amount of trouble operationally.


  make sure none of the drivers you use to connect to cassnadra are using DowngradingConsistencyRetryPolicy, or using the maligned withUsedHostsPerRemoteDc, especially allowRemoteDCsForLocalConsistencyLevel, as this may cause your driver to send requests to the remote data center before it’s populated with data.
  switch endpoint_snitch on each node to GossipingPropertyFileSnitch
  set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
  bootstrap each node in the new data center.
  if using authentication,  set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
  nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
  alter your app keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above),
  nodetool repair -pr whatever_keyspace on each node in the cluster on the new keyspace.


enjoy new data center

how to get data to new dc

Repair approach

Best done with if your repair jobs can’t be missed or stopped, either because you have a process like opscenter or repear running repairs. It also has the advantage of being very easy, and if you’ve already automated repair you’re basically done.


  let repair jobs continue..that’s it!


Rebuild approach

Faster less resource intensive, and if you have enough time to complete it while repair is stopped. Rebuild is easier to ‘resume’ than repair in many ways, so this has a number of advantages.


  run nodetool rebuild on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.
  run nodetool cleanup


YOLO rebuild with repair

This will probably overstream it’s share of data and honestly a lot of folks do this for some reason in practice:


  leave repair jobs running
  run nodetool rebuild on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.
  run nodetool cleanup on each node


Cloud strategies

There are a few valid approaches to this and none of them are wrong IMO.

region == DC, rack == AZ

Will need to get into racks and a lot of people get this wrong and imbalance the racks, but you get the advantage of more intelligent failure modes, with racks mapping to AZs.

AZ..regardless of region == DC

This allows things to be balanced easily, but you have no good option for racks then. However, some people think racks are overated, and I’d say a majority of clusters run with one rack.


Gocode Vim Plugin and Go Modules
Sat, 05 Jan 2019 17:09:26 +0000
I recently purchased Let’s Go from Alex Edwards.  I wanted an end-to-end Golang website tutorial.  It has been great.  Lots learned.

Unfortunately, he is using Go’s modules and the version of the gocode Vim plugin I was using did not support Go modules.

Solution:
Use this fork of the gocode Vim plugin and you’ll get support for Go modules.

I use Vim Plug for my Vim plugins.  Huge fan of Vundle but I like the post-actions feature of Plug.  I just had to change one line of my vimrc and re-run updates.

diff --git a/vimrc b/vimrc
index 3e8edf1..8395705 100644
--- a/vimrc
+++ b/vimrc
@@ -73,7 +73,7 @@ endif
           let editor_name='nvim'
           Plug 'zchee/deoplete-go', { 'do': 'make'}
         endif
-        Plug 'nsf/gocode', { 'rtp': 'vim', 'do': '~/.config/nvim/plugged/gocode/vim/symlink.sh' }
+        Plug 'stamblerre/gocode', { 'rtp': 'vim', 'do': '~/.vim/plugged/gocode/vim/symlink.sh' }
         Plug 'godoctor/godoctor.vim', {'for': 'go'} " Gocode refactoring tool
     " }


That is the line I had to change then run :PlugUpdate! and the new plugin was installed.

I figured all of this out due to this comment by Tommaso Sardelli on Github.  Thank you Tommaso.


Raspberry Pi Kubernetes Cluster - Part 4
Fri, 28 Dec 2018 16:35:23 +0000

    Raspberry Pi Kubenetes Cluster - Part 1

Raspberry Pi Kubenetes Cluster - Part 2

Raspberry Pi Kubenetes Cluster - Part 3

Raspberry Pi Kubenetes Cluster - Part 4

Howdy again.

In this post I’m going to show you how to create a docker image to run on ARM architecture and also how to deploy it and view it.

To start please view my basic flask application called fl8 here

If you’d like to clone and use it:

git clone git@github.com:meridth/fl8.git && cd fl8


ARM docker image
First we need to learn about QEMU

What is QEMU and QEMU installation

QEMU (Quick EMUlator) is an Open-Source hosted hypervisor, i.e. an hypervisor running on a OS just as other computer programs, which performs hardware virtualization. QEMU emulates CPUs of several architectures, e.g. x86, PPC, ARM and SPARC. It allows the execution of non-native target executables emulating the native execution and, as we require in this case, the cross-building process.

Base Docker image that includes QEMU

Please open the Dockerfile.arm and notice the first line: FROM hypriot/rpi-alpine.  This is a base image that includes the target qemu statically linked executable, qemu-arm-static in this case.  I chose hypriot/rpi-alpine because the alpine base images are much smaller than other base images.

Register QEMU in the build agent

To add QEMU in the build agent there is a specific Docker Image performing what we need, so just run in your command line:

docker run --rm --privileged multiarch/qemu-user-static:register --reset


Build image

docker build -f ./Dockerfile.arm -t meridth/rpi-fl8 .


And voila!  You now have an image that will run on Raspberry Pis.

Deployment and Service

/.run-rpi.sh is my script where I run a Kubernetes deployment with 3 replicas and a Kubernetes service.  Please read fl8-rpi-deployment.yml and fl8-rpi-service.yml.  They are only different from the other deployment and service files by labels.  Labels are key/vaule pairs that can be used by selectors later.

The deployment will pull my image from meridth/rpi-fl8 on dockerhub.  If you have uploaded your docker image somewhere you can change the deployment file to pull that image instead.

Viewing application

kubectl get pods


Choose a pod to create the port forwarding ssh tunnel.

kubectl port-forward [pod-name] [app-port]:[app-port]


Example:  kubectl port-forward rpi-fl8-5d84dd8ff6-d9tgz 5010:5010

The final result when you go to http://localhost:5010 in a browser.



Hope this helps someone else.  Cheers.

    Raspberry Pi Kubernetes Cluster - Part 4 was originally published by Jason Meridth at Jason Meridth on December 28, 2018.


Raspberry Pi Kubernetes Cluster - Part 3
Mon, 24 Dec 2018 21:59:23 +0000

    Raspberry Pi Kubenetes Cluster - Part 1

Raspberry Pi Kubenetes Cluster - Part 2

Raspberry Pi Kubenetes Cluster - Part 3

Raspberry Pi Kubenetes Cluster - Part 4

Well, it took me long enough to follow up on my previous posts.  There are reasons.


  The day job has been fun and busy
  Family life has been fun and busy
  I kept hitting annoying errors when trying to get my cluster up and running


The first two reasons are the usual reasons a person doesn’t blog. :)

The last one is what prevented me from blogging sooner.  I had mutliple issues when trying to use rak8s to setup my cluster.  I’m a big fan of Ansible and I do not like running scripts over and over.  I did read K8S on Raspbian Lite from top to bottom and realized automation would make this much better.



The issues I experienced:

apt-get update would not work

I started with the vanilla Raspbian lite image to run on my nodes and had MANY MANY issues with running apt-get update and apt-get upgrade.  The mirrors would disconnect often and just stall.  This doesn’t help my attempted usage of rak8s which does both on the cluster.yml run (which I’ll talk about later).

rak8s changes needed to run HypriotOS and kubernetes 1.13.1

Clone the repo locally and I’ll walk you through what I changed to get rak8s working for me and HypriotOS.

Change the following files:


  ansible.cfg
    
      change user from pi to pirate
    
  
  roles/kubeadm/tasks/main.yml
    
      add ignore_errors: True to Disable Swap task
      I have an open PR for this here
    
  
  group_vars/all.yml
    
      Change kubernetes_package_version to "1.13.1-00"
      Change kubernetes_version to "v1.13.1"
    
  


After you make those changes you can run ansible-playbook cluster.yml as the rak8s documentation suggests.  Please note this is after you edit inventory and copy ssh keys to raspberry pis.

Flannel networking issue once nodes are up

After I get all of the nodes up I noticed the master node was marked ast NotReady and when I ran kubectl describe node raks8000 I saw the following error:


  KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized


This error is known in kubernetes > 1.12 and flannel v0.10.0.  It is mentioned in this issue.  The fix is specifically mentioned here.  It is to run the following command

kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

After readin the issue it seems the fix will be in the next version of flannel and will be backported to v0.10.0.

A running cluster


    Raspberry Pi Kubernetes Cluster - Part 3 was originally published by Jason Meridth at Jason Meridth on December 24, 2018.


MVP how minimal
Thu, 20 Dec 2018 20:00:00 +0000
MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas.
In recent history I’ve seen up close two extreme constrasting examples of MVP:


  Mega Minimal: website and db, mostly manual on the backend
  Mega Mega: provisioning system, dynamic tuning of systems via ML, automated operations, monitoring a few others I’m leaving out.


Feedback

If we’re evaluating which approach gives us more feedback, Mega Minimal MVP is gonna win hands down here. Some will
counter they don’t want to give people a bad impression with a limited product and that’s fair, but it’s better than no impression (the dreaded never shipped MVP). The Mega Mega
MVP I referenced took months to demo. only had one of those checkboxes setup and wasn’t ever demod again. So we can categorical say that failed 
at getting any feedback.

Whereas the Mega Minimal MVP, got enough feedback and users for the founders to realize that wasn’t a business for them.
Better than after hiring a huge team and sinking a million plus into dev efforts for sure. Not the happy ending I’m sure you all were expecting, but I view that as mission accomplished.

Core Value


  Mega Minimal, they only focused on a single feature, executed well enough that people gave them some positive feedback, but not enough to justify automating everything.
  Mega Mega. I’m not sure anyone who talked about the product saw the same core value, and there were several rewrites and shifts along the way.


Advantage Mega Minimal again

What about entrants into a crowded field

Well that is harder and the MVP tends to be less minimal, because the baseline expectations are just much higher. I still lean towards Mega Minimal having a better chance at getting users, since there is a non zero chance the Mega Mega MVP will never get finished.
I still think the exercise in focusing on core value that makes your product not a me too, and even considering how you can find a niche in a crowded field
instead of just being “better”, and your MVP can be that niche differentiator.

Internal users

Sometimes a good middle ground is considering getting lots of internal users if you’re really worried about bad experiences. This has it’s
it’s definite downsides however, and you may not get diverse enough opinions. But it does give you some feedback while saving some face or bad experiences. I often think of the example of EC2 that was heavily used by Amazon, before being released to the world.
That was a luxury Amazon had, where their customer base and their user base happened to be very similar, and they had bigger scale needs than any of their early customers, so the early internal feedback loop was a very strong signal.

Summary

In the end however you want to approach MVPs is up to you, and if you find success with a meatier MVP than I have please don’t let me push you away from what works. But if you are having trouble shipping and are getting pushed all the time to add one more feature to that MVP before releasing it, consider stepping back and asking is this really core value 
for the product? Do you already have your core value? if so, consider just releasing it.


Surprise Go is ok for me now
Thu, 13 Dec 2018 20:23:00 +0000
I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve.

About 7 months ago I wrote about all the things I didn’t really care for in Go and now I either no longer am so bothered by it or things have improved.

Go Modules so far is a huge improvement over Dep and Glide for dependency management. It’s easy to setup, performant and eliminates the GOPATH silliness. I haven’t tried it yet with some of the goofier libraries that gave me problems in the past (k8s api for example) so the jury is out on that, but again pretty impressed. I now longer have to check in vendor to speed up builds. Lesson use Go Modules.

I pretty much stopped using channels for everything but shutdown signals and that fits my preferences pretty well, I use mutex and semaphores for my multithreaded code and feel no guilt about it. This cut out a lot of pain for me, and with the excellent race detector I feel really comfortable writing multi-threaded in Go now. Lesson, don’t use channels much.

Lack of generics still sometimes sucks but I usually implement some crappy casting with dynamic types if I need that. I’ve sorta made my piece with just writing more code, and am no longer so hung up. Lesson relax.

Error handling I’m still struggling with, I thought about using one of the error Wrap() libraries but an official one is in draft spec now, so I’ll wait on that. I now tend to have less nesting of functions as a result, this probably means longer functions than I like, but my code looks more “normal” now. This is a trade off I’m ok with. Lesson relax more.

I see the main virtue of Go now that it is very popular in the infrastructure space where I am and so it’s becoming the common tongue (largely replacing Python for those sorts of tasks). For this, honestly it’s about right. It’s easy to rip out command line tools and deploy binaries for every platform with no runtime install.

The community’s conservative attitude I sort of view as a feature now, in that there isn’t a bunch of different options that are popular and there is no arguing over what file format is used. This drove me up the wall initially, but I appreciate how much less time I spend on these things now.

So now I suspect Go will be my “last” programming language. It’s not the one I would have chosen, but where I am at in my career, where most of my dev work is automation and tooling it fits the bill pretty well.

Also equally important most of the people working with me didn’t have full time careers as developers or spend their time reading “Domain Driven Design” (amazing book) so adding in a bunched of nuanced stuff that maybe technically optimal but assumes the reader grasps all that nuance isn’t a good tradeoff for me.

So I think I sorta get it now. I’ll never be a cheerleader for the language but it definitely solves my problems well enough.


Collaboration vs. Critique
Fri, 18 May 2018 17:00:00 +0000
While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams.  The people aspect of software development, more often than not, tends to be the most difficult part of software engineering.  Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs.  The nature of software development tends to follow an empirical process model rather than a defined process model.  That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset.  Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation.  This means that team members are often required to work together to determine how to proceed.  Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation.  One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique.

To help illustrate what I’ve seen happen countless times both in catch-up design sessions and code reviews, consider the following two scenarios:

Scenario 1
Tom and Sally are both developers on a team maintaining a large-scale application.  Tom takes the next task in the development queue which happens to have some complex processes that will need to be addressed.  Being the good development team that they are, both Tom and Sally are aware of the requirements of the application (i.e. how the app needs to work from the user’s perspective), but they have deferred design-level discussions until the time of implementation.  After Tom gets into the process a little, seeing that the problem is non-trivial, he pings Sally to help him brainstorm different approaches to solving the problem.  Tom and Sally have been working together for over a year and have become accustomed to these sort of ad-hoc design sessions.  As they begin discussing the problem, they each start tossing ideas out on the proverbial table resulting in multiple approaches to compare and contrast.  The nature of the discussion is such that neither Tom nor Sally are embarrassed or offended when the other points out flaws in the design because there’s a sense of safety in their mutual understanding that this is a brainstorming session and that neither have thought in depth about the solutions being set froth yet.  Tom throws out a couple of ideas, but ends up shooting them down himself as he uses Sally as a sounding board for the ideas.  Sally does the same, but toward the end of the conversation suggests a slight alteration to one of Tom’s initial suggestions that they think may make it work after all.  They end the session both with a sense that they’ve worked together to arrive at the best solution.

Scenario 2
Bill and Jake are developers on another team.  They tend to work in a more siloed fashion, but they do rely upon one another for help from time to time and they are required to do code reviews prior to their code being merged into the main branch of development.  Bill takes the next task in the development queue and spends the better part of an afternoon working out a solution with a basic working skeleton of the direction he’s going.  The next day he decides that it might be good to have Jake take a look at the design to make him aware of the direction.  Seeing where Bill’s design misses a few opportunities to make the implementation more adaptable to changes in the future, Jake points out where he would have done things differently.  Bill acknowledges that Jake’s suggestions would be better and would have probably been just as easy to implement from the beginning, but inwardly he’s a bit disappointed that Jake didn’t like his design as-is and that he has to do some rework.  In the end, Bill is left with a feeling of critique rather than collaboration.

Whether it’s a high-level UML diagram or working code, how one person tends to perceive feedback on the ideas comprising a potential solution has everything to do with timing. It can be the exact same feedback they would have received either way, but when the feedback occurs often makes a difference between whether it’s perceived as collaboration or critique.  It’s all about when the conversation happens.


Testing Button Click in React with Jest
Mon, 07 May 2018 17:01:59 +0000
When building React applications you will most likely find yourself using Jest as your testing framework.  Jest has some really, really cool features built in.  But when you use Enzyme you can take your testing to the nest level.
One really cool feature is the ability to test click events via Enzyme to ensure your code responds as expected.
Before we get started you are going to want to make sure you have Jest and Enzyme installed in your application.

Installing Enzyme
Installing Jest

Sample code under test

What I would like to be able to do is pull the button out of my component and test the onClick event handler.

  
// Make sure you have your imports setup correctly
import React from 'react';
import { shallow } from 'enzyme';


it('When active link clicked, will push correct filter message', () => {
    let passedFilterType = '';
    const handleOnTotalsFilter = (filterType) => {
        passedFilterType = filterType;
    };
    const accounts = {};
    const wrapper = shallow(<MyComponent accounts={accounts} filterHeader="" onTotalsFilter={handleOnTotalsFilter} />);

    const button = wrapper.find('#archived-button');
    button.simulate('click');
    expect(passedFilterType).toBe(TotalsFilterType.archived);
});


Lets take a look at the test above

First we are going to create a callback (click handler) to catch the bubbled up values.
We use Enzyme to create our component MyComponent
We use the .find() on our wrapped component to find our <Button /> by id
After we get our button we can call .simulate(‘click’) which will act as a user clicking the button.
We can assert that the expected value bubbles up.

As you can see, simulating a click event of a rendered component is very straight forward, yet very powerful.
Till next time,


Lessons from a year of Golang
Mon, 07 May 2018 13:16:00 +0000
I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive.

This isn’t meant to evaluate if you should pick up Go or tell you what you should think of it, this is strictly meant to help people out that are new to the language but experienced in Java, Python, Ruby, C#, etc and have read some basic Go getting started guide.

Dependency management

This is probably the feature most frequently talked about by newcomers to Go and with some justification, as dependency management been a rapidly shifting area that’s nothing like what experienced Java, C#, Ruby or Python developers are used to.

I’ll cut to the chase the default tool now is Dep all other tools I’ve used such as Glide or Godep they’re now deprecated in favor of Dep, and while Dep has advanced rapidly there are some problems you’ll eventually run into (or I did):


  Dep hangs randomly and is slow, which is supposedly network traffic but it happens to everyone I know with tons of bandwidth. Regardless, I’d like an option to supply a timeout and report an error.
  Versions and transitive depency conflicts can be a real breaking issue in Go still. So without shading or it’s equivalent two package depending on different versions of a given package can break your build, there are a number or proposals to fix this but we’re not there yet.
  Dep has some goofy ways it resolves transitive dependencies and you may have to add explicit references to them in your Gopkg.toml file. You can see an example here under Updating dependencies – golang/dep.


My advice


  Avoid hangs by checking in your dependencies directly into your source repository and just using the dependency tool (dep, godep, glide it doesn’t matter) for downloading dependencies.
  Minimize transitive dependencies by keeping stuff small and using patterns like microservices when your dependency tree conflicts.


GOPATH

Something that takes some adjustment is you check out all your source code in one directory with one path (by default ~/go/src ) and include the path to the source tree to where you check out. Example:


  I want to use a package I found on github called jim/awesomeness
  I have to go to ~/go/src and mkdir -p github.com/jim
  cd into that and clone the package.
  When I reference the package in my source file it’ll be literally importing github.com/jim/awesomeness


A better guide to GOPATH and packages is here.

My advice

Don’t fight it, it’s actually not so bad once you embrace it.

Code structure

This is a hot topic and there are a few standards for the right way to structure you code from projects that do “file per class” to giant files with general concept names (think like types.go and net.go). Also if you’re used to using a lot of sub package you’re gonna to have issues with not being able to compile if for example you have two sub packages reference one another.

My Advice

In the end I was reasonably ok with something like the following:


  myproject/bin for generated executables
  myproject/cmd for command line code
  myproject/pkg for code related to the package


Now whatever you do is fine, this was just a common idiom I saw, but it wasn’t remotely all projects. I also had some luck with just jamming everything into the top level of the package and keeping packages small (and making new packages for common code that is used in several places in the code base). If I ever return to using Go for any reason I will probably just jam everything into the top level directory.

Debugging

No debugger! There are some projects attempting to add one but Rob Pike finds them a crutch.

My Advice

Lots of unit tests and print statements.

No generics

Sorta self explanatory and it causes you a lot of pain when you’re used to reaching for these.

My advice

Look at the code generation support which uses pragmas, this is not exactly the same as having generics but if you have some code that has a lot of boiler plate without them this is a valid alternative.  See this official Go Blog post for more details.

If you don’t want to use generation you really only have reflection left as a valid tool, which comes with all of it’s lack of speed and type safety.

Cross compiling

If you have certain features or dependencies you may find you cannot take advantage of one of Go’s better features cross compilation.

I ran into this when using the confluent-go/kafka library which depends on the C librdkafka library. It basically meant I had to do all my development in a Linux VM because almost all our packages relied on this.

My Advice

Avoid C dependencies at all costs.

Error handling

Go error handling is not exception base but return based, and it’s got a lot of common idioms around it:

myValue, err := doThing()
if err != nil {
	return -1, fmt.Errorf(“unable to doThing %v”, err)
  	}


Needless to say this can get very wordy when dealing with deeply nested exceptions or when you’re interacting a lot with external systems.  It is definitely a mind shift if you’re used to the throwing exceptions wherever and have one single place to catch all exceptions where they’re handled appropriately.

My Advice

I’ll be honest I never totally made my peace with this. I had good training from experienced opensource contributors to major Go projects, read all the right blog posts, definitely felt like I’d heard enough from the community on why the current state of Go error handling was great in their opinions, but the lack of stack traces was a deal breaker for me.

On the positive side, I found Dave Cheney’s advice on error handling to be the most practical and he wrote a package containing a lot of that advice, we found it invaluable as it provided those stack traces we all missed but you had to remember to use it.

Summary

A lot of people really love Go and are very productive with it, I just was never one of those people and that’s ok. However, I think the advice in this post is reasonably sound and uncontroversial. So, if you find yourself needing to write some code it Go, give this guide a quick perusal and you’ll waste a lot less time than I did getting productive in developing applications in Go.


Raspberry Pi Kubernetes Cluster - Part 2
Thu, 03 May 2018 02:13:07 +0000

    Howdy again.

Alright, my 8 port switch showed up so I was able to connect my raspberry 3B+ boards to my home network.  I plugged it in with 6 1ft CAT5 cables I had in my catch-all box that all of us nerds have.  I’d highly suggest flexible CAT 6 cables instead if you can get them, like here.   I ordered them and they showed up before I finished this post, so I am using the CAT6 cables.



The IP addresses they will receive initialy from my home router via DHCP can be determined with nmap.  Lets imagine my subnet is 192.168.1.0/24.

Once I got them on the network I did the following:



Install Raspberrian OS On SD Cards

You can get the Raspberry Pi Stretch Lite OS from here.



Then use the Etcher tool to install it to each of the 6 SD cards.



IMPORTANT

Before putting the cards into the Raspberry Pis you need to add a ssh folder to the root of the SD cards.  This will allow you to ssh to each Raspberry Pi with default credentials (username: pi and password raspberry).  Example: ssh pi@raspberry_pi_ip where raspberry_ip is obtained from the nmap command above.

Next post will be setting up kubernetes.  Thank you for reading.

Cheers.

    Raspberry Pi Kubernetes Cluster - Part 2 was originally published by Jason Meridth at Jason Meridth on May 02, 2018.


Multi-Environment Deployments with React
Tue, 10 Apr 2018 12:54:17 +0000
If you are using Create-React-App to scaffold your react application there is built in support for changing environment variables based on the NODE_ENV values, this is done by using .env files.  In short this process works by having a .env, .env.production, .env.development set of files.  When you run/build your application CRA will set the NODE_ENV value to either development or production and based on these values the correct .env file will be used.
This works great, when you have a simple deploy setup. But many times in enterprise level applications you need support for more than just 2 environments, many times it is 3-4 environments.  Common logic would suggest that you can accomplish this via the built in mechanism by having additional .env files and changing the NODE_ENV value to the value you care about.  However, CRA does not support this with doing an eject, which will eject all the default conventions and leave it to you to configure your React application.  Maybe this is a good idea, but in my case ejecting was not something I wanted to do.
Because I did not want to do an eject I needed to find another solution, and after a fair amount of searching I found a solution that seems to work for me and my needs and is about the amount of effort as I wanted Hart Hoover and it finally motivated me to purchase his “grocery list” and do this finally.  I’ve been using Minikube for local Kubernetes testing but it doesn’t give you multi-host testing abilities.  I’ve also been wanting to get deeper into my Raspberry Pi knowledge.  Lots of learning and winning.

The items I bought were:


  Six Raspberry Pi 3 Model B+ Motherboards
  Six SanDisk Ultra 32GB microSDHC UHS-I Card with Adapter, Grey/Red, Standard Packaging (SDSQUNC-032G-GN6MA)
  One Sabrent 6-Pack 22AWG Premium 3ft Micro USB Cables High Speed USB 2.0 A Male to Micro B Sync and Charge Cables Black CB-UM63
  One AmazonBasics 6-Port USB Wall Charger (60-Watt) - Black
  One GeauxRobot Raspberry Pi 3 Model B 6-layer Dog Bone Stack Clear Case Box Enclosure also for Pi 2B B+ A+ B A
  One Black Box 8-Port Switch


Here is the tweet when it all arrived:

I blame @hhoover ;). I will be building my @kubernetesio cluster once the 6pi case shows up next Wednesday. The extra pi is to upgrade my @RetroPieProject. Touch screen is an addition I want to try. Side project here I come. pic.twitter.com/EebIKbsCeH
— Jason Meridth (@jmeridth) March 31, 2018



I spent this morning finally putting it together.

Here is me getting started on the “dogbone case” to hold all of the Raspberry Pis:



The bottom and one layer above:



And the rest:









Different angles completed:





And connect the power:



Next post will be on getting the 6 sandisk cards ready and putting them in and watching the Raspberry Pis boot up and get a green light.  Stay tuned.

Cheers.

    Raspberry Pi Kubernetes Cluster - Part 1 was originally published by Jason Meridth at Jason Meridth on April 07, 2018.


Building AWS Infrastructure with Terraform: S3 Bucket Creation
Fri, 06 Apr 2018 14:28:49 +0000
If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure.  In the AWS word you can use the native CloudFormation solution, but honestly I find this painful and the docs very lacking.  Personally, I prefer Terraform by Hashicorp.  In my experience the simplicity and easy of use, not to mention the stellar documentation make this the product of choice.
This is the initial post in what I hope to be a series of post about how to use Terraform to setup/build AWS Infrastructure.
Terrform Documentation on S3 Creation -> Here
In this post I will cover 2 things

Basic bucket setup
Bucket setup as Static website

Setting up a basic bucket we can use the following

resource "aws_s3_bucket" "my-bucket" {
    bucket = "my-bucket"
    acl    = "private"

    tags {
        Any_Tag_Name = "Tag value for tracking"
    }    
}


When looking at the example above the only 2 values that are required are bucket and acl.
I have added the use of Tags to show you can add custom tags to your bucket
Another way to setup an S3 bucket is to act as a Static Web Host.   Setting this up takes a bit more configuration, but not a ton.

resource "aws_s3_bucket" "my-website-bucket" {
    bucket = "my-website-bucket"
    acl    = "public-read"

    website {
        index_document = "index.html"
        error_document = "index.html"
    }    

    policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-website-bucket/*"
        }
    ]
}
    POLICY
    tags {
          Any_Tag_Name = "Tag value for tracking"
    }
}


The example above has 2 things that need to be pointed out.

The website settings.  Make sure you setup the correct pages here for index/error

The Policy settings.  Here I am using just basic policy.  You can of course setup any policy here you want/need.
As you can see, setting up S3 buckets is very simple and straight forward.
*** Reminder: S3 bucket names MUST be globally unique ***
Till next time,


SSH - Too Many Authentication Failures
Wed, 28 Mar 2018 05:00:00 +0000

    Problem
I started seeing this error recently and had brain farted on why.

Received disconnect from 123.123.132.132: Too many authentication failures for hostname

After a bit of googling it came back to me.  This is because I’ve loaded too many keys into my ssh-agent locally (ssh-add).  Why did you do that?  Well, because it is easier than specifying the IdentityFile on the cli when trying to connect.  But there is a threshhold.  This is set by the ssh host by the MaxAuthTries setting in /etc/ssh/sshd_config.  The default is 6.

Solution 1
Clean up the keys in your ssh-agent.

ssh-add -l lists all the keys you have in your ssh-agent
ssh-add -d key deletes the key from your ssh-agent

Solution 2
You can solve this on the command line like this:

ssh -o IdentitiesOnly=yes -i ~/.ssh/example_rsa foo.example.com

What is IdentitiesOnly?  Explained in Solution 3 below.

Solution 3 (best)
Specifiy, explicitly, which key goes to which host(s) in your .ssh/config file.

You need to configure which key (“IdentityFile”) goes with which domain (or host). You also want to handle the case when the specified key doesn’t work, which would usually be because the public key isn’t in ~/.ssh/authorized_keys on the server. The default is for SSH to then try any other keys it has access to, which takes us back to too many attempts. Setting “IdentitiesOnly” to “yes” tells SSH to only try the specified key and, if that fails, fall through to password authentication (presuming the server allows it).

Your ~/.ssh/config would look like:

Host *.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myhost
Host secure.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/mysecurehost_rsa
Host *.myotherhost.domain
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myotherhost_rsa


Host is the host the key can connect to
IdentitiesOnly means only to try this specific key to connect, no others
IdentityFile is the path to the key

You can try multiple keys if needed

Host *.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myhost_rsa
  IdentityFile ~/.ssh/myhost_dsa


Hope this helps someone else.

Cheers!

    SSH - Too Many Authentication Failures was originally published by Jason Meridth at Jason Meridth on March 28, 2018.


Clear DNS Cache In Chrome
Tue, 27 Mar 2018 20:42:00 +0000

    I’m blogging this because I keep forgetting how to do it.  Yeah, yeah, I can google it.  I run this blog so I know it is always available…..anywho.

Go To:

chrome://net-internals/#dns


Click “Clear host cache” button



Hope this helps someone else.

Cheers.

    Clear DNS Cache In Chrome was originally published by Jason Meridth at Jason Meridth on March 27, 2018.


Create Docker Container from Errored Container
Mon, 26 Mar 2018 03:31:00 +0000

    When I’m trying to “dockerize” an applciation I usually have to work through some wonkiness.

To diagnose a container that has errored out, I, obviously, look at the logs via docker logs -f [container_name].  That is sometimes helpful.  It will, at minimum tell me where I need to focus on the new container I’m going to create.



Pre-requisites to being able to build a diagnosis container:


  You need to use CMD, not ENTRYPOINT in the Dockerfile
    
      with CMD you’ll be able to start a shell, with ENTRYPOINT your diagnosis container will just keep trying to run that
    
  


To create a diagnosis container, do the following:


  Check your failed container ID by docker ps -a
  Create docker image form the container with docker commit
    
      example: docker commit -m "diagnosis" [failed container id]
    
  
  Check the newly create docker image ID by docker images
  docker run -it [new container image id] sh
    
      this takes you into a container immediately after the error occurred.
    
  


Hope this helps someone else.

Cheers.

    Create Docker Container from Errored Container was originally published by Jason Meridth at Jason Meridth on March 25, 2018.


Log Early, Log Often… Saved my butt today
Wed, 21 Mar 2018 13:16:03 +0000
In a prior posting (AWS Lambda:Log Early Log often, Log EVERYTHING) I wrote about the virtues and value about having really in depth logging, especially when working with cloud services.  Well today this logging saved my ASS a ton of detective work.
Little Background

I have a background job (Lambda that is called on a schedule) to create/update data cache in a DynamoDB table.  Basically this job will pull data from one data source and attempt to push it as create/update/delete to our Dynamo table.
Today when I was running our application I noticed things were not loading right, in fact I had javascript errors because of null reference errors.  I knew that the issue had to be in our data, but was not sure what was wrong.  If I had not had a ton of logging (debug and info) I would have had to run our code locally and step though/debug code for hundreds of items of data.
However, because of in depth logging I was able to quickly go to CloudWatch and filter on a few key words and narrow hundreds/thousands of log entries down to 5.  Once I had these 5 entries I was able to expand a few of those entries and found the error within seconds.
Total time to find the error was less than 5 minutes and I never opened a code editor or stepped into code.
The moral of this story, because I log everything, including data (no PII of course) I was able to quickly find the source of the error.  Now to fix the code….
Till next time,


AWS Lambda: Log early, Log often, Log EVERYTHING
Tue, 06 Mar 2018 14:00:58 +0000
In the world of building client/server applications logs are important.  They are helpful when trying to see what is going on in your application.  I have always held the belief  that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code.
But lets be honest, in most cases when building client/server applications logs are an afterthought.  Often this is because you can pretty easily (in most cases) debug your application and step through the code.
When building a serverless applications with technologies like AWS Lambda functions (holds true for Azure Functions as well) your logging game really needs to step up.
The reason for this is that you cannot really debug your Lambda in the wild (you can to some degree locally with AWS SAM or the Serverless framework).  Because of this you need produce detailed enough logs to allow you to easily determine the WHAT and WHERE.
When I build my serverless functions I have a few guidelines I follow

Info Log calls to methods, output argument data (make sure no PII/PHI)
Error Log any failures (in try/catch or .catch for promises)
Debug Log any critical decision points
Info Log exit calls at top level methods

I also like to setup a simple and consistent format for my logs.  The example I follow for my Lambda logs is as seen below

timestamp: [logLevel] : [Class.Method] - message {data points} 


I have found that if I follow these general guidelines the pain of determine failure points in serverless environments is heavily reduced.
Till next time,


Sinon Error: Attempted to wrap undefined property ‘XYZ as function
Tue, 27 Feb 2018 13:45:29 +0000
I ran into a fun little error recently when working on a ReactJs application.  In my application I was using SinonJs to setup some spies on a method, I wanted to capture the input arguments for verification.  However, when I ran my test I received the following error.
Attempted to wrap undefined property handlOnAccountFilter as function
My method under test is setup as such

handleOnAccountFilter = (filterModel) => {
    // logic here
}


I was using the above syntax is the proposed class property feature, which will automatically bind the this context of the class to my method.
My sinon spy is setup as such

let handleOnAccountFilterSpy = null;
beforeEach(() => {
    handleOnAccountFilterSpy = sinon.spy(AccountsListingPage.prototype, 'handleOnAccountFilter');
});

afterEach(() => {
     handleOnAccountFilterSpy.restore();
});


Everything looked right, but I was still getting this error.  It turns out that this error is due in part in the way that the Class Property feature implements the handlOnAccountFilter.  When you use this feature the method/property is added to the class as an instance method/property, not as a prototype method/property.  This means that sinon is not able to gain access to it prior to creating an instance of the class.
To solve my issue I had to make a change in the implementation to the following

handleOnAccountFilter(filterModel) {
    // logic here
}


After make the above change I needed to determine how I wanted to bind this to my method (Cory show 5 ways to do this here).  I chose to bind this inside the constructor as below

constructor(props){
    super(props);

    this.handleOnAccountFilter = this.handleOnAccountFilter.bind(this);
}


I am not a huge fan of having to do this (pun intended), but oh well.  This solved my issues.
Till next time


Ensuring componentDidMount is not called in Unit Tests
Thu, 22 Feb 2018 19:45:53 +0000
If you are building a ReactJs you will often times implement componentDidMount on your components.  This is very handy at runtime, but can pose an issue for unit tests.
If you are building tests for your React app you are very likely using enzyme to create instances of your component.  The issue is that when enzyme creates the component it invokes the lifecyle methods, like componentDidMount.  Sometimes we do not want this to be called, but how to suppress this?
I have found 2 different ways to suppress/mock componentDidMount.
Method one is to redefine componentDidMount on your component for your tests.  This could have interesting side effects so use with caution.

    describe('UsefullNameHere', () => {
        
        beforeAll(() => {
            YourComponent.prototype.componentDidMount = () => {
                // can omit or add custom logic
            };
        });
    });


Basically above I am just redefining the componentDidMount method on my component.  This works and allows you to have custom logic.  Be aware that when doing above you will have changed the implementation for your component for the lifetime of your test session.
Another solution is to use a mocking framework like SinonJs.  With Sinon you can stub out the componentDidMount implementation as seen below

    describe('UsefullNameHere', () => {
        
        let componentDidMountStub = null;
        beforeAll(() => {
            componentDidMountStub = sinon.stub(YourComponent.prototype, 'componentDidMount').callsFake(function() {
                 // can omit or add custom logic
            });
        });

        afterAll(() => {
            componentDidMountStub.restore();
        });
    });


Above I am using .stub to redefine the method.  I also added .callsFake() but this can be omitted if you just want to ignore the call.  You will want to make sure you restore your stub via the afterAll, otherwise you will have stubbed out the call for the lifetime of your test session.
Till next time,


Using Manual Mocks to test the AWS SDK with Jest
Tue, 20 Feb 2018 13:56:45 +0000
Anytime you build Node applications it is highly suggested that your cover your code with tests.  When your code interacts with 3rd party API’s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests not unit tests.
If you are using Jest, one solution is utilize the built in support for manual mocks.  I have found the usage of manual mocks invaluable while testing 3rd party API’s such as the AWS.  Keep in mind just because I am using manual mocks this will remove the need for using libraries like SinonJs (a JavaScript framework for creating stubs/mocks/spies).
The way that manual mocks work in Jest is as follows (from the Jest website’s documentation).
Manual mocks are defined by writing a module in a __mocks__/ subdirectory immediately adjacent to the module. For example, to mock a module called user in the models directory, create a file called user.js and put it in the models/__mocks__ directory. Note that the __mocks__ folder is case-sensitive, so naming the directory __MOCKS__ will break on some systems. If the module you are mocking is a node module (eg: fs), the mock should be placed in the __mocks__ directory adjacent to node_modules (unless you configured roots to point to a folder other than the project root).
In my case I want to mock out the usage of the AWS-SDK for Node.
To do this I created a __mocks__ folder at the root of my solution.  I then created a aws-sdk.js file inside this folder.
Now that I have my mocks folder created with a aws-sdk.js file I am able to consume my manual mock in my jest test by simply referencing the aws-sdk via a require('aws-sdk') command.

const AWS = require('./aws-sdk');


With declaration of AWS above my code is able to a use the NPM package during normal usage, or my aws-sdk.js mock when running under the Jest context.
Below is a small sample of the code I have inside my aws-sdk.js file for my manual mock.

const stubs = require('./aws-stubs');
const AWS = {};

// This here is to allow/prevent runtime errors if you are using 
//   AWS.config to do some runtime configuration of the library.
// If you do not need any runtime configuration you can omit this.
AWS.config = {
    setPromisesDependency: (arg) => {}
};

AWS.S3 = function() {

}

// Because I care about using the S3 service's which are part of the SDK
//  I need to setup the correct identifier.
// 
AWS.S3.prototype = {
    ...AWS.S3.prototype,

    // Stub for the listObjectsV2 method in the sdk
    listObjectsV2(params){
        const stubPromise = new Promise((resolve, reject) => {
            // pulling in stub data from an external file to remove the noise
            // from this file.  See the top line for how to pull this in
            resolve(stubs.listObjects);
        });

        return {
            promise: () => {
                return stubPromise;
            }
        }
    }
};

// Export my AWS function so it can be referenced via requires
module.exports = AWS; 


A few things to point out in the code above.

I chose to use the promises implementation of the listObjectsV2.  Because of this I need to return a promise method as my result on my listObjectsV2 function.  I am sure there are other ways to accomplish this, but this worked and is pretty easy.
My function is returning stub data, but this data is described in a separate file called aws-stubs.js which sites along side of my aws-sdk.js file.  I went this route to remove the noise of having the stub data inside my aws-adk file.  You can see a full example of this here.

Now that I have everything setup my tests will no longer attempt to hit the actually aws-sdk, but when running in non-test mode they will.
Till next time,


Configure Visual Studio Code to debug Jest Tests
Fri, 16 Feb 2018 21:33:03 +0000
If you have not given Visual Studio Code a spin you really should, especially if  you are doing web/javascript/Node development.
One super awesome feature of VS Code is the ability to easily configure the ability to debug your Jest (should work just fine with other JavaScript testing frameworks) tests.  I have found that most of the time I do not need to actually step into the debugger when writing tests, but there are times that using console.log is just too much friction and I want to step into the debugger.
So how do we configure VS Code?
First you  will need to install the Jest-Cli NPM package (I am assuming you already have Jest setup to run your tests, if you do not please read the Getting-Started docs).  If you fail to do this step you will get the following error in Code when you try to run the debugger.

After you have Jest-Cli installed you will need to configure VS Code for debugging.  To do this open up the configuration by clicking Debug -> Open Configurations.  This will open up a file called launch.json.
Once launch.json is open add the following configuration

        {
            "name": "Jest Tests",
            "type": "node",
            "request": "launch",
            "program": "${workspaceRoot}/node_modules/jest-cli/bin/jest.js",
            "stopOnEntry": false,
            "args": ["--runInBand"],
            "cwd": "${workspaceRoot}",
            "preLaunchTask": null,
            "runtimeExecutable": null,
            "runtimeArgs": [
                "--nolazy"
            ],
            "env": {
                "NODE_ENV": "development"
            },
            "console": "internalConsole",
            "sourceMaps": false,
            "outFiles": []
        }


Here is a gist of a working launch.json file.
After you save the file you are almost ready to start your debugging.
Before you can debug you will want to open the debug menu (the bug icon on the left toolbar).   This will show a drop down menu with different configurations.  Make sure ‘Jest Test’ is selected.

If you have this setup correctly you should be able to set breakpoints and hit F5.
Till next time,


Going Async with Node AWS SDK with Express
Tue, 13 Feb 2018 13:00:30 +0000
When building applications in Node/Express you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary.  The ‘old school’ way was to use call backs, which often led to callback hell.  Than came along Promises which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything.  Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for async/await was introduced and this really has cleaned up and enhanced the readability of your code.
Having the ability to use async/await is great, and is supported out of the box w/ Express.  But what do you do when you using a library which still wants to use promises or callbacks? The case in point for this article is AWS Node SDK.
By default if you read through the AWS SDK documentation the examples lead you to believe that you need to use callbacks when implementing the SDK.  Well this can really lead to some nasty code in the world of Node/Express.  However, as of v2.3.0 of the AWS SDK there is support for Promises.  This is much cleaner than using callbacks, but still poses a bit of an issue if you want to use async/await in your Express routes.
However, with a bit of work you can get your promise based AWS calls to play nicely with your async/await based Express routes.  Lets take a look at how we can accomplish this.
Before you get started I am going to make a few assumptions.

You already have a Node/Express application setup
You already have the AWS SDK for Node installed, if not read here

The first thing we are going to need to do is add reference to our AWS SDK and configure it to use promises.

const AWS = require('aws-sdk');
AWS.config.setPromisesDependency(null);


After we have our SDK configured we can implement our route handler.  In my example here I am placing all the logic inside my handler.  In a real code base I would suggest better deconstruction of this code into smaller parts.

const express = require('express');
const router = express.Router();
const s3 = new AWS.S3();

router.get('/myRoute', async (req, res) => {

    const controller = new sitesController();
    const params = req.params;
    const params = { 
       Bucket: "bucket_name_here"
    };

    let results = {};
    var listPromise = s3.listObjects(params).promise();
    listPromise.then((data) => {
        results = data;
    });

    await Promise.all([listPromise]);

    res.json({data: results })
})

module.exports = router;


Lets review the code above and call out a few important items.
The first thing to notice is the addition of the async keyword in my route handler.  This is what allows us to use async/await in Node/Express.
The next thing to look at is how I am calling the s3.listObjects.  Notice I am NOT providing a callback to the method, but instead I am chaining with .promise().  This is what instructs the SDK to use promises vs callbacks.  Once I have my callback I chain a ‘then’ in order to handle my response.
The last thing to pay attention to is the line with await Promise.All([listPromise]); This is the magic forces our route handler to not return prior to the resolution of all of our Promises.  Without this your call would exit prior to the listObjects call completing.
Finally, we are simply returning our data from the listObjects call via res.json call.
That’s it, pretty straight forward, once you learn that the AWS SDK supports something other than callbacks.
Till next time,


Unable To Access Mysql With Root and No Password After New Install On Ubuntu
Wed, 31 Jan 2018 00:13:00 +0000

    This bit me in the rear end again today.  Had to reinstall mysql-server-5.7 for other reasons.

You just installed mysql-server locally for your development environment on a recent version of Ubuntu (I have 17.10 artful installed).  You did it with a blank password for root user.  You type mysql -u root and you see Access denied for user 'root'@'localhost'.



Issue:
Because you chose to not have a password for the root user, the auth_plugin for my MySQL defaulted to auth_socket.  That means if you type sudo mysql -u root you will get in.  If you don’t, then this is NOT the fix for you.

Solution:
Change the auth_plugin to mysql_native_password so that you can use the root user in the database.

$ sudo mysql -u root

mysql> USE mysql;
mysql> UPDATE user SET plugin='mysql_native_password' WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> exit;

$ sudo systemctl restart mysql
$ sudo systemctl status mysql


NB
ALWAYS set a password for mysql-server in staging/production.

Cheers.

    Unable To Access Mysql With Root and No Password After New Install On Ubuntu was originally published by Jason Meridth at Jason Meridth on January 30, 2018.


New Job
Mon, 08 Jan 2018 18:13:00 +0000

    Well, it is a new year and I’ve started a new job.  I am now a Senior Software Engineer at True Link Financial.



After interviewing with the co-founders Kai and Claire and their team, I knew I wanted to work here.

TL;DR: True Link:  We give elderly and disable (really, anyone) back their financial freedom where they may not usually have it.

Longer Version:  Imagine you have an elderly family member who may start showing signs of dimensia.  You can give them a True Link card and administer their card.  You link it to their bank account or another source of funding and you can set limitations on when, where and how the card can be used.  The family member feels freedom by not having to continually ask for money but is also protected by scammers and non-friendly people (yep, they exist).

The customer service team, the marketing team, the product team, the engineering team and everyone else at True Link are amazing.

For any nerd readers, the tech stack is currently Rails, React, AWS, Ansible.  We’ll be introducing Docker and Kubernetes soon hopefully, but always ensuring the right tools for the right job.

Looking forward to 2018.

Cheers.


    New Job was originally published by Jason Meridth at Jason Meridth on January 08, 2018.


Hello, React! - A Beginner’s Setup Tutorial
Thu, 25 May 2017 14:23:59 +0000
React  has been around for a few years now and there are quite a few tutorials available. Unfortunately, many are outdated, overly complex, or gloss over configuration for getting started. Tutorials which side-step configuration by using jsfiddle or code generator options are great when you’re wanting to just focus on the framework features itself, but many tutorials leave beginners struggling to piece things together when you’re ready to create a simple react application from scratch. This tutorial is intended to help beginners get up and going with React by manually walking through a minimal setup process.

A Simple Tutorial

This tutorial is merely intended to help walk you through the steps to getting a simple React example up and running. When you’re ready to dive into actually learning the React framework, a great list of tutorials can be found here.

There are a several build, transpiler, or bundling tools from which to select when working with React. For this tutorial, we’ll be using be using Node, NPM, Webpack, and Babel.

Step 1: Install Node

Download and install Node for your target platform. Node distributions can be obtained here.

Step 2: Create a Project Folder

From a command line prompt, create a folder where you plan to develop your example.

$> mkdir hello-react


Step 3: Initialize Project

Change directory into the example folder and use the Node Package Manager (npm) to initialize the project:

$> cd hello-react
$> npm init --yes


This results in the creation of a package.json file. While not technically necessary for this example, creating this file will allow us to persist our packaging and runtime dependencies.

Step 4: Install React

React is broken up into a core framework package and a package related to rendering to the Document Object Model (DOM).

From the hello-react folder, run the following command to install these packages and add them to your package.json file:

$> npm install --save-dev react react-dom


Step 5: Install Babel

Babel is a transpiler, which is to say it’s a tool from converting one language or language version to another. In our case, we’ll be converting EcmaScript 2015 to EcmaScript 5.

From the hello-react folder, run the following command to install babel:

$> npm install --save-dev babel-core


Step 6: Install Webpack

Webpack is a module bundler. We’ll be using it to package all of our scripts into a single script we’ll include in our example Web page.

From the hello-react folder, run the following command to install webpack globally:

$> npm install webpack --global


Step 7: Install Babel Loader

Babel loader is a Webpack plugin for using Babel to transpile scripts during the bundling process.

From the hello-react folder, run the following command to install babel loader:

$> npm install --save-dev babel-loader


Step 8: Install Babel Presets

Babel presets are collections of plugins needed to support a given feature. For example, the latest version of babel-preset-es2015 at the time this writing will install 24 plugins which enables Babel to transpile ECMAScript 2015 to ECMAScript 5. We’ll be using presets for ES2015 as well as presets for React. The React presets are primarily needed for processing of JSX.

From the hello-react folder, run the following command to install the babel presets for both ES2015 and React:

$> npm install --save-dev babel-preset-es2015 babel-preset-react


Step 9: Configure Babel

In order to tell Babel which presets we want to use when transpiling our scripts, we need to provide a babel config file.

Within the hello-react folder, create a file named .babelrc with the following contents:

{                                    
  "presets" : ["es2015", "react"]    
}                                 


Step 10: Configure Webpack

In order to tell Webpack we want to use Babel, where our entry point module is, and where we want the output bundle to be created, we need to create a Webpack config file.

Within the hello-react folder, create a file named webpack.config.js with the following contents:

const path = require('path');
 
module.exports = {
  entry: './app/index.js',
  output: {
    path: path.resolve('dist'),
    filename: 'index_bundle.js'
  },
  module: {
    rules: [
      { test: /\.js$/, loader: 'babel-loader', exclude: /node_modules/ }
    ]
  }
}

Step 11: Create a React Component

For our example, we’ll just be creating a simple component which renders the text “Hello, React!”.

First, create an app sub-folder:

$> mkdir app


Next, create a file named app/index.js with the following content:

import React from 'react';
import ReactDOM from 'react-dom';
 
class HelloWorld extends React.Component {
    render() {
          return (
                  <div>
                    Hello, React!
                  </div>
                )
        }
};
 
ReactDOM.render(<HelloWorld />, document.getElementById('root'));

Briefly, this code includes the react and react-dom modules, defines a HelloWorld class which returns an element containing the text “Hello, React!” expressed using JSX syntax, and finally renders an instance of the HelloWorld element (also using JSX syntax) to the DOM.

If you’re completely new to React, don’t worry too much about trying to fully understand the code. Once you’ve completed this tutorial and have an example up and running, you can move on to one of the aforementioned tutorials, or work through React’s Hello World example to learn more about the syntax used in this example.


  
    Note: In many examples, you will see the following syntax:
  
  

var HelloWorld = React.createClass( {
    render() {
          return (
                  <div>
                    Hello, React!
                  </div>
                )
        }
});

  
  
    This syntax is how classes were defined in older versions of React and will therefore be what you see in older tutorials. As of React version 15.5.0 use of this syntax will produce the following warning:
  
  
  
    Warning: HelloWorld: React.createClass is deprecated and will be removed in version 16. Use plain JavaScript classes instead. If you’re not yet ready to migrate, create-react-class is available on npm as a drop-in replacement.
  


Step 12: Create a Webpage

Next, we’ll create a simple html file which includes the bundled output defined in step 10 and declare a <div> element with the id “root” which is used by our react source in step 11 to render our HelloWorld component.

Within the hello-react folder, create a file named index.html with the following contents:

<html>
  <div id="root"></div>
  <script src="./dist/index_bundle.js"></script>
</html>


Step 13: Bundle the Application

To convert our app/index.js source to ECMAScript 5 and bundle it with the react and react-dom modules we’ve included, we simply need to execute webpack.

Within the hello-react folder, run the following command to create the dist/index_bundle.js file reference by our index.html file:

$> webpack


Step 14: Run the Example

Using a browser, open up the index.html file. If you’ve followed all the steps correctly, you should see the following text displayed:

Hello, React!


Conclusion

Congratulations! After completing this tutorial, you should have a pretty good idea about the steps involved in getting a basic React app up and going. Hopefully this will save some absolute beginners from spending too much time trying to piece these steps together.


Exploring TypeScript
Tue, 30 Aug 2016 18:01:06 +0000

  A  proposal to use TypeScript was recently made within my development team, so I’ve taken a bit of time to investigate the platform.  This article reflects my thoughts and conclusions on where the platform is at this point.


 


  TypeScript: What is It?



  TypeScript is a scripting language created by Microsoft which provides static typing and a class-based object-oriented programming paradigm for transpiling to JavaScript.  In contrast to other compile-to-javascript languages such as CoffeeScript and Dart, TypeScript is a superset of JavaScript which means that TypeScript introduces syntax enhancements to the JavaScript language.



  


 


  Recent Rise In Popularity



  TypeScript made it’s debut in late 2012 and was first released in April 2014.  Community interest has been fairly marginal since it’s debut, but has shown an increase since an announcement that the next version of Google’s popular Angular framework would be written in TypeScript.



  The following Google Trends chart shows the interest parallel between Angular 2 and TypeScript from 2014 to present:



  


 


  The Good



  Type System



  TypeScript provides an optional type system which can aid in catching certain types of programing errors at compile time.  The information derived from the type system also serves as the foundation for most of the tooling surrounding TypeScript.



  The following is a simple example showing a basic usage of the type system:


interface Person {
    firstName: string;
    lastName: string;
}

class Greeter {
    greeting: string;
    constructor(message: string) {
        this.greeting = message;
    }
    greet(person: Person) {
        return this.greeting + " " + person.firstName + " " + person.lastName;
    }
}

let greeter = new Greeter("Hello,");
let person = { firstName: "John", lastName: "Doe" };

document.body.innerHTML = greeter.greet(person);



  In this example, a Person interface is declared with two string properties: firstName and lastName.  Next, a Greeter class is created with a greet() function which is declared to take a parameter of type Person.  Next, instances of Greeter and Person are instantiated and the Greeter instance’s greet() function is invoked passing in the Person instance.  At compile time, TypeScript is able to detect whether the object passed to the greet() function conforms to the Person interface and whether the values assigned to the expected properties are of the expected type.



  Tooling



  While the type system and programming paradigm introduced by TypeScript are its key features, it’s really the tooling facilitated by the type system that makes the platform shine.  Being notified of syntax errors at compile time is helpful, but it’s really the productivity that stems from features such as design-time type checking, intellisense/code-completion, and refactoring that make TypeScript compelling.



  TypeScript is currently supported by many popular IDEs including Visual Studio, WebStorm, Sublime Text, Brackets, and Eclipse.



  EcmaScript Foundation



  One of the differentiators of TypeScript from other languages which transpile to JavaScript (CoffeeScript, Dart, etc.) is that TypeScript builds upon the JavaScript language.  This means that all valid JavaScript code is valid TypeScript code.



  Idiomatic JavaScript Generation



  One of the goals of the TypeScript team was to ensure the TypeScript compiler emitted idiomatic JavaScript.  This means the code produced by the TypeScript compiler is readable and generally follows normal JavaScript conventions.


 


  The Not So Good



  Type Definitions and 3rd-Party Libraries



  Typescript requires type definitions to be created for 3rd-party code to realize many of the benefits of the tooling.  While  the DefinitelyTyped project provides type definitions for the most popular JavaScript libraries used today, there will probably be the occasion where the library you want to use has no type definition file.



  Moreover, interfaces maintained by 3rd-party sources are somewhat antithetical to their primary purpose.  Interfaces should serve as contracts for the behavior of a library.  If the interfaces are maintained by a 3rd-party, however, they can’t be accurately described as “contracts” since no implicit promise is being made by the library author that the interface being provided accurately matches the library’s behavior.  It’s probably the case that this doesn’t prove to be much of an issue in practice, but at minimum I would think relying upon type definitions created by 3rd parties would eventually lead to the available type definitions lagging behind new releases of the libraries being used.



  Type System Overhead



  Introducing a typesystem is a bit of a double-edged sword.  While a type system can provide a lot of benefits, it also adds syntactical overhead to a codebase.  In some cases this can result in the code you maintain actually being harder to read and understand than the code being generated.  This can be illustrated using Anders Hejlsberg’s example presented at Build 2014.



  The TypeScript source in the first listing shows a generic sortBy method which takes a callback for retrieving the value by which to sort while the second listing shows the generated JavaScript source:


interface Entity {
	name: string;
}

function sortBy(a: T[], keyOf: (item: T) => any): T[] {
	var result = a.slice(0);
	result.sort(function(x, y) {
		var kx = keyOf(x);
		var ky = keyOf(y);
		return kx > ky ? 1: kx < ky ? -1 : 0; }); return result; } var products = [ { name: "Lawnmower", price: 395.00, id: 345801 }, { name: "Hammer", price: 5.75, id: 266701 }, { name: "Toaster", price: 19.95, id: 400670 }, { name: "Padlock", price: 4.50, id: 560004 } ]; var sorted = sortBy(products, x => x.price);
document.body.innerText = JSON.stringify(sorted, null, 4);


function sortBy(a, keyOf) {
    var result = a.slice(0);
    result.sort(function (x, y) {
        var kx = keyOf(x);
        var ky = keyOf(y);
        return kx > ky ? 1 : kx < ky ? -1 : 0;
    });
    return result;
}
var products = [
    { name: "Lawnmower", price: 395.00, id: 345801 },
    { name: "Hammer", price: 5.75, id: 266701 },
    { name: "Toaster", price: 19.95, id: 400670 },
    { name: "Padlock", price: 4.50, id: 560004 }
];
var sorted = sortBy(products, function (x) { return x.price; });
document.body.innerText = JSON.stringify(sorted, null, 4);


Comparing the two signatures, which is easier to understand?

TypeScript

 function sortBy<T>(a: T[], keyOf: (item: T) => any): T[]

JavaScript

 function sortBy(a, keyOf)

It might be reasoned that the TypeScript version should be easier to understand given that it provides more information, but many would disagree that this is in fact the case.  The reason for this is that the TypeScript version adds quite a bit of syntax to explicitly describe information that can otherwise be deduced fairly easily.  In many ways this is similar to how we process natural language.  When we communicate, we don’t encode each word with its grammatical function (e.g. “I [subject] bought [past tense verb] you [indirect object] a [indefinite article] gift [direct object].”)  Rather, we rapidly and subconsciously make guesses based on familiarity with the vocabulary, context, convention and other such signals.


   In the case of the sortBy example, we can guess at the parameters and return type for the function faster than we can parse the type syntax.  This becomes even easier if descriptive names are used (e.g. sortByKey(array, keySelector)).  Sometimes implicit expression is simply easier to understand.



  Now to be fair, there are cases where TypeScript is arguably going to be more clear than the generated JavaScript (and for similar reasons).  Consider the following listing:


class Auto{
  constructor(public wheels = 4, public doors?){
  }
}
var car = new Auto();
car.doors = 2;


var Auto = (function () {
    function Auto(wheels, doors) {
        if (wheels === void 0) { wheels = 4; }
        this.wheels = wheels;
        this.doors = doors;
    }
    return Auto;
}());
var car = new Auto();
car.doors = 2;



  In this example, the TypeScript version results in less syntax noise than the generated JavaScript version.   Of course, this is a comparison between TypeScript and it’s generated syntax rather than the following syntax many may have used:


wheels = wheels || 4;

Community Alignment


  While TypeScript is a superset of JavaScript, this deserves some qualification.  Unlike languages such as CoffeeScript and Dart which also compile to JavaScript, TypeScript starts with the EcmaScript specification as the base of it’s language.  Nevertheless, TypeScript is still a separate language.



  A team’s choice to maintain an application in TypeScript over JavaScript isn’t quite the same thing as choosing to implement an application in C# version 6 instead of C# version 5.  TypeScript isn’t the promise: “Programming with the ECMAScript of tomorrow … today!”.  Rather, it’s a language that layers a different programming paradigm on top of JavaScript.  While you can choose how much of the feature superset and programming paradigm you wish to use, the more features and approaches peculiar to TypeScript that are adopted the further the codebase will diverge from standard JavaScript syntax and conventions.



  A codebase that fully leverages TypeScript can tend to look far more like C# than standard JavaScript.  In many ways, TypeScript is the perfect front-end development environment for C# developers as it provides a familiar syntax and programming paradigm to which they are already accustomed.  Unfortunately, developers who spend most of their time in C# often struggle with JavaScript syntax, conventions, and patterns.  The same might be expected to be true for TypeScript developers who utilize the language to emulate object-oriented development in C#.



  Ultimately, the real negative I see with this is that (at least right now) TypeScript doesn’t represent how the majority of Web development is being done in the community.  This has implications on the availability of documentation, availability of online help, candidate pool size, marketability, and skill portability.



  Consider the following chart which compares the current job openings available for JavaScript and TypeScript:



  



  Source: simplyhired.com – August 2016


Now, the fact that there may be far less TypeScript jobs out there than JavaScript jobs doesn’t mean that TypeScript isn’t going to be the next big thing.  What it does mean, however, is that you are going to experience less friction in the aforementioned areas if you stick with standard EcmaScript.


  Alternatives



  For those considering TypeScript, the following are a couple of options you might consider before converting just yet.



  ECMAScript 2015



  If you’re  interested in TypeScript and currently still writing ES5 code, one step you might consider is to begin using ES2015.  In John Papa’s article: “Understanding ES5, ES2015 and TypeScript”, he writes:


Why Not Just use ES2015?  That’s a great option! Learning ES2015 is a huge leap from ES5. Once you master ES2015, I argue that going from there to TypeScript is a very small step.

In many ways, taking the time to learn ECMAScript 2015 is the best option even if you think you’re ready to start using TypeScript.  Making the journey from ES5 to ES2015 and then later on to TypeScript will help you to clearly understand which new features are standard ECMAScript and which are TypeScript … knowledge you’re likely to be fuzzy on if you move straight from ES5 to TypeScript.


  Flow



  If you’ve already become convinced that you need a type system for JavaScript development or you’re just looking to test the waters, you might consider a lighter-weight alternative to the TypeScript platform: Facebook’s Flow project.  Flow is a static type checker for JavaScript designed to gain static type checking benefits  without losing the “feel” of coding in JavaScript and in some cases it does a better job at catching type-related errors than TypeScript.


For the most part, Flow’s type system is identical to that of TypeScript, so it shouldn’t be too hard to convert to TypeScript down the road if desired.  Several IDEs have Flow support including Web Storm, Sublime Text, Atom, and of course Facebook’s own Nuclide.

As of August 2016, Flow also supports Windows.  Unfortunately this support has only recently become available, so Flow doesn’t yet enjoy the same IDE support on Windows as it does on OSX and Linux platforms.  IDE support can likely be expected to improve going forward.


  Test-Driven Development



  If you’ve found the primary appeal of TypeScript to be the immediate feedback you receive from the tooling, another methodology for achieving this (which has far greater benefits) is the practice of Test-Driven Development (TDD). The TDD methodology not only provides a rapid feedback cycle, but (if done properly) results in duplication-free code that is more maintainable by constraining the team to only developing the behavior needed by the application, and results in a regression-test suite which provides a safety net for future modifications as well as documentation for how the system is intended to be used. Of course, these same benefits can be realized with TypeScript development as well, but teams practicing TDD may find less need for TypeScript’s compiler-generated error checking.


 


  Conclusion



  After taking some time to explore TypeScript, I’ve found that aspects of its ecosystem are very compelling, particularly the tooling that’s available for the platform.  Nevertheless, it still seems a bit early to know what role the platform will play in the future of Web development.



  Personally, I like the JavaScript language and, while I see some advantages of introducing type checking, I think a wiser course for now would be to invest in learning EcmaScript 2015 and keep a watchful eye on TypeScript adoption going forward.



Git on Windows: Whence Cometh Configuration
Mon, 22 Aug 2016 09:08:03 +0000
I recently went through the process of setting up a new development environment on Windows which included installing Git  for Windows. At one point in the course of tweaking my environment, I found myself trying to determine which config file a particular setting originated. The command ‘git config –list’ showed the setting, but ‘git config –list –system’, ‘git config –list –global’, and ‘git config –list –local’ all failed to reflect the setting. Looking at the options for config, I discovered you can add a ‘–show-origin’ which led to a discovery: Git for Windows has an additional location from which it derives your configuration.

It turns out, since the last time I installed git on Windows, a change was made for the purposes of sharing git configuration across different git projects (namely, libgit2 and Git for Windows) where a Windows-specific location is now used as the lowest setting precedence (i.e. the default settings). This is the file: C:\ProgramData\Git\config. It doesn’t appear git added a way to list or edit this file as a well-known location (e.g. ‘git config –list windows’), so it’s not particularly discoverable aside from knowing about the ‘–show-origin’ switch.

So the order in which Git for Windows sources configuration information is as follows:


  C:\ProgramData\Git\config
  system config (e.g. C:\Program Files\Git\mingw64\etc\gitconfig)
  global config (%HOMEPATH%.gitconfig
  local config (repository-specific .git/config)


Perhaps this article might help the next soul who finds themselves trying to figure out from where some seemingly magical git setting is originating.


Separation of Concerns: Application Builds & Continuous Integration
Sun, 28 Feb 2016 17:32:48 +0000
I’ve always had an interest in application build processes. From the start of my career, I’ve generally been in the position of establishing the solution architecture for the projects I’ve participated in and this has usually involved establishing a baseline build process.

My  career began as a Unix C developer while still in college where much of my responsibilities required writing tools in both C and various Unix shell scripting languages which were deployed to other workstations throughout the country. From there, I moved on to Unix C-CGI Web development and worked a number of years with Make files. With the advent of Java, I begin using tools like Ant and Maven for several more years before switching to the .Net platform where I used open source build tools like NAnt until Microsoft introduced MSBuild with its 2.0 release. Upon moving to the Austin, TX area, I was greatly influenced by what was the early seat of the Alt.Net movement. It was there where I abandoned what in hindsight has always been a ridiculous idea … trying to script a build using XML. For the next 4-5 years, I used Rake to define all of my builds. Starting last year, I began using Gulp and associated tooling on the Node platform for authoring .Net builds.

Throughout this journey of working with various build technologies, I’ve formed a few opinions along the way. One of these opinions is that the Build process shouldn’t be coupled to the Continuous Integration process.

A project should have a build process which exists and can be executed independent of the particular continuous integration tool one chooses. This allows builds to be created and maintained on the developer’s local machine. The particular build steps involved in building a given application are inherently part of its ontology. What compilers and preprocessors need to be used, how dependencies are obtained and published, when and how configuration values are supplied for different environments, how and where automated test suites are run, how the application distribution is created … all of these are concerns whose definition and orchestration are particular to a given project. Such concerns should be encapsulated in a build script which lives with the rest of the application source, not as discrete build steps defined within your CI tool.

Ideally, builds should never break, but when they do it’s important to resolve the issue as quickly as possible. Not being able to run a build locally means potentially having to repeatedly introduce changes until the build is fixed. This tends to pollute the source code commit history with comments like: “Fixing the build”, “Fixing the build for realz this time”, and “Please let this be it … I’m ready to go home”. Of course, there are times when a build can break because of environmental issues that may not be mirrored locally (e.g. lack of disk space, network related issues, 3rd-party software dependencies, etc.), but encapsulating as much of your build as possible goes a long way to keeping builds running along smoothly. Anyone on your team should be able to clone/check-out the project, issue a single command from the command line (e.g. gulp, rake, psake, etc.) and watch the full build process execute including any pre-processing steps, compilation, distribution packaging and even deployment to a target environment.

Aside from being able to run a build locally, decoupling the build from the CI process allows the technologies used by each to vary independently. Switching from one CI tool to another should ideally just require installing the software, pointing it to your source control, defining the single step to issue the build, and defining the triggers that initiate the process.

The creation of a project distribution and the scheduling mechanism for how often this happens are separate concerns. Just because a CI tool allows you to script out your build steps doesn’t mean you should.


Survey of Entity Framework Unit of Work Patterns
Sun, 01 Nov 2015 21:11:13 +0000
Earlier  this year I joined a development team which chose Entity Framework for the persistence needs of a new greenfield project. While I’ve worked on a few projects which used Entity Framework here and there over the years, the bulk of my experience has been with NHibernate and, more recently, Dapper.Net. As a result, there hasn’t been all that much occasion for me to explore it in any level of depth until this year.

One area I recently took some time to research is how the Unit of Work pattern is best implemented within the context of using Entity Framework. While the topic is still relatively fresh on my mind, I thought I’d use this as an opportunity to create a catalog of various approaches I’ve encountered and include some thoughts about each approach.

Unit of Work

To start, it may be helpful to give a basic definition of the Unit of Work pattern. A Unit of Work can be defined as a collection of operations that succeed or fail as a single unit. Given a series of operations which need to be executed in response to some interaction with an application, it’s often necessary to ensure that none of the operations cause side-effects if any one of them fails. This is accomplished by having participating operations respond to either a commit or rollback message indicating whether the operation performed should be completed or reverted.

A Unit of Work can consist of different types of operations such as Web Service calls, Database operations, or even in-memory operations, however, the focus of this article will be on approaches to facilitating the Unit of Work pattern with Entity Framework.

With that out of the way, let’s take a look at various approaches to facilitating the Unit of Work pattern with Entity Framework.

Implicit Transactions

The first approach to achieving a Unit of Work around a series of Entity Framework operations is to simply create an instance of a DbContext class, make changes to one or more DbSet instances, and then call SaveChanges() on the context. Entity Framework automatically creates an implicit transaction for changesets which include INSERTs, UPDATEs, and DELETEs.

Here’s an example:

public Customer CreateCustomer(CreateCustomerRequest request)
{
  Customer customer = null;

  using (var context = new MyStoreContext())
  {
    customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
    context.Customers.Add(customer);
    context.SaveChanges();
    return customer;
  }
}


The benefit of this approach is that a transaction is created only when necessary and is kept alive only for the duration of the SaveChanges() call. Some drawbacks to this approach, however, are that it leads to opaque dependencies and adds a bit of repetitive infrastructure code to each of your applications services.

If you prefer to work directly with Entity Framework then this approach may be fine for simple needs.

TransactionScope

Another approach is to use the System.Transactions.TransactionScope class provided by the .Net framework. When any of the Entity Framework operations are used which cause a connection to be opened (e.g. SaveChanges()), the connection will enlist in the ambient transaction defined by the TransactionScope class and close the transaction once the TransactionScope is successfully completed. Here’s an example of this approach:

public Customer CreateCustomer(CreateCustomerRequest request)
{
  Customer customer = null;

  using (var transaction = new TransactionScope())
  {
    using (var context = new MyStoreContext())
    {
      customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
      context.Customers.Add(customer);
      context.SaveChanges();
      transaction.Complete();
    }

    return customer;
  }
}


In general, I find using TransactionScope to be a good general-purpose solution for defining a Unit of Work around Entity Framework operations as it works with ADO.Net, all versions of Entity Framework, and other ORMs which provides the ability to use multiple libraries if needed. Additionally, it provides a foundation for building a more comprehensive Unit of Work pattern which would allow other types of operations to enlist in the Unit of Work.

Caution should be exercised when using TransactionScope, however, as certain operations can implicitly escalate the transaction to a distributed transaction causing undesired overhead. For those choosing solutions involving TransactionScope, I would recommend educating yourself on how and when transactions are escalated.

While I find using the TransactionScope class to be a good general-purpose solution, using it directly does couple your services to a specific strategy and adds a bit of noise to your code. While it’s a viable choice, I would recommend inverting the concerns of managing the Unit of Work boundary as shown in approaches we’ll look at later.

ADO.Net Transactions

This approach involves creating an instance of DbTransaction and instructing the participating DbContext instance to use the existing transaction:

public Customer CreateCustomer(CreateCustomerRequest request)
{
  Customer customer = null;

  var connectionString = ConfigurationManager.ConnectionStrings["MyStoreContext"].ConnectionString;
  using (var connection = new SqlConnection(connectionString))
  {
    connection.Open();
    using (var transaction = connection.BeginTransaction())
    {
      using (var context = new MyStoreContext(connection))
      {
        context.Database.UseTransaction(transaction);
        try
        {
          customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
          context.Customers.Add(customer);
          context.SaveChanges();
        }
        catch (Exception e)
        {
          transaction.Rollback();
          throw;
        }
      }

      transaction.Commit();
      return customer;
    }
  }


As can be seen from the example, this approach adds quite a bit of infrastructure noise to your code. While not something I’d recommend standardizing upon, this approach provides another avenue for sharing transactions between Entity Framework and straight ADO.Net code which might prove useful in certain situations. In general, I wouldn’t recommend such an approach.

Entity Framework Transactions

The relative new-comer to the mix is the new transaction API introduced with Entity Framework 6. Here’s a basic example of it’s use:

public Customer CreateCustomer(CreateCustomerRequest request)
{
  Customer customer = null;

  using (var context = new MyStoreContext())
  {
    using (var transaction = context.Database.BeginTransaction())
    {
      try
      {
        customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
        context.Customers.Add(customer);
        context.SaveChanges();
        transaction.Commit();
      }
      catch (Exception e)
      {
        transaction.Rollback();
        throw;
      }
    }
  }

  return customer;
}


This is the approach recommended by Microsoft for achieving transactions with Entity Framework going forward. If you’re deploying applications with Entity Framework 6 and beyond, this will be your safest choice for Unit of Work implementations which only require database operation participation. Similar to a couple of the previous approaches we’ve already considered, the drawbacks of using this directly are that it creates opaque dependencies and adds repetitive infrastructure code to all of your application services. This is also a viable option, but I would recommend coupling this with other approaches we’ll look at later to improve the readability and maintainability of your application services.

Unit of Work Repository Manager

The first approach I encountered when researching how others were facilitating the Unit of Work pattern with Entity Framework was a strategy set forth by Microsoft’s guidance on the topic here. This strategy involves creating a UnitOfWork class which encapsulates an instance of the DbContext and exposes each repository as a property. Clients of repositories take a dependency upon an instance of UnitOfWork and access each repository as needed through properties on the UnitOfWork instance. The UnitOfWork type exposes a SaveChanges() method to be used when all the changes made through the repositories are to be persisted to the database. Here is an example of this approach:

public interface IUnitOfWork
{
  ICustomerRepository CustomerRepository { get; }
  IOrderRepository OrderRepository { get; }
  void Save();
}

public class UnitOfWork : IDisposable, IUnitOfWork
{
  readonly MyContext _context = new MyContext();
  ICustomerRepository _customerRepository;
  IOrderRepository _orderRepository;

  public ICustomerRepository CustomerRepository
  {
    get { return _customerRepository ?? (_customerRepository = new CustomerRepository(_context)); }
  }

  public IOrderRepository OrderRepository
  {
    get { return _orderRepository ?? (_orderRepository = new OrderRepository(_context)); }
  }

  public void Dispose()
  {
    if (_context != null)
    {
      _context.Dispose();
    }
  }

  public void Save()
  {
    _context.SaveChanges();
  }
}

public class CustomerService : ICustomerService
{
  readonly IUnitOfWork _unitOfWork;

  public CustomerService(IUnitOfWork unitOfWork)
  {
    _unitOfWork = unitOfWork;
  }

  public void CreateCustomer(CreateCustomerRequest request)
  {
    customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
    _unitOfWork.CustomerRepository.Add(customer);
    _unitOfWork.Save();
  }
}


It isn’t hard to imagine how this approach was conceived given it closely mirrors the typical implementation of the DbContext instance you find in Entity Framework guidance where public instances of DbSet are exposed for each aggregate root. Given this pattern is presented on the ASP.Net website and comes up as one of the first results when doing a search for “Entity Framework” and “Unit of Work”, I imagine this approach has gained some popularity among .Net developers. There are, however, a number of issues I have with this approach.

First, this approach leads to opaque dependencies. Due to the fact that classes interact with repositories through the UnitOfWork instance, the client interface doesn’t clearly express the inherent business-level collaborators it depends upon (i.e. any aggregate root collections).

Second, this violates the Open/Closed Principle. To add new aggregate roots to the system requires modifying the UnitOfWork each time.

Third, this violates the Single Responsibility Principle. The single responsibility of a Unit of Work implementation should be to encapsulate the behavior necessary to commit or rollback an set of operations atomically. The instantiation and management of repositories or any other component which may wish to enlist in a unit of work is a separate concern.

Lastly, this results in a nominal abstraction which is semantically coupled with Entity Framework. The example code for this approach sets forth an interface to the UnitOfWork implementation which isn’t the approach used in the aforementioned Microsoft article. Whether you take a dependency upon the interface or the implementation directly, however, the presumption of such an abstraction is to decouple the application from using Entity Framework directly. While such an abstraction might provide some benefits, it reflects Entity Framework usage semantics and as such doesn’t really decouple you from the particular persistence technology you’re using. While you could use this approach with another ORM (e.g. NHibernate), this approach is more of a reflection of Entity Framework operations (e.g. it’s flushing model) and usage patterns. As such, you probably wouldn’t arrive at this same abstraction were you to have started by defining the abstraction in terms of the behavior required by your application prior to choosing a specific ORM (i.e. following The Dependency Inversion Principle). You might even find yourself violating the Liskof Substitution Principle if you actually attempted to provide an alternate ORM implementation. Given these issues, I would advise people to avoid this approach.

Injected Unit of Work and Repositories

For those inclined to make all dependencies transparent while maintaining an abstraction from Entity Framework, the next strategy may seem the natural next step. This strategy involves creating an abstraction around the call to DbContext.SaveChanges() and requires sharing a single instance of DbContext among all the components whose operations need to participate within the underlying SaveChanges() call as a single transaction.

Here is an example:

public class CustomerService : ICustomerService
{
  readonly IUnitOfWork _unitOfWork;
  readonly ICustomerRepository _customerRepository;

  public CustomerService(IUnitOfWork unitOfWork, ICustomerRepository customerRepository)
  {
    _unitOfWork = unitOfWork;
    _customerRepository = customerRepository;
  }

  public void CreateCustomer(CreateCustomerRequest request)
  {
    customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
    _customerRepository.Add(customer);
    _unitOfWork.Save();
  }
}


While this approach improves upon the opaque design of the Repository Manager, there are several issues I find with this approach as well.

Similar to the first example, this UnitOfWork implementation is still semantically coupled to how Entity Framework is urging you to think about things. Entity Framework wants you to call SaveChanges() whenever you’re ready to flush any INSERT, UPDATE, or DELETE operations you’ve issued against the database and this abstraction basically surfaces this behavior. If you were to use an alternate framework that supported a different flushing model (e.g. NHibernate), you likely wouldn’t end up with the same abstraction.

Moreover, this approach has no definitive Unit of Work boundary. With this approach, you aren’t defining a logical Unit of Work, but are merely injecting a UnitOfWork you can participate within. When you invoke the underlying DBContext.SaveChanges() method, it isn’t explicit what work will be committed.

While this approach corrects a few design issues I find with the Repository Manager, overall I like this approach even less. At least with the Repository Manager approach you have a defined Unit of Work boundary which is kind of the whole point. My recommendation would be to avoid this approach as well.

Repository SaveChanges Method

The next strategy is basically a variation on the previous one. Rather than injecting a separate type whose sole purpose is to provide an indirect way to call the SaveChanges() method, some merely expose this through the Repository:

public class CustomerService : ICustomerService
{
  readonly ICustomerRepository _customerRepository;

  public CustomerService(ICustomerRepository customerRepository)
  {
    _customerRepository = customerRepository;
  }

  public void CreateCustomer(CreateCustomerRequest request)
  {
    customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
    _customerRepository.Add(customer);
    _customerRepository.SaveChanges();
  }
}


This approach shares many of the same issues with the previous one. While it reduces a bit of infrastructure noise, it’s still semantically coupled to Entity Framework’s approach and still lacks a defined Unit of Work boundary. Additionally, it lacks clarity as to what happens when you call the SaveChanges() method. Given the Repository pattern is intended to be a virtual collection of all the entities within your system of a given type, one might suppose a method named “SaveChanges” means that you are somehow persisting any changes made to the particular entities represented by the repository (setting aside the fact that doing so is really a subversion of the pattern’s purpose). On the contrary, it really means “save all the changes made to any entities tracked by the underlying DbContext”. I would also recommend avoiding this approach.

Unit of Work Per Request

A pattern I’m a bit embarrassed to admit has been characteristic of many projects I’ve worked on in the past (though not with EF) is to create a Unit of Work implementation which is scoped to a Web Application’s Request lifetime. Using this approach, whatever method is used to facilitate a Unit of Work is configured with a DI container using a Per-HttpRequest lifetime scope and the Unit of Work boundary is opened by the first component being injected by the UnitOfWork and committed/rolled-back when the HttpRequest is disposed by the container.

There are a few different manifestations of this approach depending upon the particular framework and strategy you’re using, but here’s a pseudo-code example of how configuring this might look for Entity Framework with the Autofac DI container:

builder.RegisterType<MyDbContext>()
        .As<DbContext>()
        .InstancePerRequest()
        .OnActivating(x =>
        {
          // start a transaction
        })
        .OnRelease(context =>
        {
          try
          {
            // commit or rollback the transaction
          }
          catch (Exception e)
          {
            // log the exception
            throw;
          }
        });

public class SomeService : ISomeService
{
  public void DoSomething()
  {
    // do some work
  }
}



While this approach eliminates the need for your services to be concerned with the Unit of Work infrastructure, the biggest issue with this is when an error happens to occur. When the application can’t successfully commit a transaction for whatever reason, the rollback occurs AFTER you’ve typically relinquished control of the request (e.g. You’ve already returned results from a controller). When this occurs, you may end up telling your customer that something happened when it actually didn’t and your client state may end up out of sync with the actual persisted state of the application.

While I used this strategy without incident for some time with NHibernate, I eventually ran into a problem and concluded that the concern of transaction boundary management inherently belongs to the application-level entry point for a particular interaction with the system. This is another approach I’d recommend avoiding.

Instantiated Unit of Work

The next strategy involves instantiating a UnitOfWork implemented using either the .Net framework TransactionScope class or the transaction API introduced by Entity Framework 6 to define a transaction boundary within the application service. Here’s an example:

public class CustomerService : ICustomerService
{
  readonly ICustomerRepository _customerRepository;

  public CustomerService(ICustomerRepository customerRepository)
  {
    _customerRepository = customerRepository;
  }

  public void CreateCustomer(CreateCustomerRequest request)
  {
    using (var unitOfWork = new UnitOfWork())
    {
      try
      {
        customer = new Customer { FirstName = request.FirstName, LastName = request.LastName };
        _customerRepository.Add(customer);        
        unitOfWork.Commit();
      }
      catch (Exception ex)
      {
        unitOfWork.Rollback();
      }
    }
  }
}


Functionally, this is a viable approach to facilitating a Unit of Work boundary with Entity Framework. A few drawbacks, however, are that the dependency upon the Unit Of Work implementation is opaque and that it’s coupled to a specific implementation. While this isn’t a terrible approach, I would recommend other approaches discussed here which either surface any dependencies being taken on the Unit of Work infrastructure or invert the concerns of transaction management completely.

Injected Unit of Work Factory

This strategy is similar to the one presented in the Instantiated Unit of Work example, but makes its dependence upon the Unit of Work infrastructure transparent and provides a point of abstraction which allows for an alternate implementation to be provided by the factory:

public class CustomerService : ICustomerService
{
  readonly ICustomerRepository _customerRepository;
  readonly IUnitOfWorkFactory _unitOfWorkFactory;

  public CustomerService(IUnitOfWorkFactory unitOfWorkFactory, ICustomerRepository customerRepository)
  {
    _customerRepository = customerRepository;
    _unitOfWorkFactory = unitOfWorkFactory;
  }

  public void CreateCustomer(CreateCustomerRequest request)
  {
    using (var unitOfWork = _unitOfWorkFactory.Create())

Method	`netcoreapp3.1`	`net5.0`
`actual.Contains(expected)`	True	True
`actual.IndexOf(expected)`	1475	-1
`actual.Contains(expected, StringComparison.CurrentCulture)`	True	False
`actual.IndexOf(expected, StringComparison.CurrentCulture)`	1475	-1
`actual.Contains(expected, StringComparison.Ordinal)`	True	True
`actual.IndexOf(expected, StringComparison.Ordinal)`	1475	1475
`actual.Contains(expected, StringComparison.InvariantCulture)`	True	False
`actual.IndexOf(expected, StringComparison.InvariantCulture)`	1475	-1