Los Techies

NServiceBus and .NET Core Generic Host

Mon, 23 Mar 2020 13:19:53 +0000

My current client is using .NET Core 2.x, with plans to upgrade to 3.x next month. As part of that system, we do quite a bit of messaging, with NServiceBus as the tool of choice to help make this easier. To get it working with our .NET Core 2.x applications, we did quite a bit of what I laid out in my Messaging Endpoints in Azure series.

Since then, NServiceBus released first-class support for the .NET Core Generic Host, which underwent a fairly large refactoring in the 2.x to 3.0 timeframe. Andrew Lock's post goes into more detail, but the gist of it is, NServiceBus has first-class support for .NET Core 3.x and later.

What that means for us is hosting NServiceBus inside a .NET Core application couldn't be easier. The NServiceBus.Extensions.Hosting package provides all the integration we need to add a hosted NServiceBus service and integrate with the built-in DI container.

Configuring NServiceBus

With any kind of hosted .NET Core application (Console, ASP.NET Core, Worker), we just need to add the extensions package:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NServiceBus.Extensions.Hosting" Version="1.0.0" />

And add the configuration directly off of the host builder:

Host.CreateDefaultBuilder(args)
    .UseNServiceBus(hostBuilderContext =>
    {
        var endpointConfiguration = new EndpointConfiguration("WebApplication");

        // configure endpoint here

        return endpointConfiguration;
    })
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
    });

Or with a Worker SDK:

<Project Sdk="Microsoft.NET.Sdk.Worker">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.2" />

It's really not much different:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .UseNServiceBus(hostBuilderContext =>
        {
            var endpointConfiguration = new EndpointConfiguration("WorkerService");

            // configure endpoint here

            return endpointConfiguration;
        });

And our endpoint is up and running.

Logging and Serialization

We're not quite there yet, though. The out-of-the-box serialization is XML (which is fine by me), but many folks prefer JSON. Additionally, the logging support inside of NServiceBus is not currently integrated with this package. For serialization, we can use the new System.Text.Json support instead of Newtonsoft.Json.

We'll pull in the community packages from Simon Cropp:

With those two packages in place, we can configure our host's serializer and logging:

Host.CreateDefaultBuilder(args)
    .UseMicrosoftLogFactoryLogging()
    .UseNServiceBus(hostBuilderContext =>
    {
        var endpointConfiguration = new EndpointConfiguration("WorkerService");

        endpointConfiguration.UseSerialization<SystemJsonSerializer>();

We now have integrated logging, hosting, and dependency injection with anything that uses the generic host.

Using the logger

Now in our handlers, we can add dependencies directly on the Microsoft logger, ILogger<T>:

public class SaySomethingHandler : IHandleMessages<SaySomething>
{
    private readonly ILogger<SaySomethingHandler> _logger;

    public SaySomethingHandler(ILogger<SaySomethingHandler> logger) 
        => _logger = logger;

    public Task Handle(SaySomething message, IMessageHandlerContext context)
    {
        _logger.LogInformation("Saying {message}", message.Message);

        return context.Reply(new SaySomethingResponse
        {
            Message = $"Back at ya {message.Message}"
        });
    }
}

And we get an integrated logging experience:

info: NServiceBus.LicenseManager[0]
      Selected active license from C:\Users\jbogard\AppData\Local\ParticularSoftware\license.xml
      License Expiration: 2020-06-16
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\jbogard\source\repos\NsbActivities\WorkerService
info: WorkerService.SaySomethingHandler[0]
      Saying Hello World!

Now with this logging and dependency injection integration, we can use any logger or container that extends the built-in abstractions. My current client (and most) use Serilog, which makes it very easy to plug in to the generic host as well.

With these packages, we'll be able to delete a lot of infrastructure code that wasn't adding any value, which is always a good thing.

Avoid In-Memory Databases for Tests

Wed, 18 Mar 2020 18:25:12 +0000

A controversial GitHub issue came to my attention a couple of weeks ago around ditching the in-memory provider for Entity Framework Core. This seemed like a no-brainer to me - these database providers are far from trivial to maintain, even for in-memory strategies. It's something our teams learned nearly a decade ago, that trying to swap out an in-memory strategy for unit testing simply doesn't provide the value folks may hope for.

It seems rather simple at first - especially in the .NET world and EF Core. EF Core's primary read API is LINQ. LINQ has two flavors - IEnumerable and IQueryable. With IQueryable, an IQueryProvider translates expression trees to...well whatever makes sense for the underlying store. There's a neat trick that you can do, however, as IEnumerable has a method, AsQueryable, to allow complex expression trees to evaluate directly against an in-memory IEnumerable.

Thus, in-memory queryables were born. So why not take advantage of this possibility for unit tests? Why not allow us to swap the implementation of some queryable to the in-memory equivalent, and allow us to write unit tests against in-memory stores?

It all seems so simple, but unfortunately, the devil is in the details.

Simple Ain't Easy

LINQ providers aren't easy. They're not even merely difficult, they're some of the most complex pieces of code you'll see. Why is that?

A LINQ provider is a compiler, a lexer, and a parser, but doesn't own any of those pieces. It's a transpiler, but instead of the output being text, it's API calls. You have to do similar operations as an actual compiler, dealing with ASTs, building your own AST (often), and figuring out how to make a very wide and complex API that is all the IQueryable surface area.

Any ORM maintainer can tell you - the code in the query provider can dwarf the code in the rest of the codebase. This is unlike other ORMs that provide a specialized API or language, such as NHibernate or the MongoDB C# driver.

LINQ surfaces a great amount of flexibility, but that flexibility makes it quite difficult to translate into SQL or any other database API that's already been specifically designed for that database. We're trying to wrap a fit-for-purpose API with a generic-purpose API, and one that can run either in-memory or translated into a database.

On top of that, a good deal of the LINQ surface area can't be translated into SQL, and there are Very Important things that don't translate into LINQ. So you have to extend IQueryable to do fun things like:

using (var context = new BloggingContext())
{
    var blogs = context.Blogs
        .Include(blog => blog.Posts)
            .ThenInclude(post => post.Author)
                .ThenInclude(author => author.Photo)
        .ToList();
}

Yikes! We also have async in the mix, so now we're at the point where the IQueryable isn't remotely the same for in-memory.

But that won't stop us from trying!

Pitfalls of In-Memory

Our teams tried a number of years ago to speed up integration tests by swapping in-memory providers, but we found a number of problems with this approach that led us to abandoning it altogether.

You MUST write a real integration test anyway

First and foremost, an in-memory provider is a pale imitation for the real thing. Even with writing in-memory tests, we still absolutely wrote integration tests against a real database. Those unit tests with in-memory looked exactly like the integration tests, just with a different provider.

Which led us to wonder - if we were writing the tests twice, what was the value of having two tests?

You could write a single test codebase, and run it twice - one with in-memory and one with the real thing, but that has other problems.

You MUST allow raw access to the underlying data API

ORMs allow you to encapsulate your data access, which is good, it allows us to be more productive by focusing on the business problem at hand. But it's also bad because it abstracts your data access, leaving developers to assume that they don't actually need to understand what is going on behind the scenes.

In our projects, we take a pragmatic approach - use the ORM's API when it works, and drop down to the database API when it becomes painful. ORMs these days make it quite easy, such as EF Core's raw SQL capabilities:

var blogs = context.Blogs
    .FromSqlRaw("SELECT * FROM dbo.Blogs")
    .ToList();

There are numerous limitations, many more than with EF6, which is why we often bring in a tool like Dapper to do complex SQL:

var employeeHierarchy = connection.Execute<EmployeeDto>(@"WITH cte_org AS (
    SELECT       
        staff_id, 
        first_name,
        manager_id
        
    FROM       
        sales.staffs
    WHERE manager_id IS NULL
    UNION ALL
    SELECT 
        e.staff_id, 
        e.first_name,
        e.manager_id
    FROM 
        sales.staffs e
        INNER JOIN cte_org o 
            ON o.staff_id = e.manager_id
)
SELECT * FROM cte_org;");

So how do we handle this scenario in our tests? Don't write the unit test (this assumes we're actually writing tests twice)? Somehow exclude it?

What I tend to find is that instead of dropping down to SQL, developers avoid SQL just so that it satisfies the tool. This is unacceptable.

The APIs don't match

The in-memory API of vanilla IQueryProvider doesn't match the LINQ query provider. This means you'll have methods that don't make sense, are no-ops, or even nonsensical for in-memory.

The most obvious example is Include, which instructs the LINQ provider to basically do a join to eagerly fetch some child records. This is to avoid multiple round trips. However, this means nothing to in-memory. You can keep it, remove it, add more, remove more, doesn't matter.

It gets worse on the flip side - when LINQ provides some APIs that aren't supported by the query provider. Since LINQ can run in-memory, it can execute anything on the client side. But when you try to run anything on the server, that won't work:

var blogs = context.Blogs
    .Where(blog => StandardizeUrl(blog.Url).Contains("dotnet"))
    .ToList();

Instead, LINQ providers allow a narrow subset of methods, and even beyond that, a limited set of core .NET methods to translate on the server. But not all obvious methods, and not even all overloads are supported. You don't know this until you actually run the LINQ query against the enormous LINQ provider.

Databases Means Transactions

If I look at a typical integration test we write, we're using both the public and non-public API in a series of transactions to interact with the system under test. Here's a typical example:

[Fact]
public async Task Should_edit_department()
{
    var adminId = await SendAsync(new CreateEdit.Command
    {
        FirstMidName = "George",
        LastName = "Costanza",
        HireDate = DateTime.Today
    });

    var admin2Id = await SendAsync(new CreateEdit.Command
    {
        FirstMidName = "George",
        LastName = "Costanza",
        HireDate = DateTime.Today
    });

    var dept = new Department
    {
        Name = "History",
        InstructorId = adminId,
        Budget = 123m,
        StartDate = DateTime.Today
    };
    await InsertAsync(dept);

    Edit.Command command = null;
    await ExecuteDbContextAsync(async (ctxt, mediator) =>
    {
        var admin2 = await FindAsync<Instructor>(admin2Id);

        command = new Edit.Command
        {
            Id = dept.Id,
            Name = "English",
            Administrator = admin2,
            StartDate = DateTime.Today.AddDays(-1),
            Budget = 456m
        };

        await mediator.Send(command);
    });

    var result = await ExecuteDbContextAsync(db => db.Departments.Where(d => d.Id == dept.Id).Include(d => d.Administrator).SingleOrDefaultAsync());

    result.Name.ShouldBe(command.Name);
    result.Administrator.Id.ShouldBe(command.Administrator.Id);
    result.StartDate.ShouldBe(command.StartDate.GetValueOrDefault());
    result.Budget.ShouldBe(command.Budget.GetValueOrDefault());
}

It's long, but it combines both public APIs (sending commands to create items) and non-public APIs (interacting directly with the DbContext to insert rows), executing an individual command for the test, then finally querying to pull an item out.

In integration tests of long ago, we'd put this entire set of operations in a transaction/unit of work. That's not at all how the application behaves, however, and we'd see many false positives that would only break when each operation was distinct. This is because ORMs use patterns like Unit of Work and Identity Map to determine what to persist and when.

With in-memory providers, there is no "ACID", everything is immediately durable. Each operation is immediately performed, and transactions do nothing! It might seem like a trivial thing, who cares if everything is always immediately durable? The problem is, just like an integration test that uses a single transaction, is that real-life behavior is much different and more complex, and will break in ways you can't predict. Enough false positives, and you wind up distrusting these unit tests.

The database enforces constraints and visibility and isolation levels that these attempts can't, and inevitably, you'll hit problems

But it's working for me!

Great! You're one of the lucky few. Your usage is trivial enough that can fit into the constraints of an in-memory provider. We've tried this (and other in-memory DBs, like SQLite), and it's always failed.

Unfortunately for the EF team, maintaining this provider for public consumption is a cost for them, and a tradeoff. They're coding that instead of coding something else. The question becomes - is the value of a (always flawed) in-memory provider worth the effort for the team?

For me, no, it's not worth negative effects for our team.

Document-Level Pessimistic Concurrency in MongoDB

Mon, 16 Mar 2020 12:28:46 +0000

In the last post, I got quite a few comments on some other ways to approach OCC. One pointed out that I wanted to explore was using a "SELECT...FOR UPDATE", which basically will grant an intent lock on the document in question. With an intent lock, we transition from optimistic concurrency, where we read a document and assume that others won't modify it, but check anyway, to pessimistic concurrency, where we know that our document will be modified by others.

Document-level locks in MongoDB can be granted in single operations or as part of a transaction. Intent locks can be granted as part of a write operation, or at the start of a transaction. Both are slightly different in their approach, one requiring an explicit write, and the other for the entire transaction, so let's look at each approach.

SELECT ... FOR UPDATE

In this approach, we want to trigger an intent exclusive lock (IX) on our first read of the document when we pull it out. The link above describes an approach with the bare API, so we can translate this into the C# version.

The general idea here is that MongoDb itself doesn't support this sort of "intentional" locking, but we can trigger it by updating some field to some new value. If another process has already locked that document, we'll immediately error out. In my case, we can just invent some field, I called it "ETag" to mirror Cosmos DB:

public class Counter
{
    public Guid Id { get; set; }
    public int Version { get; set; }
    public Guid ETag { get; set; }

    public int Value { get; set; }
}

When we first load up the document, we don't just load - we find-and-update that ETag property to some new value, BUT, inside a transaction:

using var session = await client.StartSessionAsync().ConfigureAwait(false);

var transactionOptions = new TransactionOptions(readConcern: ReadConcern.Local, writeConcern: WriteConcern.W1);

session.StartTransaction(transactionOptions);

try
{
    var update = Builders<Counter>.Update.Set(c => c.ETag, Guid.NewGuid());

    var loaded = await collection.FindOneAndUpdateAsync(session, c => c.Id == id, update);

    loaded.Value++;

    result = await collection.ReplaceOneAsync(session, c => c.Id == id, loaded, new UpdateOptions { IsUpsert = false });

    await session.CommitTransactionAsync();
}
catch
{
    await session.AbortTransactionAsync();
    throw;
}

return result;

In that first write operation, or trivial change locks the document for the duration of the transaction. After we perform whatever operation we need to on the document, we write back as normal - no version checking.

When we do this, our throughput drops fairly drastically - since we're locking ahead of time, we disallow any other concurrent write operations. Another concurrent write at the beginning of the operation will simply error out.

In practice, we'd likely want to have some sort of transparent retry mechanism around our operation - especially in scenarios where we're likely to see write collisions. We'd also likely want to introduce some jitter or randomness in our delays, since two operations retrying at the same time will likely collide again.

It can get fairly complicated, which is why you'd only want to introduce this in scenarios where optimistic locking isn't appropriate or viable. In fact, you can see an example of this in action in the NServiceBus MongoDB storage library - it's designed for concurrent operations, and uses pessimistic locking to do so.

Wrapping it up

With optimistic and pessimistic locking solutions possible, we see the flexibility of the MongoDB API. However, I do wish that it were easier and more explicit to call out locking strategies. This might be possible with other APIs on top of the MongoDB client, but as others have pointed out, you're far less likely to "foot-gun" yourself with a client API that encapsulates all of this for us.

Immutability in DTOs?

Thu, 27 Feb 2020 14:11:00 +0000

Something that comes up every so often is the question of whether or not Data Transfer Objects should be immutable - that is, should our design of the classes and types of a DTO enforce immutability.

To answer this question, we first need to look at what purpose a DTO serves. As the name explicitly calls out, it is an object that is used to carry data between processes. In practice, we don't literally transport an object back and forth between processes, and instead there is some form of serialization.

Where the waters get muddied is when types are used to describe message contracts - whether in asynchronous communication in the form of messaging or APIs. Given our usage of DTOs these days, why might we want to enforce immutability?

Benefits of immutability

The primary benefit of immutability is, well, you can't change the object! That makes a lot of sense on the receiving side of a communication - I shouldn't be able to change the message, even it's a deserialized version of that message.

There are some cases where I might want to be able to change a message as it flows through the system. For example, with a Document Message, I often pass the same message between many endpoints, and each endpoint processes the message and adds its own information as it goes along, sometimes with an attached Routing Slip.

This is rarer case, however, and what often winds up happening is I'm not mutating the original message, but the deserialized copy of the message, arriving in the form of the object.

By far the majority of cases are the receiver should not modify the message upon receipt. Sounds great! So why don't we actually do this?

Immutability in a mutable world

The main issue with immutability is that it depends on your language, platform, and even tooling, to support this.

In my systems, the default message content type is application/json, and the senders/receivers are .NET/C# or JavaScript/TypeScript. The main hurdle is that neither of these languages, or the platforms and tooling, support immutability out-of-the-box.

In order to declare an Order message as immutable, it's fairly complex. You have to make sure all write paths are initialized on construction:

public class Order {
  public Order(Customer customer, List<LineItem> lineItems) {
    Customer = customer;
    LineItems = new ReadOnlyCollection<LineItem>(lineItems);
  }
  public Customer Customer { get; }
  public IReadOnlyCollection<LineItem> LineItems { get; }
}
public class Customer {
  public Customer (string firstName, string lastName) {
    // my fingers are tired
  }
  public string FirstName { get; }
  public string LastName { get; }
  // etc
}
public class LineItem {
  public class LineItem(int quantity, decimal total, Product product) {
    // mommy are we there yet
  }
  public decimal Total { get; }
  public int Quantity { get; }
  public Product Product { get; }
}

Because C# doesn't have first-class support for an "immutable" type (like an F# Record Type), it takes a LOT of typing to build out these structures.

When you're done, you'll find that constructing these types is a huge pain. You'll get very long construction declarations:

var order = new Order(
   new Customer(
     "Bob",
     Saget"),
   new {
     new LineItem(10, 100m,
       new Product(
       )
   }
 );

Gross! How do we know what value corresponds to which property? We can add named arguments to make it readable, but those are optional, and still don't correspond to the actual property names (all camelCase). Typically you see no parameter names, making it quite difficult to understand how this all fits together.

Contrast this with a normal C# Object Initializer statement:

var order = new Order {
  Customer = new Customer {
    FirstName = "Bob",
    LastName = "Saget"
  },
  LineItems = new {
    new LineItem {
      Quantity = 10,
      Total = 100m,
      Product = new Product {
        
      }
    }
  }
};

I can see exactly how the entire object is built, one member at a time.

One other major issue with immutability in C# is the support for serializers and deserializers to work is quite a pain. If these tools see something without a setter, that's a problem. If we don't have a no-arg constructor, that's a problem. So we wind up having to bend the tools to be able to handle our types, and very often, with lots of compromises (private, unused setters etc.).

Finally, the nail in the coffin for me, is that when we introduce immutability via constructors, this introduces a breakable contract - a method with fixed arguments. If we add a value to our message, we introduce the possibility that receivers won't be able to properly deserialize.

Given all this, I find it's generally a net negative to attempt immutable DTOs. If the language, framework, and tooling supported it, that would be a different story.

MediatR 8.0 Released

Tue, 31 Dec 2019 14:54:19 +0000

This release brings some (minor) breaking changes to the public API. First, we added a non-generic overload to Send on IMediator:

public interface IMediator
{
    Task<TResponse> Send<TResponse>(IRequest<TResponse> request, CancellationToken cancellationToken = default);

+   Task<object> Send(object request, CancellationToken cancellationToken = default);

    Task Publish(object notification, CancellationToken cancellationToken = default);

    Task Publish<TNotification>(TNotification notification, CancellationToken cancellationToken = default)
        where TNotification : INotification;
}

Second, we've modified the Mediator class to include the CancellationToken for the virtual PublishCore method that allows implementors to override the publishing strategy and accept the notification itself:

-        protected virtual async Task PublishCore(IEnumerable<Func<Task>> allHandlers)
+        protected virtual async Task PublishCore(IEnumerable<Func<INotification, CancellationToken, Task>> allHandlers, INotification notification, CancellationToken cancellationToken)
         {
             foreach (var handler in allHandlers)
             {
-                await handler().ConfigureAwait(false);
+                await handler(notification, cancellationToken).ConfigureAwait(false);
             }
         }

Finally, a new feature! We've added some new built-in pipeline behaviors for handling exceptions. You can now handle specific exceptions (similar to an Exception Filter in ASP.NET Core):

public interface IRequestExceptionHandler<in TRequest, TResponse, TException>
    where TException : Exception
{
    Task Handle(TRequest request, 
        TException exception, 
        RequestExceptionHandlerState<TResponse> state, 
        CancellationToken cancellationToken);
}

Alternatively, if you don't want to provide alternate responses based on exceptions, you can provide exception actions:

public interface IRequestExceptionAction<in TRequest, in TException>
    where TException : Exception
{
    Task Execute(TRequest request, TException exception, CancellationToken cancellationToken);
}

These are a bit simpler if you want to just provide some exception logging. I've also updated the MS DI Extensions package, which registers these new behaviors and interfaces. Enjoy!

User Secrets in Docker-based .NET Core Worker Applications

Mon, 16 Dec 2019 16:53:16 +0000

As part of the recent Message Endpoints in Azure series, I wanted to check out the new .NET Core 3.0 Worker templates to see how the templates have improved the situation (actually, a lot), but there are still some things missing from the Worker SDK versus the Web SDK.

.NET Core Workers, introduced initially as background services in .NET Core 2.x, are now a top-level dotnet and Visual Studio template:

When you create the worker template, you're also given the option to enable Docker support:

In that series, I connect to my instance of Azure Service Bus in the cloud. However, I don't want to commit the connection string to source control, so I use User Secrets to store the connection string locally. Out-of-the-box, this SDK will include a user secrets ID in the project file, however, if you actually try to use the secrets when running, it won't work!

The first reason is because there was an issue in the Worker MSBuild task pipeline that looked for either the Web SDK or an explicit reference to the User Secrets NuGet package to decide to include the User Secrets in the configuration sources - but that's been fixed (workaround here).

However, it still doesn't work if you run a worker in a Docker container inside Visual Studio. The reason here is because when you run in a Docker container, the Visual Studio tooling does a lot of work to make running the container easier - including mapping a bunch of volumes. On that list is User Secrets - however - user secrets are only mapped for Web SDK!

Running our worker app inside a Docker container during development means we don't have any user secrets - and the solutions aren't great. Environment variables, Docker secrets, all are more annoying than just using the original secrets.

Mapping the volume

Instead of mucking around with custom solutions, ideally we can just map the user secrets volume ourselves. Luckily, there's an easy way to do so - we can specify custom Docker command line arguments:

<PropertyGroup>
  <TargetFramework>netcoreapp3.1</TargetFramework>
  <UserSecretsId>89595f67-a846-41e5-a74e-f876488ea8be</UserSecretsId>
  <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
  <DockerfileRunArguments>-v "$(AppData)/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro"</DockerfileRunArguments>
</PropertyGroup>

The DockerfileRunArguments element we use to pass through our additional volume mapping, which you can verify with a normal Web SDK running under Docker to see what it passes through.

I've opened a GitHub issue with the SDK tools, but in the meantime, we can use this workaround. With this in place, our Worker application can now use User Secrets whether it's running on our host or in a container.

Contoso University Vertical Slice App Updated to ASP.NET Core 3.0

Mon, 18 Nov 2019 09:15:01 +0000

To keep a running example of "how we do web apps", I've updated my Contoso University example app to ASP.NET Core 3.0. This sample app is just a re-jiggering of Microsoft's Contoso University Razor Pages sample app . It shows how we (Headspring) typically use:

CQRS w/ MediatR
Razor Pages models w/ AutoMapper
Validation w/ Fluent Validation
Conventional HTML w/ HtmlTags
Database migrations w/ RoundhousE
Integration testing w/ xUnit
Vertical Slice Architecture

The original application didn't really have much/any behavior to speak of in the EF models, so there's not any unit tests, just integration tests. If the app was more than CRUD, we'd refactor handler behavior down to the domain model.

The build script is just pure PowerShell, but in our typical systems, we'd use an actual task-based script runner, like psake or FAKE. Otherwise, it's pretty close to our "normal" stack and usage.

Updating the app to ASP.NET Core 3.0 was quite straightforward, the most I had to do was update some of the models and database to match the updated sample, and convert the services configuration to use the simplified "AddRazorPages" syntax.

Enjoy!

Document-Level Optimistic Concurrency in MongoDB

Wed, 30 Oct 2019 19:42:58 +0000

I've had a number of projects now that have used MongoDB, and each time, I've needed to dig deep into the transaction support. But in addition to transaction support, I needed to understand the concurrency and locking models of Mongo. Unlike many other NoSQL databases, Mongo has locks at the global, database, or collection level, but not at the document level (or row-level, like SQL).

If two processes read a document, then update it, as long as those updates don't collide they'll both succeed, and we'll potentially lose data.

Mongo has had a lot of work done to strengthen its distributed story (see the Jepsen analysis), it has no built-in support for optimistic concurrency control at a document level. With SQL Server, you can use Snapshot Isolation to guarantee no other process has modified data since you read. With Cosmos DB, an etag value is used to check the version of the document being written vs. what exists.

Luckily, it's fairly straightforward to implement document-level optimistic concurrency control. But first, let's prove that without OCC, we can have bad writes.

Without OCC

To start, I'm going to create a very simple document, just an identifier and a counter:

public class Counter
{
    public Guid Id { get; set; }
    public int Value  { get; set; }
}

I'm just going to spin off some tasks to increment the value of the counter, keeping track of the number of writes and final value:

var document = await collection.AsQueryable().Where(doc => doc.Id == id).SingleAsync();

Console.WriteLine($"Before  : {document.Value}");

var tasks = Enumerable.Range(0, 100).Select(async i =>
{
    var loaded = await collection.AsQueryable().Where(doc => doc.Id == id).SingleAsync();

    loaded.Value++;

    long result;
    do
    {
        result = (await collection.ReplaceOneAsync(c => c.Id == id, loaded,
            new UpdateOptions {IsUpsert = false})).ModifiedCount;
    } while (result != 1);

    return result;
}).ToList();

var total = await Task.WhenAll(tasks);

document = await collection.AsQueryable().Where(doc => doc.Id == id).SingleAsync();

Console.WriteLine($"After   : {document.Value}");
Console.WriteLine($"Modified: {total.Sum(r => r)}");

Each pass, I load up the document and increment the value by 1. However, I can have multiple tasks executing at once, so two tasks might read the same value, but only increment by 1. To force a dual write, I continue to update until the collection lock is released. In a real-world scenario, there would be delays between reads/writes that would introduce this issue.

When I run this, I should see an initial value of 0, a final value of 100, and a modified count of 100. But I don't, because some value overwrote each other:

Before  : 1
After   : 92
Modified: 100

I modified 100 times, but the counter only made it up to 92! Let's add some optimistic concurrency to improve things.

With OCC

Some implementations of OCC use a timestamp, but that often isn't precise enough, so instead I'm using a monotonic counter as my version. It starts at zero and goes up from there:

public class Counter
{
    public Guid Id { get; set; }
    public int Version { get; set; }
    public int Value  { get; set; }
}

Version design is a bit more complex, I'm just keeping things simple but we can get as complicated as we want. Now when I update, I'm going to make sure that I both increment my counter and version, and when I send the update, I'll include an additional clause against the originally read version:

var tasks = Enumerable.Range(0, 100).Select(async i =>
{
    var loaded = await collection.AsQueryable().Where(doc => doc.Id == id).SingleAsync();
    var version = loaded.Version;

    loaded.Value++;
    loaded.Version++;

    var result = await collection.ReplaceOneAsync(
        c => c.Id == id && c.Version == version, 
        loaded, new UpdateOptions { IsUpsert = false });

    return result;
}).ToList();

I removed the "retry" to make sure that I don't get any overwrites, and with this in place, the final values line up:

Before  : 0
After   : 92
Modified: 92

However, if I really wanted to make sure that I actually get all of those updates in, I'd need to retry the entire operation:

var tasks = Enumerable.Range(0, 100).Select(async i =>
{
    ReplaceOneResult result;

    do
    {
        var loaded = await collection.AsQueryable()
                           .Where(doc => doc.Id == id)
                           .SingleAsync();
        var version = loaded.Version;

        loaded.Value++;
        loaded.Version++;

        result = await collection.ReplaceOneAsync(
            c => c.Id == id && c.Version == version, loaded,
            new UpdateOptions {IsUpsert = false});
    } while (result.ModifiedCount != 1);

    return result;
}).ToList();

With a simple retry in place, I make sure I reload the document in question, get a refreshed version, and now all my numbers add up:

Before  : 0
After   : 100
Modified: 100

Exactly what I was looking for!

A general solution

This works just fine if I "remember" to include that Where clause correctly, but there's a better way if we want a general solution. For that, I'd do pretty much what I would have in the Life Beyond Distributed Transactions series - introduce a Repository, Unit of Work, and Identity Map. These can all start very simple, but I can encapsulate all of the version checking/managing in these objects instead of forcing all users to remember to include that Where clause.

If there's even a remote possibility that you'll see concurrent updates, you'll likely go down the path of optimistic concurrency. Luckily, a basic solution isn't that much code.

Building Messaging Endpoints in Azure: Functions

Thu, 24 Oct 2019 21:04:04 +0000

Posts in this series:

In our last post, we looked at deploying message endpoints in containers, eventually out to Azure Container Instances. While fairly straightforward, this approach is fairly close to Infrastructure-as-a-Service. I can scale, but I can't auto-scale, and even if I used Kubernetes, I can't scale based on exceeding my lead time SLA (time from when a message enters the queue until when it is consumed).

But what about the serverless option of Azure, Azure Functions? Can this offer me a better experience for building a message endpoint?

So far, the answer is "not really", but it will highly depend on your workload or needs. The programming and hosting model is vastly different than containers or web jobs, so we first need to understand how our function will get triggered.

Choosing a trigger

Something needs to kick off our endpoint, and for this, we have a couple of choices. The most basic choice is the Azure Service Bus binding for Azure Functions. I mention "basic" - because it is. You're not really building an endpoint here, but a single function. It may seem like a small difference, but it's really not. When I'm building endpoints, the handler is just one piece of the puzzle - I have the host configuration, logging, tracing, error handling, and beyond that, complex messaging patterns.

The other option is a pre-release version of the NServiceBus support for Azure Functions, which dramatically alters the development model of functions itself and you're back to developing message handlers (instead of merely functions).

First, let's look at the out-of-the-box binding.

Azure Service Bus binding

With Azure Functions, with their own special SDK package, you're creating a "Functions" project and choosing the "Azure Service Bus" trigger. All this really does behind the scenes is create a project with the correct NuGet package references:

<ItemGroup>
  <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="3.0.6" />
  <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.29" />
</ItemGroup>

Yes it's a little odd - you're adding a package for "WebJobs" but this is a Functions application. That's because the WebJobs triggers and Functions triggers share the same infrastructure, but with a different hosting/deployment model.

In any case, you can now create a function with Many Attributes. Here's one for a simple request/response:

public static class SayFunctionSomethingHandler
{
    [FunctionName("SayFunctionSomethingHandler")]
    [return: ServiceBus("NsbAzureHosting.Sender", Connection = "AzureServiceBus")]
    public static SayFunctionSomethingResponse Run(
        [ServiceBusTrigger("NsbAzureHosting.FunctionReceiver", Connection = "AzureServiceBus")]
        SayFunctionSomethingCommand command, 
        ILogger log)
    {
        log.LogInformation($"C# ServiceBus topic trigger function processed message: {command.Message}");

        return new SayFunctionSomethingResponse { Message = command.Message + " back at ya!"};
    }
}

We have to declare the function name, as an attribute, the trigger, as an attribute, and the return value also as an attribute. Service Bus client takes care of deserializing my message from JSON (assuming the content type of the message was application/json).

Functions (or the Azure Service Bus client) don't understand the concept of "request/response" or "pub/sub" for that matter, so it's up to you to build these concepts on top. If you want to subscribe to an event, you need to set up the topic and subscription inside of the broker.

There's no support for request/response, return addresses, or correlated replies. For us, if we want to "reply" back to the receiver, we'd need to create a client, pick off some reply address from the headers, and generate and send the message.

In the above example, I've hardcoded the reply queue, NsbAzureHosting.Sender, so it's not even following the correct message pattern. With request/reply, the receiver should be ignorant of the sender, just as modern email clients are.

So while all this works, you don't get the more advanced features of a full message endpoint and all the patterns of the Enterprise Integration Patterns book. You have to roll a lot yourself.

You also get only very primitive retry capabilities - retries are immediate and once exhausted the message goes to the dead-letter queue. With NServiceBus, we get immediate and delayed retries - very helpful when we're using some external resource whose downtime won't get resolved within milliseconds.

With all this in mind, let's look at the NServiceBus function support.

NServiceBus bindings

I won't rehash the entire sample, but there are some key differences in this setup than a "normal" functions setup. Azure Functions doesn't have a lot of the extensibility support that NServiceBus, so it won't give you things like Outbox, deferred messages, idempotent receivers, sagas, and so on. So NServiceBus gets around this by still hosting an "endpoint", and delegating the message handling to the endpoint from inside your function:

[FunctionName(EndpointName)]
public static async Task Run(
    [ServiceBusTrigger(queueName: EndpointName)]
    Message message,
    ILogger logger,
    ExecutionContext executionContext)
{
    await endpoint.Process(message, executionContext, logger);
}

The endpoint is what processes the message inside a full execution pipeline, so you can focus on building out full message handlers, instead of just functions:

public class TriggerMessageHandler : IHandleMessages<TriggerMessage>
{
    private static readonly ILog Log = LogManager.GetLogger<TriggerMessageHandler>();

    public Task Handle(TriggerMessage message, IMessageHandlerContext context)
    {
        Log.Warn($"Handling {nameof(TriggerMessage)} in {nameof(TriggerMessageHandler)}");
        return context.SendLocal(new FollowupMessage());
    }
}

Now inside of our message handler, we get the full IMessageHandlerContext, and not just the ExecutionContext of a function, which is fairly limited. Now we can reply, publish, defer, set timeouts, kick off sagas, all inside the full-featured NServiceBus message endpoint.

Neither of these options are great, but for very simple message handlers, an Azure Function can suffice. While an Azure Function isn't close to a "PaaS Message Endpoint", it is close to a "PaaS Message Handler", and that might be sufficient for your needs.

In the last post, I'll look ahead to see how our situation may improve with gulp Kubernetes.

Building Messaging Endpoints in Azure: Container Instances

Tue, 01 Oct 2019 15:18:57 +0000

Posts in this series:

In the last post, we looked at Azure WebJobs as a means of deploying messaging endpoints. And while that may work for smaller loads and simpler systems, as the number of message endpoints grows, dealing with a "sidecar" in a WebJob starts to become untenable.

Once we graduate from WebJobs, what's next? What can balance the ease of deployment of WebJobs with the flexibility to scale only the endpoint as needed?

The closest we come to this is Azure Container Instances. Unlike Kubernetes in Azure Kubernetes Service (AKS), you don't have to manage a cluster yourself. This might change in the future, as Kubernetes becomes more widespread, but for now ACIs are a much simpler step than full on Kubernetes.

With ACIs, I can decide how large each individual instance is, and how many instances to run. As load increases or decreases, I can (manually) spin up or down services. Initially, we might keep things simple and create relatively small instance sizes, and then provision larger ones as we need to.

But first, we need a container!

Deploying Into a Container

From an application perspective, nothing much changes. In fact, we can use the exact same application from our WebJobs instance, except, we don't need to do the ConfigureWebJobs part. It's just a console application!

Different from WebJobs is the instructions to "run" the endpoint. With WebJobs, we needed that run.cmd file. With a container, we'll need a Dockerfile to describe out to build and run our container instance:

FROM microsoft/dotnet:2.2-runtime-alpine AS base

FROM microsoft/dotnet:2.2-sdk-alpine AS build
WORKDIR /src
COPY . .
WORKDIR /src/DockerReceiver
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "DockerReceiver.dll"]

In .NET Core 3.0, this can be simplified to have a self-running executable but with this still on 2.2, we have to define the entrypoint as dotnet DockerReceiver.dll.

With this in place, I can test locally by using Docker directly, or Docker Compose. I went with Docker Compose since it's got out-of-the-box support in Visual Studio, and I can define environment variables more easily:

version: '3.4'

services:
  dockerreceiver:
    image: nsbazurehosting-dockerreceiver
    build:
      context: .
      dockerfile: DockerReceiver/Dockerfile

Then in my local overrides:

version: '3.4'

services:
  dockerreceiver:
    environment:
      - USER_SECRETS_ID=<user secrets guid here>
    volumes:
      - $APPDATA/Microsoft/UserSecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID

With this in place, I can easily run my application inside or outside of a container. I don't really need to configure any networks or anything like that - the containers don't communicate with each other, only with Azure Service Bus (and ASB doesn't come in a containerized format for developers).

I can run with Docker Compose locally, and send a message to make sure everything is connected:

dockerreceiver_1  |                                                                                                                                                                                                                                                             dockerreceiver_1  | info: DockerReceiver.SaySomethingAlsoHandler[0]                                                                                                                                                                                                             dockerreceiver_1  |       Received message: Hello World

Now that we have our image up and running, it's time to deploy it into Azure.

Building and Running in Azure

We first need a place to store our container, and for that, we can use Azure Container Registry, which will contain the built container images from our build process. We can do this from the Azure CLI:

az acr create --resource-group NsbAzureHosting --name nsbazurehosting --sku Basic

With the container registry up, we can add a step to our Azure Build Pipeline to build our container:

steps:
- task: Docker@1
  displayName: 'Build DockerReceiver'
  inputs:
    azureSubscriptionEndpoint: # subscription #
    azureContainerRegistry: nsbazurehosting.azurecr.io
    dockerFile: DockerReceiver/Dockerfile
    imageName: 'dockerreceiver:$(Build.BuildId)'
    useDefaultContext: false
    buildContext: ./

And deploy our container image:

steps:
- task: Docker@1
  displayName: 'Push DockerReceiver'
  inputs:
    azureSubscriptionEndpoint: # subscription #
    azureContainerRegistry: nsbazurehosting.azurecr.io
    command: 'Push an image'
    imageName: 'dockerreceiver:$(Build.BuildId)'

And now our image is pushed! The final piece is to deploy in our build pipeline. Unfortunately, we don't have any built-in step for pushing an Azure Container Instance, but we can use the Azure CLI task to do so:

az container create
    --resource-group nsbazurehosting
    --name nsbazurehosting
    --image nsbazurehosting.azurecr.io/dockerreceiver:latest
    --cpu 1
    --memory 1.5
    --registry-login-server nsbazurehosting.azurecr.io
    --registry-username $servicePrincipalId
    --registry-password $servicePrincipalKey

Not too horrible, but we do need to make sure we allow access to the service principal in the script to authenticate properly. You'll also notice that I was lazy and just picked the latest image, instead of the one based on the build ID.

In terms of the container size, I've just kept things small (I'm cheap), but we can adjust the size as necessary. With this in place, we can push code, our container builds, and gets deployed to Azure.

Using Azure Container Instances

Unfortunately, ACIs are a bit of a red-headed stepchild in Azure. There's not a lot of documentation, and it seems like Azure is pushing us towards AKS and Kubernetes instead of individual instances.

For larger teams or ones that want to completely own their infrastructure, we can build complex topologies, but I don't have time for that.

ACIs also don't have any kind of dynamic scale-out, they're originally designed for spinning up and then down, and the billing reflects this. Recently, the price came down enough to be just about equal as running a same size App Service instance.

However, we can't dynamically increase instances or sizes, so if you want something like that, Kubernetes will be the way to go. It's worth starting with just ACIs, since it won't require a k8s expert on staff.

Up next, I'll be look at hashtag serverless with Azure Functions.

Building Messaging Endpoints in Azure: WebJobs

Thu, 05 Sep 2019 18:23:14 +0000

Posts in this series:

In the last post, I looked at creating a generic host endpoint that many of the deployed versions in Azure can share. By using a hosted service, we can then host NServiceBus in just about anything that can work with the .NET Core generic host. The differences then come to hosting and scaling models.

First up is the closest we have to "Platform-as-a-Service" for background tasks - Azure WebJobs. WebJobs can be any executable/script, but a very common model for building is to use the Azure WebJobs SDK.

Azure WebJobs are a fairly robust implementation of a hosted service - all of the triggers and execution models just piggyback on top of an IHostedService implementation. Instead of configuring individual executables with separate trigger models, we can host multiple "jobs" inside a single deployed host.

So what does this mean for our humble generic host we created earlier?

Picking a WebJobs model

In the last post where we created our own IHostedService instance, this would be separate from the WebJobs SDK "host". So we have a couple of options:

Use our generic host inside a stock WebJob
Use a trigger inside a WebJobs SDK host

With the first option, we can also technically host the WebJobs SDK host, since multiple hosts are supported, so really the question becomes, "will we have other WebJobs to execute or not?"

With the WebJobs SDK, we can host any kind of triggered job, from "cron"-style jobs, to message-driven, continuous and more.

It's really dependent on the other things we have going on outside of our generic host. If we wanted to go strictly with Azure Web Jobs SDK, we'd have to create a "continuous" trigger and ditch our generic host. There's really not much benefit to that choice that I've found, so the path of least resistance is to simply host our generic host inside an executable deployed as-is.

Configuring our WebJobs Project

The WebJobs project is really just a console application project. It doesn't really matter up front, but we can choose to output as an assembly or EXE. Either way, the piece we need to connect just a plain console application to WebJobs is some way to run the WebJob. For this, we create a file called "run.cmd" with instructions on how to run the WebJob:

dotnet WebJobReceiver.dll

If we go pure EXE, we'd just have the EXE file name in our run.cmd file. Finally, we just need to make sure when we build/deploy our application, this file is included in the final published version. We can do this in our .csproj file:

<ItemGroup>
  <Content Include="run.cmd">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </Content>
</ItemGroup>

With this in place, we can dotnet publish our project and the we can deploy this out to Azure. But where should this get deployed?

Deploying Azure WebJobs

Azure WebJobs can't be deployed just by themselves - they have to deployed as part of an Azure AppService (and not a Linux AppService, either). This is somewhat annoying - anything we push out has to be tied with an AppService, for both build and deployment. You can technically push out a WebJob independent of an AppService deployment - but it's a bit ugly.

Azure WebJobs also have a few other disadvantages:

They share the host AppService's resources
They cannot scale independently of the host AppService

In short, if we don't expect too many messages, a WebJob will be fine. We can also technically deploy a WebJob alongside a "null" parent AppService. The parent AppService can be blank/nothing. But again, it's a bit weird.

Deploying a WebJob means we need to combine the packages of the parent AppService and child WebJobs, which is fairly straightforward to do in an Azure DevOps build pipeline. We first have a step to publish the AppService:

steps:
- task: DotNetCoreCLI@2
  displayName: 'Publish Sender'
  inputs:
    command: publish
    arguments: '-c Release --no-build --no-restore -o $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: false
    workingDirectory: Sender

Then one to publish the WebJob:

steps:
- task: DotNetCoreCLI@2
  displayName: 'Publish WebJobReceiver'
  inputs:
    command: publish
    publishWebProjects: false
    projects: WebJobReceiver
    arguments: '-c Release --no-build --no-restore -o $(Build.ArtifactStagingDirectory)\Sender\app_data\jobs\continuous\WebJobReceiver'
    zipAfterPublish: false
    modifyOutputPath: false

Very important here is the output folder for our project - we have to publish our WebJob into a very specific folder in the parent AppService's published output. Finally, we publish the App Service, which includes the WebJob:

steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: Sender'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)\Sender'
    ArtifactName: Sender

With this in place, we can deploy this artifact to an Azure App Service, and see our WebJob in the Azure portal:

And our message endpoint is deployed!

Inevitably the question comes up here - why not a ServiceBus trigger? Why go through all this? We'll get to this in a later post when we look at Azure Functions, but next up, Azure Container Instances.

Integration Testing with xUnit

Tue, 27 Aug 2019 20:40:13 +0000

A few years back, I had given up on xUnit in favor of Fixie because of the flexibility that Fixie provides. The xUnit project is highly opinionated, and geared strictly towards unit tests. It's great for that.

A broader testing strategy includes much more than just unit tests. With Fixie, I can implement any of the XUnit Test Patterns to implement a comprehensive automated test strategy (rather than, say, having different test frameworks for different kinds of tests).

In unit tests, each test method is highly isolated. In integration tests, this is usually not the case. Integration tests usually "touch" a lot more than a single class, and almost always, interact with other processes, files, and I/O. Unit tests are in-process, integration tests are out-of-process.

We can write our integration tests like our unit tests, but it's not always advantageous to do so because:

Shared state (database)
Expensive initialization

A typical integration test

If we look at a "normal" integration test we'd write on a more or less real-world project, its code would look something like:

Set up data through the back door
Set up data through the front door
Build inputs
Send inputs to system
Verify direct outputs
Verify side effects

One very simple example looks something like:

[Fact]
public async Task Should_edit_student()
{
    var createCommand = new Create.Command
    {
        FirstMidName = "Joe",
        LastName = "Schmoe",
        EnrollmentDate = DateTime.Today
    };

    var studentId = await SendAsync(createCommand);

    var editCommand = new Edit.Command
    {
        Id = studentId,
        FirstMidName = "Mary",
        LastName = "Smith",
        EnrollmentDate = DateTime.Today.AddYears(-1)
    };

    await SendAsync(editCommand);

    var student = await FindAsync<Student>(studentId);

    student.ShouldNotBeNull();
    student.FirstMidName.ShouldBe(editCommand.FirstMidName);
    student.LastName.ShouldBe(editCommand.LastName);
    student.EnrollmentDate.ShouldBe(editCommand.EnrollmentDate.GetValueOrDefault());
}

We're trying to test "editing", but we're doing it through the commands actually used by the application. In a real app, ASP.NET Core would modelbind HTTP request parameters to the Edit.Command object. I don't care to test with modelbinding/HTTP, so we go one layer below - send the command down, and test the result.

To do so, we need some setup, namely an original record to edit. The first set of code there does this through the front door, by sending the original "Create" command down.

From here on out, each awaited action is in its own individual transaction, mimicking as much as possible how these interactions would occur in the real world.

But with these styles of tests, there comes a couple of problems:

Some setup I only want to do once, for all tests, similar to the real world
Assertions are more complicated as these interactions can have many side effects

The first problem can be straightforward, but in the second, I usually tackle by switching my tests to a different pattern - "Testcase Class per Fixture". This lets me have a common setup with multiple test methods that each have different specific assertions.

With this in mind, how might we address both issues, with xUnit?

Shared Fixture Design

Each of these issues basically comes down to sharing context between tests. And there are a few ways to do these in xUnit - with collection fixtures and class fixtures.

Collection fixtures allow me to share context amongst many tests. Class fixtures allow me to share context in a class. The lifecycle of each determines what fixture I use for when:

Collection fixtures are set up once per collection
Class fixtures are set up once per test class
Constructors/lifetimes are set up once per test method

That last one is important - if I do set up in an xUnit constructor or IAsyncLifetime on a test class, it executes once per test method - probably not what I want! What I'm looking for looks something like:

In each class, the Fixture contains the "Arrange/Act" parts of the test, and each test method contains the "Assert" part. That way, our test method names can describe the assertion from a behavioral perspective.

For our shared context, we'd want to create a collection fixture. This could include:

Database stuff
App stuff
Configuration stuff
Container stuff

Anything that gets set up in your application's startup is a good candidate for our shared context. Here's one example of a collection definition that uses both ASP.NET Core hosting stuff and Mongo2Go

public class SharedFixture : IAsyncLifetime
{
    private MongoDbRunner _runner;
    public MongoClient Client { get; private set; }
    public IMongoDatabase Database { get; private set; }

    public async Task InitializeAsync() {
        _runner = MongoDbRunner.Start();
        Client = new MongoClient(_runner.ConnectionString);
        Database = Client.GetDatabase("db");
        var hostBuilder = Program.CreateWebHostBuilder(new string[0]);
        var host = hostBuilder.Build();
        ServiceProvider = host.Services;
    }

    public Task DisposeAsync() {
        _runner?.Dispose();
        _runner = null;
        return Task.CompletedTask;
    }
}

Then we create a collection fixture definition in a separate class:

[CollectionDefinition(nameof(SharedFixture))]
public class SharedFixtureCollection : ICollectionFixture<SharedFixture>
{ }

Now that we have a definition of a shared fixture, we can try to use it in test. But first, we need to build out the test class fixture.

Class Fixture Design

For a class fixture, we're doing Arrange/Act as part of its design. But that also means we'll need to use our collection fixture. Our class fixture needs to use our collection fixture, and xUnit supports this.

Here's an example of a class fixture, inside a test class:

public class MyTestClass_For_Some_Context
    : IClassFixture<MyTestClass_For_Some_Context.Fixture> {
    private readonly Fixture _fixture;
    public MyTestClass_For_Some_Context(Fixture fixture)
        => _fixture = fixture;
    
    [Collection(nameof(SharedFixture)]
    public class Fixture : IAsyncLifetime {
        private readonly SharedFixture _sharedFixture;
        public Fixture(SharedFixture sharedFixture)
            => _sharedFixture = sharedFixture;

        public async Task InitializeAsync() {
            // Arrange, Act
        }
        public Order Order { get; set; }
        public OtherContext Context { get; set; }
        // no need for DisposeAsync
    }

    [Fact]
    public void Should_have_one_behavior() {
       // Assert
    }

    [Fact]
    public void Should_have_other_behavior() {
       // Assert
    }
}

The general idea is that fixtures must be supplied via the constructor, so I have to create a bit of a nested doll here. The class fixture takes the shared fixture, then my test class takes the class fixture. I use the InitializeAsync method for the "Arrange/Act" part of my test, then capture any direct/indirect output on properties on my Fixture class.

Then, in each test method, I only have asserts which look at the values in the Fixture instance supplied in my test method.

With this setup, my "Arrange/Act" parts are only executed once per test class. Each test method can be then very explicit about the behavior to test (the test method name) and assert only one specific aspect.

It's Ugly

Writing tests this manner allows me to fit inside xUnit's lifecycle configuration - but it's super ugly. I have attributes with magic values (the collection name), marker interfaces, nested classes (though this was my doing) and in general a lot of hoops.

What I would like to do is just have an easy way to define global, shared state, and define a separate lifecycle for integration tests. I don't mind using the IAsyncLifetime part but it's a bit annoying to have to work through a separate fixture class to do so.

So while it's feasible to write tests in this way, I'd suggest avoiding it. Having global shared state is possible, but combining those with class fixtures is just too complicated.

Instead, either use a framework more suited for this style of tests (Fixie or any of the BDD-style libraries), or just combine all asserts into one single test method.

AutoMapper LINQ Support Deep Dive

Fri, 16 Aug 2019 14:21:02 +0000

My favorite feature of AutoMapper is its LINQ support. If you're using AutoMapper, and not using its queryable extensions, you're missing out!

Normal AutoMapper usage is something like:

var dest = _mapper.Map<Dest>(source);

Which would be equivalent to:

var dest = new Dest {
    Thing = source.Thing,
    Thing2 = source.Thing2,
    FooBarBaz = source.Foo?.Bar?.Baz
};

The main problem people run into here is that typically that source object is some object filled in from a data source, whether it's a relational or non-relational source. This implies that the original fetch pulled a lot more information back out than we needed to.

Enter projections!

Constraining data fetching with projection

If we wanted to fetch only the data we needed from the server, the quickest path to do so would be a projection at the SQL level:

SELECT
    src.Thing,
    src.Thing2,
    bar.Baz as FooBarBaz
FROM Source src
LEFT OUTER JOIN Foo foo on src.FooId = foo.Id
LEFT OUTER JOIN Bar bar on foo.BarId = bar.Id

Well...that's already getting a bit ugly. Plus, all that joining information is already represented in our object model, why do we have to drop down to such a low level?

If we're using LINQ with our data provider, then we can use the query provider to perform a Select projection

var dest = await dbContext.Destinations
   .Where(d => d.Id = id)
   .Select(d => new Dest {
       Thing = source.Thing,
       Thing2 = source.Thing2,
       FooBarBaz = source.Foo.Bar.Baz.
   })
   .FirstOrDefaultAsync();

And underneath the covers, the query provider parses the Select expression tree and converts that to SQL, so that the SELECT query against the database is only what we need and our query provider skips that data/domain model altogether to go from SQL straight to our destination type.

All is great! Except, of course, we're back to boring code, but this time, it's projections!

AutoMapper LINQ Support

Enter AutoMapper's LINQ support. Traditional AutoMapper usage is in in-memory objects, but several years ago, we also added the ability to automatically build out those Select projections for you as well:

// Before
var dest = await dbContext.Destinations
   .Where(d => d.Id = id)
   .Select(d => new Dest { // Just automap this dumb junk
       Thing = source.Thing,
       Thing2 = source.Thing2,
       FooBarBaz = source.Foo.Bar.Baz.
   })
   .FirstOrDefaultAsync();

// After
var dest = await dbContext.Destinations
   .Where(d => d.Id = id)
   .ProjectTo<Dest>() // oh so pretty
   .FirstOrDefaultAsync();

These two statements are exactly equivalent. AutoMapper behind the scenes builds up the Select projection expression tree exactly how you would do this yourself, except it uses the AutoMapper mapping configuration to do so.

We get the best of both worlds here - enforcement of our destination type conventions, got rid of all that dumb projection code, and safety with configuration validation.

But how does this all work?

Underneath the Covers

Behind the scenes, it all starts with extending out IQueryable (not IEnumerable) to create the ProjectTo method:

public static class Extensions {
    public static IQueryable<TDestination> ProjectTo<TDestination>(
        this IQueryable source,
        IConfigurationProvider configuration)
    => new ProjectionExpression(source, configuration.ExpressionBuilder)
        .To<TDestination>();
}

I've pushed all the projection logic to a separate object, ProjectionExpression. One critical thing we need to do is make sure we're returning the exact same IQueryable instance at the end, so that you can do the extended behaviors that many ORMs support, such as async queries.

Next, we build up an expression tree that needs to be fed in to IQueryable.Select:

public class ProjectionExpression {
    // ctor etc
    public IQueryable<TResult> To<TResult>() {
        return (IQueryable<TResult>) _builder.GetMapExpression(
                _source.ElementType, 
                typeof(TResult))
            .Aggregate(_source, Select);
    }
}

GetMapExpression return IEnumerable<LambdaExpression>, which we then use to reduce to the final Select call on the IQueryable instance:

private static IQueryable Select(IQueryable source, LambdaExpression lambda) => source.Provider.CreateQuery(
        Expression.Call(
            null,
            QueryableSelectMethod.MakeGenericMethod(source.ElementType, lambda.ReturnType),
            new[] { source.Expression, Expression.Quote(lambda) }
            )
        );

Calling in to the underlying IQueryProvider instance ensures that we're using the query provider, instead of just plain Queryable, to create the query. The underlying query provider often does "more" things that need to be tracked, and this also makes sure that the final IQueryable is the one from the original query provider - not the built-in one in .NET.

The original GetMapExpression method on the IExpressionBuilder instance isn't too too exciting, we do some caching of building out expressions, and have a chain of responsibility pattern in place to decide how to bind each destination member based on different rules (things that need to be mapped, enumerables, strings etc.).

We start by finding the mapping configuration for the source/destination type pair, then for each destination member, pick an appropriate expression building strategy based on that the source/destination type pair for each member, including any member configuration you've put in place.

The end result is a Select call to the correct IQueryProvider instance, with a fully built-out expression tree, that the underlying IQueryProvider instance can then take and build out the correct data projection straight from the server.

And because we simply chain off of IQueryProvider, we can extend any query provider that also handles Select.

When things go wrong

The expression trees we build up need to be parsed and understood by the underlying query provider. This is not always the case, and if you search EF or EF Core for issues containing the words "AutoMapper", you'll find many, many issues with that expression parsing/translation (> 100 in the EF Core repository alone). It's not a perfect system, and expression tree parsing is a hard problem.

When you run into an issue with the expression tree, you'll get some weird wonky error message from the query provider. When you do, the easiest thing to diagnose is to drop back down in to "raw" LINQ, calling Select manually. Once you do, you can remove members one-by-one until you find the underlying problem mapping, and join dozens of others opening issues in the appropriate GitHub repository :).

For nearly all cases of straightforward projection, it works great! So if you're using AutoMapper, check out ProjectTo, as it's the greatest (mapping) thing since sliced bread.

AutoMapper 9.0 Released

Mon, 12 Aug 2019 13:35:37 +0000

As with the other major releases of AutoMapper, this one introduces breaking API changes with relatively few other additions. The major breaking changes (see upgrade guide for details) include:

Removing the static API
Removing "dynamic" maps (automatically created maps)
Fix on IMappingAction to include a ResolutionContext parameter

The motivation behind removing the static API is that most new users to AutoMapper will do so through the DI extensions packages, using services.AddAutoMapper(typeof(Startup)). If you want a static usage of AutoMapper, you can still do so, but you're in charge of creating a static holder and referencing that.

The bigger, more unpopular change, is removing dynamic maps. Dynamic maps have a long history in AutoMapper, and it was actually removed once before added back in. I've never used them, will never use them, and they go against the underlying design philosophy of the project. I don't want to support features I don't use or recommend, so this is getting the axe.

Let this be a lesson to other OSS authors - never add features to your library you'd recommend avoiding, no matter how much a feature is "wanted".

I've also released the latest version of the DI package, and will roll out updates to other ancillary packages shortly.

Enjoy! Or not ;)

Immutability in Message Types

Thu, 08 Aug 2019 18:50:31 +0000

In just about every custom I've worked with, eventually the topic of immutability in messages comes up. Messages are immutable in concept, that is, we shouldn't change them (except in the case of document messages). Since messages are generally immutable in concept, why not make them immutable in our applications dealing with the messages themselves?

Immutabilty has a number of benefits, ask Mark Seemann laid out in a discussion question on the NServiceBus discussion forum. Typically, a message/command/event is defined in type-oriented systems as a Data Transfer Object (DTO):

public class CreateOrder: ICommand
{
    public int OrderId { get; set; }
    public DateTime Date { get; set; }
    public int CustomerId { get; set; }
}

Above is an example of a command message contract defined in NServiceBus. There are a number of problems with mutable message types:

Message receivers can modify the deserialized representation, "changing the facts"
No invariants specified. What's required? What's valid? What can be null?

The natural reaction to these issues are to add behavior to our message types.

Building Immutability

If we want immutability, and record-type-like behavior, we'll need to do a few things. First is to worry about serialization/deserialization. With DTOs, it's rather easy to guarantee that our wire format can be serialized. As long as we stick to "normal" primitives, public setters, and "normal" collection types, everything works well.

But as anyone who has tried to create fully encapsulated domain models that are also data models, we encapsulation brings pain with dealing with data hydration. In encapsulated domain models, there's usually some level of compromise I undergo to get my domain model good enough for our purposes. It will never be perfect - but perfection is not the goal, shipping is.

To build immutabilty in C#, we can guarantee this with interfaces (full sample from the NServiceBus docs):

public interface ICreateOrder : ICommand
{
    int OrderId { get; }
    DateTime Date { get; }
    int CustomerId { get; }
}

We need to use an interface because many serializers don't support C#'s read-only properties. If you define your message type as readonly properties:

public class CreateOrder : ICommand
{
    public CreateOrder(int orderId, DateTime date, int customerId)
    {
        OrderId = orderId;
        Date = date;
        CustomerId = customerId;
    }

    public int OrderId { get; }
    public DateTime Date { get; }
    public int CustomerId { get; }
}

Then behind the scenes, the C# compiler creates a readonly backing field, and the constructor sets this readonly field. Instead, you have to create private setters to allow deserialization to even be possible

public class CreateOrder : ICreateOrder
{
    public CreateOrder(int orderId, DateTime date, int customerId)
    {
        OrderId = orderId;
        Date = date;
        CustomerId = customerId;
    }

    public int OrderId { get; private set; }
    public DateTime Date { get; private set; }
    public int CustomerId { get; private set; }
}

Then your handler just uses that readonly interface. I also see that my IDE sees that those private setters are redundant for code purposes, but I can't remove them, or my serialization won't work.

We get immutability, but at a price - more code to write, and funny, still not semantically valid code that has to know how exactly our serializers work. Is it even worth it?

What Even Is A Message

In many systems, we substitute "type" for "message" or "contract". But that's not what a message is - our message is what's on the wire. The schema or contract is the agreement or specification for what that message should look like, and expected application-level semantics around processing that message.

In the web API world, quite a lot of work has gone into building specifications around APIs, first with Swagger and now around the Open API specification. In fact, Swagger is now built around Open API Spec (OAS). A wealth of tooling now supports defining, describing, and using Open API-spec-defined APIs.

Not so much on the durable message side of things. Many specifications exist on message protocols, but not the messages themselves. It's really up to producers and consumers to build specifications on the content of the messages.

It's a common attempt in messaging systems to try and define the schema as the type, but the problem is a type system is semantically similar, but not equivalent, to a schema. You wind up having the issues that WSDL had - schemas that had too strict rules or assumptions on specific runtime's type systems. Anyone that's tried to consume a Java web service from .NET will know how off this was.

Ultimately, the type is not the message, but a convenient mechanism of describing the schema and providing serialization. It isn't the message, though, and we should refrain from trying to put object-oriented concepts on top of it. Those don't translate to wire formats, or even message schema specifications.

Schemas and Contracts and Classes (oh my)

The third tenet of Service Oriented Architecture is:

Services share schema and contract, not class (or type)

But what immutability in C# is trying to do is share and enforce classes/types amongst producers/consumers, moving the DTO past just a serialization helper to an actual behavioral object. Building behavior into our serialization type would break a tenet of SOA.

Our producers and consumers should instead focus on the message schema and contract, and application protocols around such, than the manifestation in a specific type system. Ideally, we could define our schemas and contracts, and share those, but we're still a ways off from being able to do so (like you could with WSDL or Open API).

Until then, we can share types, but only if we understand that the type is not the message, the schema defines the message, and the type is only a convenience mechanism to assist in describing and materializing a message.

Today, just use a DTO as your message type, keep it simple, and keep an eye out for efforts to help define (and validate) async messages, and tooling to help define message types/objects based on your target framework/runtime of choice.

Gocode Vim Plugin and Go Modules

Sat, 05 Jan 2019 17:09:26 +0000

I recently purchased Let’s Go from Alex Edwards. I wanted an end-to-end Golang website tutorial. It has been great. Lots learned.

Unfortunately, he is using Go’s modules and the version of the gocode Vim plugin I was using did not support Go modules.

Solution:

Use this fork of the gocode Vim plugin and you’ll get support for Go modules.

I use Vim Plug for my Vim plugins. Huge fan of Vundle but I like the post-actions feature of Plug. I just had to change one line of my vimrc and re-run updates.

diff --git a/vimrc b/vimrc
index 3e8edf1..8395705 100644
--- a/vimrc
+++ b/vimrc
@@ -73,7 +73,7 @@ endif
           let editor_name='nvim'
           Plug 'zchee/deoplete-go', { 'do': 'make'}
         endif
-        Plug 'nsf/gocode', { 'rtp': 'vim', 'do': '~/.config/nvim/plugged/gocode/vim/symlink.sh' }
+        Plug 'stamblerre/gocode', { 'rtp': 'vim', 'do': '~/.vim/plugged/gocode/vim/symlink.sh' }
         Plug 'godoctor/godoctor.vim', {'for': 'go'} " Gocode refactoring tool
     " }

That is the line I had to change then run :PlugUpdate! and the new plugin was installed.

I figured all of this out due to this comment by Tommaso Sardelli on Github. Thank you Tommaso.

Raspberry Pi Kubernetes Cluster - Part 4

Fri, 28 Dec 2018 16:35:23 +0000

Raspberry Pi Kubenetes Cluster - Part 1

Raspberry Pi Kubenetes Cluster - Part 2

Raspberry Pi Kubenetes Cluster - Part 3

Raspberry Pi Kubenetes Cluster - Part 4

Howdy again.

In this post I’m going to show you how to create a docker image to run on ARM architecture and also how to deploy it and view it.

To start please view my basic flask application called fl8 here

If you’d like to clone and use it:

git clone git@github.com:meridth/fl8.git && cd fl8

ARM docker image

First we need to learn about QEMU

What is QEMU and QEMU installation

QEMU (Quick EMUlator) is an Open-Source hosted hypervisor, i.e. an hypervisor running on a OS just as other computer programs, which performs hardware virtualization. QEMU emulates CPUs of several architectures, e.g. x86, PPC, ARM and SPARC. It allows the execution of non-native target executables emulating the native execution and, as we require in this case, the cross-building process.

Base Docker image that includes QEMU

Please open the Dockerfile.arm and notice the first line: FROM hypriot/rpi-alpine. This is a base image that includes the target qemu statically linked executable, qemu-arm-static in this case. I chose hypriot/rpi-alpine because the alpine base images are much smaller than other base images.

Register QEMU in the build agent

To add QEMU in the build agent there is a specific Docker Image performing what we need, so just run in your command line:

docker run --rm --privileged multiarch/qemu-user-static:register --reset

Build image

docker build -f ./Dockerfile.arm -t meridth/rpi-fl8 .

And voila! You now have an image that will run on Raspberry Pis.

Deployment and Service

/.run-rpi.sh is my script where I run a Kubernetes deployment with 3 replicas and a Kubernetes service. Please read fl8-rpi-deployment.yml and fl8-rpi-service.yml. They are only different from the other deployment and service files by labels. Labels are key/vaule pairs that can be used by selectors later.

The deployment will pull my image from meridth/rpi-fl8 on dockerhub. If you have uploaded your docker image somewhere you can change the deployment file to pull that image instead.

Viewing application

kubectl get pods

Choose a pod to create the port forwarding ssh tunnel.

kubectl port-forward [pod-name] [app-port]:[app-port]

Example: kubectl port-forward rpi-fl8-5d84dd8ff6-d9tgz 5010:5010

The final result when you go to http://localhost:5010 in a browser.

Hope this helps someone else. Cheers.

Raspberry Pi Kubernetes Cluster - Part 4 was originally published by Jason Meridth at Jason Meridth on December 28, 2018.

Raspberry Pi Kubernetes Cluster - Part 3

Mon, 24 Dec 2018 21:59:23 +0000

Raspberry Pi Kubenetes Cluster - Part 1

Raspberry Pi Kubenetes Cluster - Part 2

Raspberry Pi Kubenetes Cluster - Part 3

Raspberry Pi Kubenetes Cluster - Part 4

Well, it took me long enough to follow up on my previous posts. There are reasons.

The day job has been fun and busy
Family life has been fun and busy
I kept hitting annoying errors when trying to get my cluster up and running

The first two reasons are the usual reasons a person doesn’t blog. :)

The last one is what prevented me from blogging sooner. I had mutliple issues when trying to use rak8s to setup my cluster. I’m a big fan of Ansible and I do not like running scripts over and over. I did read K8S on Raspbian Lite from top to bottom and realized automation would make this much better.

The issues I experienced:

apt-get update would not work

I started with the vanilla Raspbian lite image to run on my nodes and had MANY MANY issues with running apt-get update and apt-get upgrade. The mirrors would disconnect often and just stall. This doesn’t help my attempted usage of rak8s which does both on the cluster.yml run (which I’ll talk about later).

rak8s changes needed to run HypriotOS and kubernetes 1.13.1

Clone the repo locally and I’ll walk you through what I changed to get rak8s working for me and HypriotOS.

Change the following files:

ansible.cfg
- change user from pi to pirate
roles/kubeadm/tasks/main.yml
- add ignore_errors: True to Disable Swap task
- I have an open PR for this here
group_vars/all.yml
- Change kubernetes_package_version to "1.13.1-00"
- Change kubernetes_version to "v1.13.1"

After you make those changes you can run ansible-playbook cluster.yml as the rak8s documentation suggests. Please note this is after you edit inventory and copy ssh keys to raspberry pis.

Flannel networking issue once nodes are up

After I get all of the nodes up I noticed the master node was marked ast NotReady and when I ran kubectl describe node raks8000 I saw the following error:

KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

This error is known in kubernetes > 1.12 and flannel v0.10.0. It is mentioned in this issue. The fix is specifically mentioned here. It is to run the following command

kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

After readin the issue it seems the fix will be in the next version of flannel and will be backported to v0.10.0.

A running cluster

Raspberry Pi Kubernetes Cluster - Part 3 was originally published by Jason Meridth at Jason Meridth on December 24, 2018.

MVP how minimal

Thu, 20 Dec 2018 20:00:00 +0000

MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas. In recent history I’ve seen up close two extreme constrasting examples of MVP:

Mega Minimal: website and db, mostly manual on the backend
Mega Mega: provisioning system, dynamic tuning of systems via ML, automated operations, monitoring a few others I’m leaving out.

Feedback

If we’re evaluating which approach gives us more feedback, Mega Minimal MVP is gonna win hands down here. Some will counter they don’t want to give people a bad impression with a limited product and that’s fair, but it’s better than no impression (the dreaded never shipped MVP). The Mega Mega MVP I referenced took months to demo. only had one of those checkboxes setup and wasn’t ever demod again. So we can categorical say that failed at getting any feedback.

Whereas the Mega Minimal MVP, got enough feedback and users for the founders to realize that wasn’t a business for them. Better than after hiring a huge team and sinking a million plus into dev efforts for sure. Not the happy ending I’m sure you all were expecting, but I view that as mission accomplished.

Core Value

Mega Minimal, they only focused on a single feature, executed well enough that people gave them some positive feedback, but not enough to justify automating everything.
Mega Mega. I’m not sure anyone who talked about the product saw the same core value, and there were several rewrites and shifts along the way.

Advantage Mega Minimal again

What about entrants into a crowded field

Well that is harder and the MVP tends to be less minimal, because the baseline expectations are just much higher. I still lean towards Mega Minimal having a better chance at getting users, since there is a non zero chance the Mega Mega MVP will never get finished. I still think the exercise in focusing on core value that makes your product not a me too, and even considering how you can find a niche in a crowded field instead of just being “better”, and your MVP can be that niche differentiator.

Internal users

Sometimes a good middle ground is considering getting lots of internal users if you’re really worried about bad experiences. This has it’s it’s definite downsides however, and you may not get diverse enough opinions. But it does give you some feedback while saving some face or bad experiences. I often think of the example of EC2 that was heavily used by Amazon, before being released to the world. That was a luxury Amazon had, where their customer base and their user base happened to be very similar, and they had bigger scale needs than any of their early customers, so the early internal feedback loop was a very strong signal.

Summary

In the end however you want to approach MVPs is up to you, and if you find success with a meatier MVP than I have please don’t let me push you away from what works. But if you are having trouble shipping and are getting pushed all the time to add one more feature to that MVP before releasing it, consider stepping back and asking is this really core value for the product? Do you already have your core value? if so, consider just releasing it.

Surprise Go is ok for me now

Thu, 13 Dec 2018 20:23:00 +0000

I’m surprised to say this, I am ok using Go now. It’s not my style but I am able to build most anything I want to with it, and the tooling around it continues to improve.

About 7 months ago I wrote about all the things I didn’t really care for in Go and now I either no longer am so bothered by it or things have improved.

Go Modules so far is a huge improvement over Dep and Glide for dependency management. It’s easy to setup, performant and eliminates the GOPATH silliness. I haven’t tried it yet with some of the goofier libraries that gave me problems in the past (k8s api for example) so the jury is out on that, but again pretty impressed. I now longer have to check in vendor to speed up builds. Lesson use Go Modules.

I pretty much stopped using channels for everything but shutdown signals and that fits my preferences pretty well, I use mutex and semaphores for my multithreaded code and feel no guilt about it. This cut out a lot of pain for me, and with the excellent race detector I feel really comfortable writing multi-threaded in Go now. Lesson, don’t use channels much.

Lack of generics still sometimes sucks but I usually implement some crappy casting with dynamic types if I need that. I’ve sorta made my piece with just writing more code, and am no longer so hung up. Lesson relax.

Error handling I’m still struggling with, I thought about using one of the error Wrap() libraries but an official one is in draft spec now, so I’ll wait on that. I now tend to have less nesting of functions as a result, this probably means longer functions than I like, but my code looks more “normal” now. This is a trade off I’m ok with. Lesson relax more.

I see the main virtue of Go now that it is very popular in the infrastructure space where I am and so it’s becoming the common tongue (largely replacing Python for those sorts of tasks). For this, honestly it’s about right. It’s easy to rip out command line tools and deploy binaries for every platform with no runtime install.

The community’s conservative attitude I sort of view as a feature now, in that there isn’t a bunch of different options that are popular and there is no arguing over what file format is used. This drove me up the wall initially, but I appreciate how much less time I spend on these things now.

So now I suspect Go will be my “last” programming language. It’s not the one I would have chosen, but where I am at in my career, where most of my dev work is automation and tooling it fits the bill pretty well.

Also equally important most of the people working with me didn’t have full time careers as developers or spend their time reading “Domain Driven Design” (amazing book) so adding in a bunched of nuanced stuff that maybe technically optimal but assumes the reader grasps all that nuance isn’t a good tradeoff for me.

So I think I sorta get it now. I’ll never be a cheerleader for the language but it definitely solves my problems well enough.

Collaboration vs. Critique

Fri, 18 May 2018 17:00:00 +0000

While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams. The people aspect of software development, more often than not, tends to be the most difficult part of software engineering. Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs. The nature of software development tends to follow an empirical process model rather than a defined process model. That is to say, software developers tend to be confronted with new problems every day and most of problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset. Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation. This means that team members are often required to work together to determine how to proceed. Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation. One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique.

To help illustrate what I’ve seen happen countless times both in catch-up design sessions and code reviews, consider the following two scenarios:

Scenario 1

Tom and Sally are both developers on a team maintaining a large-scale application. Tom takes the next task in the development queue which happens to have some complex processes that will need to be addressed. Being the good development team that they are, both Tom and Sally are aware of the requirements of the application (i.e. how the app needs to work from the user’s perspective), but they have deferred design-level discussions until the time of implementation. After Tom gets into the process a little, seeing that the problem is non-trivial, he pings Sally to help him brainstorm different approaches to solving the problem. Tom and Sally have been working together for over a year and have become accustomed to these sort of ad-hoc design sessions. As they begin discussing the problem, they each start tossing ideas out on the proverbial table resulting in multiple approaches to compare and contrast. The nature of the discussion is such that neither Tom nor Sally are embarrassed or offended when the other points out flaws in the design because there’s a sense of safety in their mutual understanding that this is a brainstorming session and that neither have thought in depth about the solutions being set froth yet. Tom throws out a couple of ideas, but ends up shooting them down himself as he uses Sally as a sounding board for the ideas. Sally does the same, but toward the end of the conversation suggests a slight alteration to one of Tom’s initial suggestions that they think may make it work after all. They end the session both with a sense that they’ve worked together to arrive at the best solution.

Scenario 2

Bill and Jake are developers on another team. They tend to work in a more siloed fashion, but they do rely upon one another for help from time to time and they are required to do code reviews prior to their code being merged into the main branch of development. Bill takes the next task in the development queue and spends the better part of an afternoon working out a solution with a basic working skeleton of the direction he’s going. The next day he decides that it might be good to have Jake take a look at the design to make him aware of the direction. Seeing where Bill’s design misses a few opportunities to make the implementation more adaptable to changes in the future, Jake points out where he would have done things differently. Bill acknowledges that Jake’s suggestions would be better and would have probably been just as easy to implement from the beginning, but inwardly he’s a bit disappointed that Jake didn’t like his design as-is and that he has to do some rework. In the end, Bill is left with a feeling of critique rather than collaboration.

Whether it’s a high-level UML diagram or working code, how one person tends to perceive feedback on the ideas comprising a potential solution has everything to do with timing. It can be the exact same feedback they would have received either way, but when the feedback occurs often makes a difference between whether it’s perceived as collaboration or critique. It’s all about when the conversation happens.

Testing Button Click in React with Jest

Mon, 07 May 2018 17:01:59 +0000

When building React applications you will most likely find yourself using Jest as your testing framework. Jest has some really, really cool features built in. But when you use Enzyme you can take your testing to the nest level.

One really cool feature is the ability to test click events via Enzyme to ensure your code responds as expected.

Before we get started you are going to want to make sure you have Jest and Enzyme installed in your application.

Installing Enzyme
Installing Jest

Sample code under test

What I would like to be able to do is pull the button out of my component and test the onClick event handler.

  
// Make sure you have your imports setup correctly
import React from 'react';
import { shallow } from 'enzyme';


it('When active link clicked, will push correct filter message', () => {
    let passedFilterType = '';
    const handleOnTotalsFilter = (filterType) => {
        passedFilterType = filterType;
    };
    const accounts = {};
    const wrapper = shallow(<MyComponent accounts={accounts} filterHeader="" onTotalsFilter={handleOnTotalsFilter} />);

    const button = wrapper.find('#archived-button');
    button.simulate('click');
    expect(passedFilterType).toBe(TotalsFilterType.archived);
});

Lets take a look at the test above

First we are going to create a callback (click handler) to catch the bubbled up values.
We use Enzyme to create our component MyComponent
We use the .find() on our wrapped component to find our <Button /> by id
After we get our button we can call .simulate(‘click’) which will act as a user clicking the button.
We can assert that the expected value bubbles up.

As you can see, simulating a click event of a rendered component is very straight forward, yet very powerful.

Till next time,

Lessons from a year of Golang

Mon, 07 May 2018 13:16:00 +0000

I’m hoping to share in a non-negative way help others avoid the pitfalls I ran into with my most recent work building infrastructure software on top of a Kubernetes using Go, it sounded like an awesome job at first but I ran into a lot of problems getting productive.

This isn’t meant to evaluate if you should pick up Go or tell you what you should think of it, this is strictly meant to help people out that are new to the language but experienced in Java, Python, Ruby, C#, etc and have read some basic Go getting started guide.

Dependency management

This is probably the feature most frequently talked about by newcomers to Go and with some justification, as dependency management been a rapidly shifting area that’s nothing like what experienced Java, C#, Ruby or Python developers are used to.

I’ll cut to the chase the default tool now is Dep all other tools I’ve used such as Glide or Godep they’re now deprecated in favor of Dep, and while Dep has advanced rapidly there are some problems you’ll eventually run into (or I did):

Dep hangs randomly and is slow, which is supposedly network traffic but it happens to everyone I know with tons of bandwidth. Regardless, I’d like an option to supply a timeout and report an error.
Versions and transitive depency conflicts can be a real breaking issue in Go still. So without shading or it’s equivalent two package depending on different versions of a given package can break your build, there are a number or proposals to fix this but we’re not there yet.
Dep has some goofy ways it resolves transitive dependencies and you may have to add explicit references to them in your Gopkg.toml file. You can see an example here under Updating dependencies – golang/dep.

My advice

Avoid hangs by checking in your dependencies directly into your source repository and just using the dependency tool (dep, godep, glide it doesn’t matter) for downloading dependencies.
Minimize transitive dependencies by keeping stuff small and using patterns like microservices when your dependency tree conflicts.

GOPATH

Something that takes some adjustment is you check out all your source code in one directory with one path (by default ~/go/src ) and include the path to the source tree to where you check out. Example:

I want to use a package I found on github called jim/awesomeness
I have to go to ~/go/src and mkdir -p github.com/jim
cd into that and clone the package.
When I reference the package in my source file it’ll be literally importing github.com/jim/awesomeness

A better guide to GOPATH and packages is here.

My advice

Don’t fight it, it’s actually not so bad once you embrace it.

Code structure

This is a hot topic and there are a few standards for the right way to structure you code from projects that do “file per class” to giant files with general concept names (think like types.go and net.go). Also if you’re used to using a lot of sub package you’re gonna to have issues with not being able to compile if for example you have two sub packages reference one another.

My Advice

In the end I was reasonably ok with something like the following:

myproject/bin for generated executables
myproject/cmd for command line code
myproject/pkg for code related to the package

Now whatever you do is fine, this was just a common idiom I saw, but it wasn’t remotely all projects. I also had some luck with just jamming everything into the top level of the package and keeping packages small (and making new packages for common code that is used in several places in the code base). If I ever return to using Go for any reason I will probably just jam everything into the top level directory.

Debugging

No debugger! There are some projects attempting to add one but Rob Pike finds them a crutch.

My Advice

Lots of unit tests and print statements.

No generics

Sorta self explanatory and it causes you a lot of pain when you’re used to reaching for these.

My advice

Look at the code generation support which uses pragmas, this is not exactly the same as having generics but if you have some code that has a lot of boiler plate without them this is a valid alternative. See this official Go Blog post for more details.

If you don’t want to use generation you really only have reflection left as a valid tool, which comes with all of it’s lack of speed and type safety.

Cross compiling

If you have certain features or dependencies you may find you cannot take advantage of one of Go’s better features cross compilation.

I ran into this when using the confluent-go/kafka library which depends on the C librdkafka library. It basically meant I had to do all my development in a Linux VM because almost all our packages relied on this.

My Advice

Avoid C dependencies at all costs.

Error handling

Go error handling is not exception base but return based, and it’s got a lot of common idioms around it:

myValue, err := doThing()
if err != nil {
	return -1, fmt.Errorf(“unable to doThing %v”, err)
  	}

Needless to say this can get very wordy when dealing with deeply nested exceptions or when you’re interacting a lot with external systems. It is definitely a mind shift if you’re used to the throwing exceptions wherever and have one single place to catch all exceptions where they’re handled appropriately.

My Advice

I’ll be honest I never totally made my peace with this. I had good training from experienced opensource contributors to major Go projects, read all the right blog posts, definitely felt like I’d heard enough from the community on why the current state of Go error handling was great in their opinions, but the lack of stack traces was a deal breaker for me.

On the positive side, I found Dave Cheney’s advice on error handling to be the most practical and he wrote a package containing a lot of that advice, we found it invaluable as it provided those stack traces we all missed but you had to remember to use it.

Summary

A lot of people really love Go and are very productive with it, I just was never one of those people and that’s ok. However, I think the advice in this post is reasonably sound and uncontroversial. So, if you find yourself needing to write some code it Go, give this guide a quick perusal and you’ll waste a lot less time than I did getting productive in developing applications in Go.

Raspberry Pi Kubernetes Cluster - Part 2

Thu, 03 May 2018 02:13:07 +0000

Howdy again.

Alright, my 8 port switch showed up so I was able to connect my raspberry 3B+ boards to my home network. I plugged it in with 6 1ft CAT5 cables I had in my catch-all box that all of us nerds have. I’d highly suggest flexible CAT 6 cables instead if you can get them, like here. I ordered them and they showed up before I finished this post, so I am using the CAT6 cables.

The IP addresses they will receive initialy from my home router via DHCP can be determined with nmap. Lets imagine my subnet is 192.168.1.0/24.

Once I got them on the network I did the following:

Install Raspberrian OS On SD Cards

You can get the Raspberry Pi Stretch Lite OS from here.

Then use the Etcher tool to install it to each of the 6 SD cards.

IMPORTANT

Before putting the cards into the Raspberry Pis you need to add a ssh folder to the root of the SD cards. This will allow you to ssh to each Raspberry Pi with default credentials (username: pi and password raspberry). Example: ssh pi@raspberry_pi_ip where raspberry_ip is obtained from the nmap command above.

Next post will be setting up kubernetes. Thank you for reading.

Cheers.

Raspberry Pi Kubernetes Cluster - Part 2 was originally published by Jason Meridth at Jason Meridth on May 02, 2018.

Multi-Environment Deployments with React

Tue, 10 Apr 2018 12:54:17 +0000

If you are using Create-React-App to scaffold your react application there is built in support for changing environment variables based on the NODE_ENV values, this is done by using .env files. In short this process works by having a .env, .env.production, .env.development set of files. When you run/build your application CRA will set the NODE_ENV value to either development or production and based on these values the correct .env file will be used.

This works great, when you have a simple deploy setup. But many times in enterprise level applications you need support for more than just 2 environments, many times it is 3-4 environments. Common logic would suggest that you can accomplish this via the built in mechanism by having additional .env files and changing the NODE_ENV value to the value you care about. However, CRA does not support this with doing an eject, which will eject all the default conventions and leave it to you to configure your React application. Maybe this is a good idea, but in my case ejecting was not something I wanted to do.

Because I did not want to do an eject I needed to find another solution, and after a fair amount of searching I found a solution that seems to work for me and my needs and is about the amount of effort as I wanted </article>
<article>
<h1>Raspberry Pi Kubernetes Cluster - Part 1</h1>
<p>Sat, 07 Apr 2018 14:01:00 +0000</p>
<content:encoded>
<p>Howdy</p>

<p>This is going to be the first post about my setup of a Raspberry Pi Kubernetes Cluster. I saw a post by <a href= Hart Hoover and it finally motivated me to purchase his “grocery list” and do this finally. I’ve been using Minikube for local Kubernetes testing but it doesn’t give you multi-host testing abilities. I’ve also been wanting to get deeper into my Raspberry Pi knowledge. Lots of learning and winning.

The items I bought were:

Here is the tweet when it all arrived:

I blame @hhoover ;). I will be building my @kubernetesio cluster once the 6pi case shows up next Wednesday. The extra pi is to upgrade my @RetroPieProject. Touch screen is an addition I want to try. Side project here I come. pic.twitter.com/EebIKbsCeH
— Jason Meridth (@jmeridth) March 31, 2018

I spent this morning finally putting it together.

Here is me getting started on the “dogbone case” to hold all of the Raspberry Pis:

The bottom and one layer above:

And the rest:

Different angles completed:

And connect the power:

Next post will be on getting the 6 sandisk cards ready and putting them in and watching the Raspberry Pis boot up and get a green light. Stay tuned.

Cheers.

Raspberry Pi Kubernetes Cluster - Part 1 was originally published by Jason Meridth at Jason Meridth on April 07, 2018.

Building AWS Infrastructure with Terraform: S3 Bucket Creation

Fri, 06 Apr 2018 14:28:49 +0000

If you are going to be working with any cloud provider it is highly suggested that you script out the creation/maintenance of your infrastructure. In the AWS word you can use the native CloudFormation solution, but honestly I find this painful and the docs very lacking. Personally, I prefer Terraform by Hashicorp. In my experience the simplicity and easy of use, not to mention the stellar documentation make this the product of choice.

This is the initial post in what I hope to be a series of post about how to use Terraform to setup/build AWS Infrastructure.

Terrform Documentation on S3 Creation -> Here

In this post I will cover 2 things

Basic bucket setup
Bucket setup as Static website

Setting up a basic bucket we can use the following

resource "aws_s3_bucket" "my-bucket" {
    bucket = "my-bucket"
    acl    = "private"

    tags {
        Any_Tag_Name = "Tag value for tracking"
    }    
}

When looking at the example above the only 2 values that are required are bucket and acl.

I have added the use of Tags to show you can add custom tags to your bucket

Another way to setup an S3 bucket is to act as a Static Web Host. Setting this up takes a bit more configuration, but not a ton.

resource "aws_s3_bucket" "my-website-bucket" {
    bucket = "my-website-bucket"
    acl    = "public-read"

    website {
        index_document = "index.html"
        error_document = "index.html"
    }    

    policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-website-bucket/*"
        }
    ]
}
    POLICY
    tags {
          Any_Tag_Name = "Tag value for tracking"
    }
}

The example above has 2 things that need to be pointed out.

The website settings. Make sure you setup the correct pages here for index/error

The Policy settings. Here I am using just basic policy. You can of course setup any policy here you want/need.

As you can see, setting up S3 buckets is very simple and straight forward.

*** Reminder: S3 bucket names MUST be globally unique ***

Till next time,

SSH - Too Many Authentication Failures

Wed, 28 Mar 2018 05:00:00 +0000

Problem

I started seeing this error recently and had brain farted on why.

Received disconnect from 123.123.132.132: Too many authentication failures for hostname

After a bit of googling it came back to me. This is because I’ve loaded too many keys into my ssh-agent locally (ssh-add). Why did you do that? Well, because it is easier than specifying the IdentityFile on the cli when trying to connect. But there is a threshhold. This is set by the ssh host by the MaxAuthTries setting in /etc/ssh/sshd_config. The default is 6.

Solution 1

Clean up the keys in your ssh-agent.

ssh-add -l lists all the keys you have in your ssh-agent ssh-add -d key deletes the key from your ssh-agent

Solution 2

You can solve this on the command line like this:

ssh -o IdentitiesOnly=yes -i ~/.ssh/example_rsa foo.example.com

What is IdentitiesOnly? Explained in Solution 3 below.

Solution 3 (best)

Specifiy, explicitly, which key goes to which host(s) in your .ssh/config file.

You need to configure which key (“IdentityFile”) goes with which domain (or host). You also want to handle the case when the specified key doesn’t work, which would usually be because the public key isn’t in ~/.ssh/authorized_keys on the server. The default is for SSH to then try any other keys it has access to, which takes us back to too many attempts. Setting “IdentitiesOnly” to “yes” tells SSH to only try the specified key and, if that fails, fall through to password authentication (presuming the server allows it).

Your ~/.ssh/config would look like:

Host *.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myhost
Host secure.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/mysecurehost_rsa
Host *.myotherhost.domain
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myotherhost_rsa

Host is the host the key can connect to IdentitiesOnly means only to try this specific key to connect, no others IdentityFile is the path to the key

You can try multiple keys if needed

Host *.myhost.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/myhost_rsa
  IdentityFile ~/.ssh/myhost_dsa

Hope this helps someone else.

Cheers!

SSH - Too Many Authentication Failures was originally published by Jason Meridth at Jason Meridth on March 28, 2018.

Clear DNS Cache In Chrome

Tue, 27 Mar 2018 20:42:00 +0000

I’m blogging this because I keep forgetting how to do it. Yeah, yeah, I can google it. I run this blog so I know it is always available…..anywho.

Go To:

chrome://net-internals/#dns

Click “Clear host cache” button

Hope this helps someone else.

Cheers.

Clear DNS Cache In Chrome was originally published by Jason Meridth at Jason Meridth on March 27, 2018.

Create Docker Container from Errored Container

Mon, 26 Mar 2018 03:31:00 +0000

When I’m trying to “dockerize” an applciation I usually have to work through some wonkiness.

To diagnose a container that has errored out, I, obviously, look at the logs via docker logs -f [container_name]. That is sometimes helpful. It will, at minimum tell me where I need to focus on the new container I’m going to create.

Pre-requisites to being able to build a diagnosis container:

You need to use CMD, not ENTRYPOINT in the Dockerfile
- with CMD you’ll be able to start a shell, with ENTRYPOINT your diagnosis container will just keep trying to run that

To create a diagnosis container, do the following:

Check your failed container ID by docker ps -a
Create docker image form the container with docker commit
- example: docker commit -m "diagnosis" [failed container id]
Check the newly create docker image ID by docker images
docker run -it [new container image id] sh
- this takes you into a container immediately after the error occurred.

Hope this helps someone else.

Cheers.

Create Docker Container from Errored Container was originally published by Jason Meridth at Jason Meridth on March 25, 2018.

Log Early, Log Often… Saved my butt today

Wed, 21 Mar 2018 13:16:03 +0000

In a prior posting (AWS Lambda:Log Early Log often, Log EVERYTHING) I wrote about the virtues and value about having really in depth logging, especially when working with cloud services. Well today this logging saved my ASS a ton of detective work.

Little Background
I have a background job (Lambda that is called on a schedule) to create/update data cache in a DynamoDB table. Basically this job will pull data from one data source and attempt to push it as create/update/delete to our Dynamo table.

Today when I was running our application I noticed things were not loading right, in fact I had javascript errors because of null reference errors. I knew that the issue had to be in our data, but was not sure what was wrong. If I had not had a ton of logging (debug and info) I would have had to run our code locally and step though/debug code for hundreds of items of data.

However, because of in depth logging I was able to quickly go to CloudWatch and filter on a few key words and narrow hundreds/thousands of log entries down to 5. Once I had these 5 entries I was able to expand a few of those entries and found the error within seconds.

Total time to find the error was less than 5 minutes and I never opened a code editor or stepped into code.

The moral of this story, because I log everything, including data (no PII of course) I was able to quickly find the source of the error. Now to fix the code….

Till next time,

AWS Lambda: Log early, Log often, Log EVERYTHING

Tue, 06 Mar 2018 14:00:58 +0000

In the world of building client/server applications logs are important. They are helpful when trying to see what is going on in your application. I have always held the belief that your logs need to be detailed enough to allow you to determine the WHAT and WHERE without even looking at the code.

But lets be honest, in most cases when building client/server applications logs are an afterthought. Often this is because you can pretty easily (in most cases) debug your application and step through the code.

When building a serverless applications with technologies like AWS Lambda functions (holds true for Azure Functions as well) your logging game really needs to step up.

The reason for this is that you cannot really debug your Lambda in the wild (you can to some degree locally with AWS SAM or the Serverless framework). Because of this you need produce detailed enough logs to allow you to easily determine the WHAT and WHERE.

When I build my serverless functions I have a few guidelines I follow

Info Log calls to methods, output argument data (make sure no PII/PHI)
Error Log any failures (in try/catch or .catch for promises)
Debug Log any critical decision points
Info Log exit calls at top level methods

I also like to setup a simple and consistent format for my logs. The example I follow for my Lambda logs is as seen below

timestamp: [logLevel] : [Class.Method] - message {data points}

I have found that if I follow these general guidelines the pain of determine failure points in serverless environments is heavily reduced.

Till next time,

Sinon Error: Attempted to wrap undefined property ‘XYZ as function

Tue, 27 Feb 2018 13:45:29 +0000

I ran into a fun little error recently when working on a ReactJs application. In my application I was using SinonJs to setup some spies on a method, I wanted to capture the input arguments for verification. However, when I ran my test I received the following error.

Attempted to wrap undefined property handlOnAccountFilter as function

My method under test is setup as such

handleOnAccountFilter = (filterModel) => {
    // logic here
}

I was using the above syntax is the proposed class property feature, which will automatically bind the this context of the class to my method.

My sinon spy is setup as such

let handleOnAccountFilterSpy = null;
beforeEach(() => {
    handleOnAccountFilterSpy = sinon.spy(AccountsListingPage.prototype, 'handleOnAccountFilter');
});

afterEach(() => {
     handleOnAccountFilterSpy.restore();
});

Everything looked right, but I was still getting this error. It turns out that this error is due in part in the way that the Class Property feature implements the handlOnAccountFilter. When you use this feature the method/property is added to the class as an instance method/property, not as a prototype method/property. This means that sinon is not able to gain access to it prior to creating an instance of the class.

To solve my issue I had to make a change in the implementation to the following

handleOnAccountFilter(filterModel) {
    // logic here
}

After make the above change I needed to determine how I wanted to bind this to my method (Cory show 5 ways to do this here). I chose to bind this inside the constructor as below

constructor(props){
    super(props);

    this.handleOnAccountFilter = this.handleOnAccountFilter.bind(this);
}

I am not a huge fan of having to do this (pun intended), but oh well. This solved my issues.

Till next time

Ensuring componentDidMount is not called in Unit Tests

Thu, 22 Feb 2018 19:45:53 +0000

If you are building a ReactJs you will often times implement componentDidMount on your components. This is very handy at runtime, but can pose an issue for unit tests.

If you are building tests for your React app you are very likely using enzyme to create instances of your component. The issue is that when enzyme creates the component it invokes the lifecyle methods, like componentDidMount. Sometimes we do not want this to be called, but how to suppress this?

I have found 2 different ways to suppress/mock componentDidMount.

Method one is to redefine componentDidMount on your component for your tests. This could have interesting side effects so use with caution.

    describe('UsefullNameHere', () => {
        
        beforeAll(() => {
            YourComponent.prototype.componentDidMount = () => {
                // can omit or add custom logic
            };
        });
    });

Basically above I am just redefining the componentDidMount method on my component. This works and allows you to have custom logic. Be aware that when doing above you will have changed the implementation for your component for the lifetime of your test session.

Another solution is to use a mocking framework like SinonJs. With Sinon you can stub out the componentDidMount implementation as seen below

    describe('UsefullNameHere', () => {
        
        let componentDidMountStub = null;
        beforeAll(() => {
            componentDidMountStub = sinon.stub(YourComponent.prototype, 'componentDidMount').callsFake(function() {
                 // can omit or add custom logic
            });
        });

        afterAll(() => {
            componentDidMountStub.restore();
        });
    });

Above I am using .stub to redefine the method. I also added .callsFake() but this can be omitted if you just want to ignore the call. You will want to make sure you restore your stub via the afterAll, otherwise you will have stubbed out the call for the lifetime of your test session.

Till next time,

Los Techies Welcomes Derik Whittaker

Wed, 21 Feb 2018 11:00:00 +0000

Los Techies would like to introduce, and extend a welcome to Derik Whittaker. Derik is a C# MVP, member of the AspInsiders group, community speaker, and Pluralsight author. Derik was previously a contributor at CodeBetter.com. Welcome, Derik!

Ditch the Repository Pattern Already

Tue, 20 Feb 2018 21:00:00 +0000

One pattern that still seems particularly common among .Net developers is the Repository pattern. I began using this pattern with NHibernate around 2006 and only abandoned its use a few years ago.

I had read several articles over the years advocating abandoning the Repository pattern in favor of other suggested approaches which served as a pebble in my shoe for a few years, but there were a few design principles whose application seemed to keep motivating me to use the pattern. It wasn’t until a change of tooling and a shift in thinking about how these principles should be applied that I finally felt comfortable ditching the use of repositories, so I thought I’d recount my journey to provide some food for thought for those who still feel compelled to use the pattern.

Mental Obstacle 1: Testing Isolation

What I remember being the biggest barrier to moving away from the use of repositories was writing tests for components which interacted with the database. About a year or so before I actually abandoned use of the pattern, I remember trying to stub out a class derived from Entity Framework’s DbContext after reading an anti-repository blog post. I don’t remember the details now, but I remember it being painful and even exploring use of a 3rd-party library designed to help write tests for components dependent upon Entity Framework. I gave up after a while, concluding it just wasn’t worth the effort. It wasn’t as if my previous approach was pain-free, as at that point I was accustomed to stubbing out particularly complex repository method calls, but as with many things we often don’t notice friction to which we’ve become accustomed for one reason or another. I had assumed that doing all that work to stub out my repositories was what I should be doing.

Another principle that I picked up from somewhere (maybe the big xUnit Test Patterns book? … I don’t remember) that seemed to keep me bound to my repositories was that you shouldn’t write tests that depend upon dependencies you don’t own. I believed at the time that I should be writing tests for Application Layer services (which later morphed into discrete dispatched command handlers) and the idea of stubbing out either NHIbernate or Entity Framework violated my sensibilities.

Mental Obstacle 2: The Dependency Inversion Principle Adherence

The Dependency Inversion Principle seems to be a source of confusion for many which stems in part from the similarity of wording with the practice of Dependency Injection as well as from the fact that the pattern’s formal definition reflects the platform from whence the principle was conceived (i.e. C++). One might say that the abstract definition of the Dependency Inversion Principle was too dependent upon the details of its origin (ba dum tss). I’ve written about the principle a few times (perhaps my most succinct being this Stack Overflow answer), but put simply, the Dependency Inversion Principle has at its primary goal the decoupling of the portions of your application which define policy from the portions which define implementation. That is to say, this principle seeks to keep the portions of your application which govern what your application does (e.g. workflow, business logic, etc.) from being tightly coupled to the portions of your application which govern the low level details of how it gets done (e.g. persistence to an Sql Server database, use of Redis for caching, etc.).

A good example of a violation of this principle, which I recall from my NHibernate days, was that once upon a time NHibernate was tightly coupled to log4net. This was later corrected, but at one time the NHibernate assembly had a hard dependency on log4net. You could use a different logging library for your own code if you wanted, and you could use binding redirects to use a different version of log4net if you wanted, but at one time if you had a dependency on NHibernate then you had to deploy the log4net library. I think this went unnoticed by many due to the fact that most developers who used NHibernate also used log4net.

When I first learned about the principle, I immediately recognized that it seemed to have limited advertized value for most business applications in light of what Udi Dahan labeled The Fallacy Of ReUse. That is to say, properly understood, the Dependency Inversion Principle has as its primary goal the reuse of components and keeping those components decoupled from dependencies which would keep them from being easily reused with other implementation components, but your application and business logic isn’t something that is likely to ever be reused in a different context. The take away from that is basically that the advertized value of adhering to the Dependency Inversion Principle is really more applicable to libraries like NHibernate, Automapper, etc. and not so much to that workflow your team built for Acme Inc.’s distribution system. Nevertheless, the Dependency Inversion Principle had a practical value of implementing an architecture style Jeffrey Palermo labeled the Onion Architecture. Specifically, in contrast to traditional 3-layered architecture models where UI, Business, and Data Access layers precluded using something like Data Access Logic Components to encapsulate an ORM to map data directly to entities within the Business Layer, inverting the dependencies between the Business Layer and the Data Access layer provided the ability for the application to interact with the database while also seemingly abstracting away the details of the data access technology used.

While I always saw the fallacy in strictly trying to apply the Dependency Inversion Principle to invert the implementation details of how I got my data from my application layer so that I’d someday be able to use the application in a completely different context, it seemed the academically astute and in vogue way of doing Domain-driven Design at the time, seemed consistent with the GoF’s advice to program to an interface rather than an implementation, and provided an easier way to write isolation tests than trying to partially stub out ORM types.

The Catalyst

For the longest time, I resisted using Entity Framework. I had become fairly proficient at using NHibernate and I just saw it as plain stupid to use a framework that was years behind NHibernate in features and maturity, especially when it had such a steep learning curve. A combination of things happened, though. A lot of the NHibernate supporters (like many within the Alt.Net crowd) moved on to other platforms like Ruby and Node; anything with Microsoft’s name on it eventually seems to gain market share whether it’s better or not; and Entity Framework eventually did seem to mostly catch up with NHibernate in features, and surpassed it in some areas. So, eventually I found it impossible to avoid using Entity Framework which led to me trying to apply the same patterns I’d used before with this newer-to-me framework.

To be honest, everything mostly worked, especially for the really simple stuff. Eventually, though, I began to see little ways I had to modify my abstraction to accommodate differences in how Entity Framework did things from how NHibernate did things. What I discovered was that, while my repositories allowed my application code to be physically decoupled from the ORM, the way I was using the repositories was in small ways semantically coupled to the framework. I wish I had kept some sort of record every time I ran into something, as the only real thing I can recall now were motivations with certain design approaches to expose the SaveChanges method for Unit of Work implementations I don’t want to make more of the semantic coupling argument against repositories than it’s worth, but observing little places where my abstractions were leaking, combined with the pebble in my shoe of developers who I felt were far better than me were saying I shouldn’t use them lead me to begin rethinking things.

More Effective Testing Strategies

It was actually a few years before I stopped using repositories that I stopped stubbing out repositories. Around 2010, I learned that you can use Test-Driven Development to achieve 100% test coverage for the code for which you’re responsible, but when you plug your code in for the first time with that team that wasn’t designing to the same specification and not writing any tests at all that things may not work. It was then that I got turned on to Acceptance Test Driven Development. What I found was that writing high-level subcutaneous tests (i.e. skipping the UI layer, but otherwise end-to-end) was overall easier, was possible to align with acceptance criteria contained within a user story, provided more assurance that everything worked as a whole, and was easier to get teams on board with. Later on, I surmised that I really shouldn’t have been writing isolation tests for components which, for the most part, are just specialized facades anyway. All an isolation test for a facade really says is “did I delegate this operation correctly” and if you’re not careful you can end up just writing a whole bunch of tests that basically just validate whether you correctly configured your mocking library.

So, by the time I started rethinking my use of repositories, I had long since stopped using them for test isolation.

Taking the Plunge

It was actually about a year after I had become convinced that repositories were unnecessary, useless abstractions that I started working with a new codebase I had the opportunity to steer. Once I eliminated them from the equation, everything got so much simpler. Having been repository-free for about two years now, I think I’d have a hard time joining a team that had an affinity for them.

Conclusion

If you’re still using repositories and you don’t have some other hangup you still need to get over like writing unit tests for your controllers or application services then give the repository-free lifestyle a try. I bet you’ll love it.

Using Manual Mocks to test the AWS SDK with Jest

Tue, 20 Feb 2018 13:56:45 +0000

Anytime you build Node applications it is highly suggested that your cover your code with tests. When your code interacts with 3rd party API’s such as AWS you will most certainly want to mock/stub your calls in order to prevent external calls (if you actually want to do external calls, these are called integration tests not unit tests.

If you are using Jest, one solution is utilize the built in support for manual mocks. I have found the usage of manual mocks invaluable while testing 3rd party API’s such as the AWS. Keep in mind just because I am using manual mocks this will remove the need for using libraries like SinonJs (a JavaScript framework for creating stubs/mocks/spies).

The way that manual mocks work in Jest is as follows (from the Jest website’s documentation).

Manual mocks are defined by writing a module in a __mocks__/ subdirectory immediately adjacent to the module. For example, to mock a module called user in the models directory, create a file called user.js and put it in the models/__mocks__ directory. Note that the __mocks__ folder is case-sensitive, so naming the directory __MOCKS__ will break on some systems. If the module you are mocking is a node module (eg: fs), the mock should be placed in the __mocks__ directory adjacent to node_modules (unless you configured roots to point to a folder other than the project root).

In my case I want to mock out the usage of the AWS-SDK for Node.

To do this I created a __mocks__ folder at the root of my solution. I then created a aws-sdk.js file inside this folder.

Now that I have my mocks folder created with a aws-sdk.js file I am able to consume my manual mock in my jest test by simply referencing the aws-sdk via a require('aws-sdk') command.

const AWS = require('./aws-sdk');

With declaration of AWS above my code is able to a use the NPM package during normal usage, or my aws-sdk.js mock when running under the Jest context.

Below is a small sample of the code I have inside my aws-sdk.js file for my manual mock.

const stubs = require('./aws-stubs');
const AWS = {};

// This here is to allow/prevent runtime errors if you are using 
//   AWS.config to do some runtime configuration of the library.
// If you do not need any runtime configuration you can omit this.
AWS.config = {
    setPromisesDependency: (arg) => {}
};

AWS.S3 = function() {

}

// Because I care about using the S3 service's which are part of the SDK
//  I need to setup the correct identifier.
// 
AWS.S3.prototype = {
    ...AWS.S3.prototype,

    // Stub for the listObjectsV2 method in the sdk
    listObjectsV2(params){
        const stubPromise = new Promise((resolve, reject) => {
            // pulling in stub data from an external file to remove the noise
            // from this file.  See the top line for how to pull this in
            resolve(stubs.listObjects);
        });

        return {
            promise: () => {
                return stubPromise;
            }
        }
    }
};

// Export my AWS function so it can be referenced via requires
module.exports = AWS;

A few things to point out in the code above.

I chose to use the promises implementation of the listObjectsV2. Because of this I need to return a promise method as my result on my listObjectsV2 function. I am sure there are other ways to accomplish this, but this worked and is pretty easy.
My function is returning stub data, but this data is described in a separate file called aws-stubs.js which sites along side of my aws-sdk.js file. I went this route to remove the noise of having the stub data inside my aws-adk file. You can see a full example of this here.

Now that I have everything setup my tests will no longer attempt to hit the actually aws-sdk, but when running in non-test mode they will.

Till next time,

Configure Visual Studio Code to debug Jest Tests

Fri, 16 Feb 2018 21:33:03 +0000

If you have not given Visual Studio Code a spin you really should, especially if you are doing web/javascript/Node development.

One super awesome feature of VS Code is the ability to easily configure the ability to debug your Jest (should work just fine with other JavaScript testing frameworks) tests. I have found that most of the time I do not need to actually step into the debugger when writing tests, but there are times that using console.log is just too much friction and I want to step into the debugger.

So how do we configure VS Code?

First you will need to install the Jest-Cli NPM package (I am assuming you already have Jest setup to run your tests, if you do not please read the Getting-Started docs). If you fail to do this step you will get the following error in Code when you try to run the debugger.

After you have Jest-Cli installed you will need to configure VS Code for debugging. To do this open up the configuration by clicking Debug -> Open Configurations. This will open up a file called launch.json.

Once launch.json is open add the following configuration

        {
            "name": "Jest Tests",
            "type": "node",
            "request": "launch",
            "program": "${workspaceRoot}/node_modules/jest-cli/bin/jest.js",
            "stopOnEntry": false,
            "args": ["--runInBand"],
            "cwd": "${workspaceRoot}",
            "preLaunchTask": null,
            "runtimeExecutable": null,
            "runtimeArgs": [
                "--nolazy"
            ],
            "env": {
                "NODE_ENV": "development"
            },
            "console": "internalConsole",
            "sourceMaps": false,
            "outFiles": []
        }

Here is a gist of a working launch.json file.

After you save the file you are almost ready to start your debugging.

Before you can debug you will want to open the debug menu (the bug icon on the left toolbar). This will show a drop down menu with different configurations. Make sure ‘Jest Test’ is selected.

If you have this setup correctly you should be able to set breakpoints and hit F5.

Till next time,

On Migrating Los Techies to Github Pages

Fri, 16 Feb 2018 20:00:00 +0000

We recently migrated Los Techies from a multi-site installation of WordPress to Github Pages, so I thought I’d share some of the more unique portions of the process. For a straightforward guide on migrating from WordPress to Github Pages, Tomomi Imura has published an excellent guide available here that covers exporting content, setting up a new Jekyll site (what Github Pages uses as its static site engine), porting the comments, and DNS configuration. The purpose of this post is really just to cover some of the unique aspects that related to our particular installation.

Step 1: Exporting Content

Having recently migrated my personal blog from WordPress to Github Pages using the aforementioned guide, I thought the process of doing the same for Los Techies would be relatively easy. Unfortunately, due to the fact that we had a woefully out-of-date installation of WordPress, migrating Los Techies proved to be a bit problematic. First, the WordPress to Jekyll Exporter plugin wasn’t compatible with our version of WordPress. Additionally, our installation of WordPress couldn’t be upgraded in place for various reasons. As a result, I ended up taking the rather labor-intensive path of exporting each author’s content using the default WordPress XML export and then, for each author, importing into an up-to-date installation of WordPress using the hosting site with which I previously hosting my personal blog, exporting the posts using the Jekyll Exporter plugin, and then deleting the posts in preparation for the next iteration. This resulted in a collection of zipped, mostly ready posts for each author.

Step 2: Configuring Authors

Our previous platform utilized the multi-site features of WordPress to facilitate a single site with multiple contributors. By default, Jekyll looks for content within a special folder in the root of the site named _posts, but there are several issues with trying to represent multiple contributors within the _posts folder. Fortunately Jekyll has a feature called Collections which allows you to set up groups of posts which can each have their own associated configuration properties. Once each of the author’s posts were copied to corresponding collection folders, a series of scripts were written to create author-specific index.html, archive.html, and tags.html files which are used by a custom post layout. Additionally, due to the way the WordPress content was exported, the permalinks generated for each post did not reflect the author’s subdirectory, so another script was written to strip out all the generated permalinks.

Step 3: Correcting Liquid Errors

Jekyll uses a language called Liquid as its templating engine. Once all the content was in place, all posts which contained double curly braces were interpreted as Liquid commands which ended up breaking the build process. For that, each offending post had to be edited to wrap the content in Liquid directives {% raw %} … {% endraw %} to keep the content from being interpreted by the Liquid parser. Additionally, there were a few other odd things which were causing issues (such as posts with non-breaking space characters) for which more scripts were written to modify the posts to non-offending content.

Step 4: Enabling Disqus

The next step was to get Disqus comments working for the posts. By default, Disqus will use the page URL as the page identifier, so as long as the paths match then enabling Disqus should just work. The WordPress Disqus plugin we were using utilized a unique post id and guid as the Disqus page identifier, so the Disqus javascript had to be configured to use these properties. These values were preserved by the Jekyll exporter, but unfortunately the generated id property in the Jekyll front matter was getting internally overridden by Jekyll so another script had to be written to modify all the posts to rename the properties used for these values. Properties were added to the Collection configuration in the main _config.yml to designate the Disqus shortname for each author and allow people to toggle whether disqus was enabled or disabled for their posts.

Step 5: Converting Gists

Many authors at Los Techies used a Gist WordPress plugin to embed code samples within their posts. Github Pages supports a jekyll-gist plugin, so another script was written to modify all the posts to use Liquid syntax to denote the gists. This mostly worked, but there were still a number of posts which had to be manually edited to deal with different ways people were denoting their gists. In retrospect, it would have been better to use JavaScript rather than the Jekyll gist plugin due to the size of the Los Techies site. Every plugin you use adds time to the overall build process which can become problematic as we’ll touch on next.

Step 6: Excessive Build-time Mitigation

The first iteration of the conversion used the Liquid syntax for generating the sidebar content which lists recent site-wide posts, recent author-specific posts, and the list of contributing authors. This resulted in extremely long build times, but it worked and who cares once the site is rendered, right? Well, what I found out was that Github has a hard cut off of 10 minutes for Jekyll site builds. If your site doesn’t build within 10 minutes, the process gets killed. At first I thought “Oh no! After all this effort, Github just isn’t going to support a site our size!” I then realized that rather than having every page loop over all the content, I could create a Jekyll template to generate JSON content one time and then use JavaScript to retrieve the content and dynamically generate the sidebar DOM elements. This sped up the build significantly, taking the build from close to a half-hour to just a few minutes.

Step 8: Converting WordPress Uploaded Content

Another headache that presented itself is how WordPress represented uploaded content. Everything that anyone had ever uploaded to the site for images and downloads used within their posts were stored in a cryptic folder structure. Each folder had to be interrogated to see which files contained therein matched what author, the folder structure had to be reworked to accommodate the nature of the Jekyll site, and more scripts had to be written to edit everyone’s posts to change paths to the new content. Of course, the scripts only worked for about 95% of the posts, a number of posts had to be edited manually to fix things like non-printable characters being used in file names, etc.

Step 9: Handling Redirects

The final step to get the initial version of the conversion complete was to handle redirects which were formally being handled by .httpacess. The Los Techies site started off using Community Server prior to migrating to WordPress and redirects were set up using .httpaccess to maintain the paths to all the previous content locations. Github Pages doesn’t support .httpaccess, but it does support a Jekyll redirect plugin. Unfortunately, it requires adding a redirect property to each post requiring a redirect and we had several thousand, so I had to write another script to read the .httpaccess file and figure out which post went with each line. Another unfortunate aspect of using the Jekyll redirect plugin is that it adds overhead to the build time which, as discussed earlier, can become an issue.

Step 10: Enabling Aggregation

Once the conversion was complete, I decided to dedicate some time to figuring out how we might be able to add the ability to aggregate posts from external feeds. The first step to this was finding a service that could aggregate feeds together. You might think there would be a number of things that do this, and while I did find at least a half-dozen services, there were only a couple I found that allowed you to maintain a single feed and add/remove new feeds while preserving the aggregated feed. Most seemed to only allow you to do a one-time aggregation. For this I settled on a site named feed.informer.com. Next, I replaced the landing page with JavaScript that dynamically built the site from the aggregated feed along with replacing the recent author posts section that did the same and a special external template capable of making an individual post look like it’s actually hosted on Los Techies. The final result was a site that displays a mixture of local content along with aggregated content.

Conclusion

Overall, the conversion was way more work than I anticipated, but I believe worth the effort. The site is now much faster than it used to be and we aren’t having to pay a hosting service to host our site.

Going Async with Node AWS SDK with Express

Tue, 13 Feb 2018 13:00:30 +0000

When building applications in Node/Express you will quickly come to realize that everything is done asynchronously . But how you accomplish these tasks async can vary. The ‘old school’ way was to use call backs, which often led to callback hell. Than came along Promises which we thought was going to solve all the worlds problems, turned out they helped, but did not solve everything. Finally in Node 8.0 (ok, you could use them in Node 7.6) the support for async/await was introduced and this really has cleaned up and enhanced the readability of your code.

Having the ability to use async/await is great, and is supported out of the box w/ Express. But what do you do when you using a library which still wants to use promises or callbacks? The case in point for this article is AWS Node SDK.

By default if you read through the AWS SDK documentation the examples lead you to believe that you need to use callbacks when implementing the SDK. Well this can really lead to some nasty code in the world of Node/Express. However, as of v2.3.0 of the AWS SDK there is support for Promises. This is much cleaner than using callbacks, but still poses a bit of an issue if you want to use async/await in your Express routes.

However, with a bit of work you can get your promise based AWS calls to play nicely with your async/await based Express routes. Lets take a look at how we can accomplish this.

Before you get started I am going to make a few assumptions.

You already have a Node/Express application setup
You already have the AWS SDK for Node installed, if not read here

The first thing we are going to need to do is add reference to our AWS SDK and configure it to use promises.

const AWS = require('aws-sdk');
AWS.config.setPromisesDependency(null);

After we have our SDK configured we can implement our route handler. In my example here I am placing all the logic inside my handler. In a real code base I would suggest better deconstruction of this code into smaller parts.

const express = require('express');
const router = express.Router();
const s3 = new AWS.S3();

router.get('/myRoute', async (req, res) => {

    const controller = new sitesController();
    const params = req.params;
    const params = { 
       Bucket: "bucket_name_here"
    };

    let results = {};
    var listPromise = s3.listObjects(params).promise();
    listPromise.then((data) => {
        results = data;
    });

    await Promise.all([listPromise]);

    res.json({data: results })
})

module.exports = router;

Lets review the code above and call out a few important items.

The first thing to notice is the addition of the async keyword in my route handler. This is what allows us to use async/await in Node/Express.

The next thing to look at is how I am calling the s3.listObjects. Notice I am NOT providing a callback to the method, but instead I am chaining with .promise(). This is what instructs the SDK to use promises vs callbacks. Once I have my callback I chain a ‘then’ in order to handle my response.

The last thing to pay attention to is the line with await Promise.All([listPromise]); This is the magic forces our route handler to not return prior to the resolution of all of our Promises. Without this your call would exit prior to the listObjects call completing.

Finally, we are simply returning our data from the listObjects call via res.json call.

That’s it, pretty straight forward, once you learn that the AWS SDK supports something other than callbacks.

Till next time,

Unable To Access Mysql With Root and No Password After New Install On Ubuntu

Wed, 31 Jan 2018 00:13:00 +0000

This bit me in the rear end again today. Had to reinstall mysql-server-5.7 for other reasons.

You just installed mysql-server locally for your development environment on a recent version of Ubuntu (I have 17.10 artful installed). You did it with a blank password for root user. You type mysql -u root and you see Access denied for user 'root'@'localhost'.

Issue: Because you chose to not have a password for the root user, the auth_plugin for my MySQL defaulted to auth_socket. That means if you type sudo mysql -u root you will get in. If you don’t, then this is NOT the fix for you.

Solution: Change the auth_plugin to mysql_native_password so that you can use the root user in the database.

$ sudo mysql -u root

mysql> USE mysql;
mysql> UPDATE user SET plugin='mysql_native_password' WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> exit;

$ sudo systemctl restart mysql
$ sudo systemctl status mysql

NB ALWAYS set a password for mysql-server in staging/production.

Cheers.

Unable To Access Mysql With Root and No Password After New Install On Ubuntu was originally published by Jason Meridth at Jason Meridth on January 30, 2018.

New Job

Mon, 08 Jan 2018 18:13:00 +0000

Well, it is a new year and I’ve started a new job. I am now a Senior Software Engineer at True Link Financial.

After interviewing with the co-founders Kai and Claire and their team, I knew I wanted to work here.

TL;DR: True Link: We give elderly and disable (really, anyone) back their financial freedom where they may not usually have it.

Longer Version: Imagine you have an elderly family member who may start showing signs of dimensia. You can give them a True Link card and administer their card. You link it to their bank account or another source of funding and you can set limitations on when, where and how the card can be used. The family member feels freedom by not having to continually ask for money but is also protected by scammers and non-friendly people (yep, they exist).

The customer service team, the marketing team, the product team, the engineering team and everyone else at True Link are amazing.

For any nerd readers, the tech stack is currently Rails, React, AWS, Ansible. We’ll be introducing Docker and Kubernetes soon hopefully, but always ensuring the right tools for the right job.

Looking forward to 2018.

Cheers.

New Job was originally published by Jason Meridth at Jason Meridth on January 08, 2018.

Hello, React! – A Beginner’s Setup Tutorial

Thu, 25 May 2017 08:00:32 +0000

React has been around for a few years now and there are quite a few tutorials available. Unfortunately, many are outdated, overly complex, or gloss over configuration for getting started. Tutorials which side-step configuration by using jsfiddle or code generator options are great when you’re wanting to just focus on the framework features itself, but many tutorials leave beginners struggling to piece things together when you’re ready to create a simple react application from scratch. This tutorial is intended to help beginners get up and going with React by manually walking through a minimal setup process.

A Simple Tutorial

This tutorial is merely intended to help walk you through the steps to getting a simple React example up and running. When you’re ready to dive into actually learning the React framework, a great list of tutorials can be found here.

There are a several build, transpiler, or bundling tools from which to select when working with React. For this tutorial, we’ll be using be using Node, NPM, Webpack, and Babel.

Step 1: Install Node

Download and install Node for your target platform. Node distributions can be obtained here.

Step 2: Create a Project Folder

From a command line prompt, create a folder where you plan to develop your example.

$> mkdir hello-react

Step 3: Initialize Project

Change directory into the example folder and use the Node Package Manager (npm) to initialize the project:

$> cd hello-react
$> npm init --yes

This results in the creation of a package.json file. While not technically necessary for this example, creating this file will allow us to persist our packaging and runtime dependencies.

Step 4: Install React

React is broken up into a core framework package and a package related to rendering to the Document Object Model (DOM).

From the hello-react folder, run the following command to install these packages and add them to your package.json file:

$> npm install --save-dev react react-dom

Step 5: Install Babel

Babel is a transpiler, which is to say it’s a tool from converting one language or language version to another. In our case, we’ll be converting EcmaScript 2015 to EcmaScript 5.

From the hello-react folder, run the following command to install babel:

$> npm install --save-dev babel-core

Step 6: Install Webpack

Webpack is a module bundler. We’ll be using it to package all of our scripts into a single script we’ll include in our example Web page.

From the hello-react folder, run the following command to install webpack globally:

$> npm install webpack --global

Step 7: Install Babel Loader

Babel loader is a Webpack plugin for using Babel to transpile scripts during the bundling process.

From the hello-react folder, run the following command to install babel loader:

$> npm install --save-dev babel-loader

Step 8: Install Babel Presets

Babel presets are collections of plugins needed to support a given feature. For example, the latest version of babel-preset-es2015 at the time this writing will install 24 plugins which enables Babel to transpile ECMAScript 2015 to ECMAScript 5. We’ll be using presets for ES2015 as well as presets for React. The React presets are primarily needed for processing of JSX.

From the hello-react folder, run the following command to install the babel presets for both ES2015 and React:

$> npm install --save-dev babel-preset-es2015 babel-preset-react

Step 9: Configure Babel

In order to tell Babel which presets we want to use when transpiling our scripts, we need to provide a babel config file.

Within the hello-react folder, create a file named .babelrc with the following contents:

{                                    
  "presets" : ["es2015", "react"]    
}

Step 10: Configure Webpack

In order to tell Webpack we want to use Babel, where our entry point module is, and where we want the output bundle to be created, we need to create a Webpack config file.

Within the hello-react folder, create a file named webpack.config.js with the following contents:

const path = require('path');
 
module.exports = {
  entry: './app/index.js',
  output: {
    path: path.resolve('dist'),
    filename: 'index_bundle.js'
  },
  module: {
    rules: [
      { test: /\.js$/, loader: 'babel-loader', exclude: /node_modules/ }
    ]
  }
}

Step 11: Create a React Component

For our example, we’ll just be creating a simple component which renders the text “Hello, React!”.

First, create an app sub-folder:

$> mkdir app

Next, create a file named app/index.js with the following content:

import React from 'react';
import ReactDOM from 'react-dom';
 
class HelloWorld extends React.Component {
    render() {
          return (
                  <div>
                    Hello, React!
                  </div>
                )
        }
};
 
ReactDOM.render(<HelloWorld />, document.getElementById('root'));

Briefly, this code includes the react and react-dom modules, defines a HelloWorld class which returns an element containing the text “Hello, React!” expressed using JSX syntax, and finally renders an instance of the HelloWorld element (also using JSX syntax) to the DOM.

If you’re completely new to React, don’t worry too much about trying to fully understand the code. Once you’ve completed this tutorial and have an example up and running, you can move on to one of the aforementioned tutorials, or work through React’s Hello World example to learn more about the syntax used in this example.

Note: In many examples, you will see the following syntax:

var HelloWorld = React.createClass( {
    render() {
          return (
                  <div>
                    Hello, React!
                  </div>
                )
        }
});

This syntax is how classes were defined in older versions of React and will therefore be what you see in older tutorials. As of React version 15.5.0 use of this syntax will produce the following warning:

Warning: HelloWorld: React.createClass is deprecated and will be removed in version 16. Use plain JavaScript classes instead. If you’re not yet ready to migrate, create-react-class is available on npm as a drop-in replacement.

Step 12: Create a Webpage

Next, we’ll create a simple html file which includes the bundled output defined in step 10 and declare a <div> element with the id “root” which is used by our react source in step 11 to render our HelloWorld component.

Within the hello-react folder, create a file named index.html with the following contents:

<html>
  <div id="root"></div>
  <script src="./dist/index_bundle.js"></script>
</html>

Step 13: Bundle the Application

To convert our app/index.js source to ECMAScript 5 and bundle it with the react and react-dom modules we’ve included, we simply need to execute webpack.

Within the hello-react folder, run the following command to create the dist/index_bundle.js file reference by our index.html file:

$> webpack

Step 14: Run the Example

Using a browser, open up the index.html file. If you’ve followed all the steps correctly, you should see the following text displayed:

Hello, React!

Conclusion

Congratulations! After completing this tutorial, you should have a pretty good idea about the steps involved in getting a basic React app up and going. Hopefully this will save some absolute beginners from spending too much time trying to piece these steps together.

Up into the Swarm

Sat, 08 Apr 2017 20:59:26 +0000

Last Thursday evening I had the opportunity to give a presentation at the Docker Meetup in Austin TX about how to containerize a Node JS application and deploy it into a Docker Swarm. I also demonstrated techniques that can be used to reduce friction in the development process when using containers.

The meeting was recorded but unfortunately sound only is available after approximately 16 minutes. You might want to just scroll forward to this point.

Video: https://youtu.be/g786WiS5O8A

Slides and code: https://github.com/gnschenker/pets-node

New Year, New Blog

Thu, 26 Jan 2017 03:39:05 +0000

One of my resolutions this year was to take ownership of my digital content, and as such, I’ve launched a new blog at jimmybogard.com. I’m keeping all my existing content on Los Techies, where I’ve been humbled to be a part of for the past almost 10 years. Hundreds of posts, thousands of comments, and innumerable wrong opinions on software and systems, it’s been a great ride.

If you’re still subscribed to my FeedBurner feed – nothing to change, you’ll get everything as it should. If you’re only subscribed to the Los Techies feed…well you’ll need to subscribe to my feed now.

Big thanks to everyone at Los Techies that’s put up with me over the years, especially our site admin Jason, who has become far more knowledgable about WordPress than he ever probably wanted.