Los Techies

AutoMapper and MediatR Roadmaps

Tue, 08 Jul 2025 15:15:32 +0000

One of my main goals of commercialization of AutoMapper and MediatR was being able to finally invest time in these projects where basically all new work stopped when I lost corporate sponsorship. I wanted to take some time to share where I'd like to take these projects now that I have that sponsorship back.

Tracking official .NET support

Firstly, the latest releases bring back netstandard2.0 support for both AutoMapper and MediatR which had dropped both years ago. MediatR was actually still on net6.0 prior to this release which was already out of support for months.

It wasn't exactly easy, especially because of how much net8.0 and net9.0 have diverged from netstandard2.0 not just in terms of APIs but C# language features, but having been part of several ASP.NET 4.x to ASP.NET Core migrations, having netstandard2.0 support makes this transition quite a bit easier. In the past we'd have to conditionally reference packages because there was no longer a common package version between say .NET 8 and .NET 4.8. That's something that I wish I had before that now I do.

AutoMapper Roadmap

One of the biggest complaints I hear about AutoMapper is that it's hard to debug - you're trading compile-time errors for runtime exceptions. We spent a LOT of time baking in better exception handling into the expression trees generated (resulting in worse performance, but better diagnostics), but that isn't always enough.

The answer here is source generators, but I'm not interested in merely copying other library's approaches. What I want to target is source generators that:

Plug in to AutoMapper's rich extensibility model
Stay true to AutoMapper's design philosophy
Support IQueryables (my favorite feature)
Track the features of AutoMapper's in-memory mapping
Support mapping validation (critical for any mapping tool)

Debuggability is my main focus here, although obviously performance would be a secondary win. Source generators have come a LONG way since I first looked at them when they were first released, so I'm excited to extend AutoMapper's functionality in this area.

This one is pretty big, so that's going to be my focus initially.

MediatR Roadmap

Some folks have asked or even pointed to other libraries that do source generation of basically a copy of MediatR's API. I am looking at that, but there's been quite a few things on MediatR's backlog that I want to look at first. Source generation in mediators I find a bit less interesting in real-world projects, outside of philosophical debates.

MediatR is commonly used in concert with Vertical Slice Architecture, and a number of its features came out of using it in these scenarios (like behaviors). Today, a lot of features are tied in to the feature set of the stock Microsoft DI container. Unfortunately, features are only really added to that container if the ASP.NET Core team needs them. Even my PR to support generic constraints took like 5 years to merge in.

Moving away from relying on those DI features would mean I could do much more interesting things in the "application use case pipeline" that aren't possible with C#/DI alone, like:

Applying behaviors based on customized policies
Baking in support for result patterns
Direct support for application use cases
- Blazor (sending a request from the client to a handler on the server)
- Minimal APIs (scaffolding to separate API logic from application logic)
- Domain events via notifications and EF/other ORMs

The idea of behaviors came from reviewing many production systems using MediatR and folding in that into first-class features. I am going to continue on this track.

What else are you interested in?

AutoMapper and MediatR Commercial Editions Launch Today

Wed, 02 Jul 2025 15:00:12 +0000

Today I'm excited to announce the official launch and release of the commercial editions of AutoMapper and MediatR. Both of these libraries have moved under their new corporate owner (me), Lucky Penny Software. I formed this company to house these projects separate from my consulting company, but it's just me there, I'm the sole corporate overlord.

The GitHub repositories have transferred to the new GitHub organization (along with their ownership) here:

With these, I've launched new home pages for each library:

As well as a storefront site to purchase and manage licenses at:

https://luckypennysoftware.com

It's quite a bit to dig in to, so let's go over the details!

What's the new license?

As discussed before, I wanted to release these libraries under a dual-license model:

It's a common dual-license model that many other OSS companies have chosen (MongoDB etc.) and had success with.

Under the commercial license, I've created a tier-based licensing model based on team size. There are no individual per-seat licenses, only licensing based on the number of developers.

How much will it cost?

With a tier-based pricing approach, I wanted a pricing model that scales with team size and allows for company growth without a lot of hassle. There are 3 paid tiers:

Standard - 1-10 developers
Professional - 11-50 developers
Enterprise - Unlimited developers

Pricing is a subscription model, with both monthly and annual subscriptions (with a discount for annual subscriptions), as well as an option to bundle both libraries at a discount. You can find the details here (with all options), priced to your currency or in USD:

You can also find the details of what subscription benefits you'll get at the links above, including:

Private Discord channels
Priority support
Early access to new releases
Support for all currently supported versions of .NET Framework 4.x and .NET (netstandard2.0, net8.0, net9.0)
And more (as I build it)

All subscription payments are managed through Paddle, which supports...many different countries, currencies, and payment providers.

Do you have free licenses for <insert situation here>?

Yes! Besides the RPL license, I'm also including a Community edition under the Commercial license that is free for:

Companies and individuals under $5,000,000 in gross annual revenue
Non-profits under $5,000,000 in annual total budget (expenditure)
Educational/classroom use
Non-production environments

You're still required to register for a license key, but this is only for auditing purposes.

How do I get the commercial versions?

To make everyone's lives easier, these new major versions of AutoMapper and MediatR on NuGet are released under the new dual license agreement:

When you install these versions, you'll now be prompted for license acceptance. Once you obtain a license key, you'll be able to set the license key as:

services.AddAutoMapper(cfg => /* or AddMediatR */
    cfg.LicenseKey = "<License key here>";
});

I don't restrict usage of these products with a missing/invalid/expired license key, but you'll see some messages in your logs prompting you to supply a valid key.

What about the existing versions?

I've created archived versions of the final releases of these two libraries:

Per those existing license agreements, you're free to fork, download, print out and read by the fireplace. Those archives will live on for anyone to use as they like.

If you're an existing user, you don't need to do anything. The existing NuGet packages (prior to the major versions listed above) are bound by the license agreements at the time of their release and will also live on.

Why Lucky Penny?

Because she was my first dog! Although she's no longer with us anymore, I loved her spunk and her spirit and wanted to honor her memory with my company name (and logo). Here she is judging, always judging:

Penny the dog

I named her Penny because 1) she was found by the side of a busy highway miles from anywhere (lucky for both of us) and 2) her copper color. So, Lucky Penny Software!

Lucky Penny Software logo

It's been a long journey to get here but I'm excited about what the future holds for these libraries that have amassed more than 1.1 billion downloads. Thanks everyone for your patience and support as I worked to launch!

AutoMapper and MediatR Licensing Update

Wed, 16 Apr 2025 05:15:28 +0000

In my last post, I shared the news that I've decided to take a commercialization route for AutoMapper and MediatR to ensure their long-term success. While that post was heavy on the motivation, it was intentionally light on the details. I did share that I wanted to be transparent on that process, and this post is part of that transparency.

There is a TON of information out there on possible models for sustainable open source, such as:

Consulting services
Open core
Hosted services
Dual license
a dozen others

Of course besides my previous situation, "be fortunate enough to work at a place that values and directly sponsors your work." This is the easiest place to be, but for projects that reach some threshold of users/downloads/complexity, maintainers must rely on sponsorship in some form or fashion. And when that sponsorship goes away for whatever reason, well, here we are.

Of the many options available, the most viable option is to move AutoMapper and MediatR to a dual license model. This looks to be the best choice after carefully examining the options and consulting with many other OSS maintainers who have already made this journey.

Dual Licensing Model

When I first started thinking about how I might go about this, I asked myself, "who bears the most responsibility in ensuring the sustainability of the OSS projects on which they depend?" which is a long winded way of saying "who should pay?" But another way of thinking of this is "who should NOT pay?" Looking at how others do this as well as how I want to approach it, I want to make these libraries free for:

Developers using it in an OSS setting
Individuals/students/hobbyists (using AutoMapper for fun not profit)
Non-profit/charities (maybe not for fun but also not for profit)
Startups or small companies (below some revenue/funding threshold)
Non-commercial setting (this I'm not sure is absolutely necessary with the other categories)
Non-production environments (instead of any trial period etc.)

I don't know if this exact verbiage is what will be the end result, but this is my overall goal.

Then for who I'm targeting for paid for-licenses, it's for-profit businesses using these libraries for commercial activities. Looking at my clients over the years who've used my libraries, it's a mix of these free/commercial categories.

In terms of a model for commercial licensing, I want to ensure that paid licenses add value beyond "I can download the license." This is the more fun part of this exercise for me, where I can try the things I never really could before without a more direct form of sponsorship/funding. I have a lot of ideas here, but nothing ready to share yet. If you have an idea of "if my company paid for a license, what else would I want to have included?" I would love to hear about it!

I am looking at a tiered license model but no per-seat licenses. I don't want to charge individual developers anything—that seems like a pain for everyone involved and I'm trying to keep things simple. "A new developer gets onboarded and now we need a new license" is too much for me to deal with and goes against the spirit of these libraries—the benefit is to the entire team, regardless of the number of developers.

I don't know what those tiers will be exactly, I'm figuring that out next. I do expect some blanket enterprise, site-wide licenses that hopefully makes everything simpler for everyone. I've been on the other side of the table, getting licenses approved internally with clients, and I understand predictability and simplicity go a long way.

Thoughts on Pricing

As for pricing, I don't have details yet, and probably won't until launch in the next couple months. Range-wise, it's hard to compare to other commercial or dual-licensed products out there, since I don't want to do any individual or per-seat license and that seems to be the norm. I am however keenly aware of how much tooling and library products cost as I have to pay for many of these myself.

But if I were to compare to the cost for a team of 10 or 50 or 100 for their IDEs, I would expect my commercial license price to be a fraction of that.

Thanks again to everyone that's reached out with kind words and support, and to the community for their patience while I figure things out.

AutoMapper and MediatR Going Commercial

Wed, 02 Apr 2025 13:00:12 +0000

Yes, another one of "those posts". But tl;dr:

In order to ensure the long-term sustainability of my OSS projects, I will be commercializing AutoMapper and MediatR.

I did not post this on April 1st for obvious reasons. But first a little background on how I got to this point.

How I Got Here

These two projects originated at my time at Headspring, a consulting company I worked at for over 12 years. About 5 years ago, in January 2020, I decided to strike off on my own and give solo consulting a try. Although it was a scary leap, it's been more rewarding than I could have possibly hoped for, in almost every area.

The area that it didn't work out well, and not at all intentionally, was OSS work:

You can see exactly where my contributions cratered and flat-lined. And that's just commits—issues, PRs, discussions, all my time dried up. This wasn't the intention but was a natural side effect of me focusing on my consulting business.

At Headspring, my time on OSS was directly encouraged and sponsored by them. I could use time between projects to invest back in existing OSS or new OSS, because it benefited the client, the company, and the employees (me and my coworkers).

With me leaving that company, and that company then selling to Accenture later that year, I had no direct major sponsor of my OSS work anymore. My free time was being spent growing and ensuring the success of my consulting company, which being solo, is...kinda important.

Taking time to see how things have been going on all fronts, I had a bit of a shock looking at my OSS work. I realized that model is not sustainable for the long-term success of these projects, which I still endorse and believe in. I need to be able to pay for my time to work on these projects, and get direct feedback from paying clients, like I had earlier at Headspring.

What Will This Look Like?

The short answer is "I don't know exactly". I'm working out those details now and will share them when I figure it out. I have lots of examples of what does and doesn't work well, at least from my perspective, as well as what I consider will work well for these projects.

Short term, nothing will change. I'll still be as (un)responsive on GitHub issues, and I just pushed out a couple releases of any existing work.

My goal is to be able to pay for the time to spend actually improving these projects, building out communities, helping more users, and in general, doing the things that people have asked me MANY times over the years that I should do, but I didn't, because it was not my job. OSS was/is/never will be a hobby for me. I want to change it to at least be part of my job and to fund real work.

I can’t rely on donations, I don't want to make developers pay anything or do anything to punish/annoy them, and I certainly don't think it's Microsoft's job to "pay me the money." Past that, I'm still figuring it out.

When Will This Happen?

I don't know, it's still just me that owns everything. It's still using my free time to sort it out, as my day job is still a consultant. But I plan to be open with this whole process. I'm sure I'll surprise someone but the goal here is to be transparent.

Personally, I'm both filled with excitement and dread—doing these projects for so long has been incredibly rewarding, especially as this is code that came directly out of many, many long-lived production-deployed projects at Headspring. But I don't want these projects to wither and die on the vine, I want them to grow and evolve and thrive. But not just these projects—I want ALL my OSS projects (Respawn etc.) to thrive. This is how it needs to happen.

Final Thanks

Thanks to all that have contributed over the years, and especially to Lucian Bargaoanu who really helped pick up the torch with AutoMapper after I more or less fell off the map. Also thanks to my GitHub sponsors, as many a pint has been purchased with your generous support. And finally thanks to the community, I never hoped anything I built would help anyone beyond my clients, coworkers, and company, but it's always nice to hear that it has.

MediatR 12.5.0 Released

Tue, 01 Apr 2025 18:50:06 +0000

I pushed out MediatR 12.5 today:

This is mainly a regular minor release with a couple extra interesting features:

Adding convenience method to register open behaviors
Better cancellation token support (it's passed now everywhere including behaviors)

And some other cleanup items as well. Enjoy!

AutoMapper 14.0 Released

Wed, 19 Feb 2025 13:41:24 +0000

I pushed out version 14.0 (!) of AutoMapper over the weekend:

This release targets .NET 8 (up from .NET 6 from the previous release). It's mainly a bug fix release, with some quality-of-life improvements in configuration validation where we gather up all the possible validation errors before reporting them in an aggregate exception.

Enjoy!

Integrating the Particular Service Platform with Aspire

Tue, 24 Sep 2024 22:48:57 +0000

I've been playing around with Aspire for a bit mainly to understand "is this a thing I should care about?" and part of what I wanted to do is take a complex "hello world" distributed system and convert it to Aspire. Along the way, Particular Software also released container support for their Service Platform, so it also seemed like a good opportunity to try it out.

I'll follow up in another post about Aspire impressions, but the NServiceBus part was actually relatively simple. Many Aspire integrations have some kind of 1st-party support where you can do things like:

var rmqPassword = builder.AddParameter("messaging-password");
var dbPassword = builder.AddParameter("db-password");

var broker = builder.AddRabbitMQ(name: "broker", password: rmqPassword, port: 5672)
    .WithDataVolume()
    .WithManagementPlugin()
    .WithEndpoint("management", e => e.Port = 15672)
    .WithHealthCheck();
var mongo = builder.AddMongoDB("mongo");
var sql = builder.AddSqlServer("sql", password: dbPassword)
    .WithHealthCheck()
    .WithDataVolume()
    .AddDatabase("sqldata");

And now my system has RabbitMQ, MongoDB, and SQL Server up and running in containers. There's a lot of stock configuration going on behind AddSqlServer and similar methods but we don't have to use those convenience methods if we don't want to.

The overall Service Platform architecture looks something like:

The "instances" here are running containers that we need to configure in Aspire. On top of that, we might also want to have Service Pulse (another container) and Service Insight (a Windows-only WPF app) running, and these all require extra configuration. Also, the Error and Audit instances use RavenDB as their backing store but Particular also has an image there. The Docker Hub site has links to docs on both the instance and containers.

First up, we need to provide our license to the running containers as raw text in an environment variable, so we'll just read our license (this is just for local development):

var license = File.ReadAllText(                                              
    Path.Combine(                                                            
        Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),
        "ParticularSoftware",                                                
        "license.xml"));

Next, we need our RavenDB instance. There's a special image from Particular, so we'll use the AddContainer method to add our custom image to our Aspire distributed application:

builder                                                                             
    .AddContainer("servicecontroldb", "particular/servicecontrol-ravendb", "latest")
    .WithBindMount("AppHost-servicecontroldb-data", "/opt/RavenDB/Server/RavenData")
    .WithEndpoint(8080, 8080);

The container docs say that we must mount a persistent volume to that path, so we use the WithBindMount method to mount the volume following the Aspire docs.

Next up are the Particular containers!

Setting up the Service Control Error instance

From the Particular docs, we see that we need to supply configuration for:

Transport type (RabbitMQ, Azure Service Bus, etc.)
Connection string to the transport
Connection string to the Raven DB instance
Audit instance URLs
License

Plus port mapping. Pretty quickly I ran into a few challenges:

The Service Control image can start before RabbitMQ is "ready", resulting in connection failures
Service Insight, the WPF app, is Windows only so I need to connect to Service Control from a VM

The base configuration is fairly straightforward, we specify the container and image, with environment variables:

builder                                                                                             
    .AddContainer("servicecontrol", "particular/servicecontrol")                                    
    .WithEnvironment("TransportType", "RabbitMQ.QuorumConventionalRouting")                         
    .WithEnvironment("ConnectionString", "host=host.docker.internal")                               
    .WithEnvironment("RavenDB_ConnectionString", "http://host.docker.internal:8080")                
    .WithEnvironment("RemoteInstances", "[{\"api_uri\":\"http://host.docker.internal:44444/api\"}]")
    .WithEnvironment("PARTICULARSOFTWARE_LICENSE", license)                                         
    .WithArgs("--setup-and-run")

But the other two challenges are a bit harder to deal with. There is no built-in way in Aspire to "wait" for other resources to start. This isn't new to Aspire - in the past we had to write custom hooks in Docker Compose to wait for our dependencies' health checks to come back. The extensibility is there to do such a thing, so I found an extension to do just that.

The second problem was...a long slog to figure out. It's possible to have a Parallels VM be able to communicate with Docker containers running in the Mac host. However, I could not get this to work with Aspire. After doing side-by-side comparisons between container manifests running inside/outside of Aspire, I found the culprit:

"PortBindings": {
	"8080/tcp": [
		{
-			"HostIp": "",
+			"HostIp": "127.0.0.1",
			"HostPort": "8000"
		}
	]
},

With the Docker CLI, doing -p 8080:8000 does not set the host IP. Aspire does however, which means I can only access this container via localhost. Not ideal because my Windows VM is definitely not able to access that. Instead of using WithEndpoint or similar, I have to drop down to container runtime args:

.WithContainerRuntimeArgs("-p", "33333:33333")
.WaitFor(rabbitMqResource);

Now my Service Control instance is up and running!

Setting up Service Control Audit, Monitoring, and Service Pulse

Following our previous example, we can finish out our configuration for the other container instances:

builder                                                                              
    .AddContainer("servicecontrolaudit", "particular/servicecontrol-audit")          
    .WithEnvironment("TransportType", "RabbitMQ.QuorumConventionalRouting")          
    .WithEnvironment("ConnectionString", "host=host.docker.internal")                
    .WithEnvironment("RavenDB_ConnectionString", "http://host.docker.internal:8080") 
    .WithEnvironment("PARTICULARSOFTWARE_LICENSE", license)                          
    .WithArgs("--setup-and-run")                                                     
    .WithEndpoint(44444, 44444)                                                      
    .WaitFor(rabbitMqResource);                                                      
                                                                                     
builder                                                                              
    .AddContainer("servicecontrolmonitoring", "particular/servicecontrol-monitoring")
    .WithEnvironment("TransportType", "RabbitMQ.QuorumConventionalRouting")          
    .WithEnvironment("ConnectionString", "host=host.docker.internal")                
    .WithEnvironment("PARTICULARSOFTWARE_LICENSE", license)                          
    .WithArgs("--setup-and-run")                                                     
    .WithEndpoint(33633, 33633)                                                      
    .WaitFor(rabbitMqResource);                                                      
                                                                                     
builder                                                                              
    .AddContainer("servicepulse", "particular/servicepulse")                         
    .WithEnvironment("SERVICECONTROL_URL", "http://host.docker.internal:33333")      
    .WithEnvironment("MONITORING_URL", "http://host.docker.internal:33633")          
    .WithEnvironment("PARTICULARSOFTWARE_LICENSE", license)                          
    .WithEndpoint(9090, 9090)                                                        
    .WaitFor(rabbitMqResource);

With all this in place in my Service Pulse instance is up and running:

And on the Service Insight side, I had to do the Parallels trick of using my hosts file to create a special "localhost.mac" entry to point to the Mac host:

10.211.55.2 localhost.mac

With this in place, I can configure Service Insight in Windows to connect to the Docker Service Pulse instance running in Docker on the Mac:

All my NServiceBus messages and traces now show up just fine:

Most of the work I had to do was not really Aspire-related, but just configuring Aspire to pass in the appropriate configuration to the containers. You can find the full code to my configuration here:

Code Example

Enjoy!

Tales from the .NET Migration Trenches - Turning Off the Lights

Thu, 05 Sep 2024 15:25:16 +0000

Posts in this series:

In the last post, we looked at migrating our middleware, which we tackle in an as-needed basis. When a controller needs middleware to be migrated, we migrate that middleware over. If the entire app needs the middleware, it needs to come rather early.

Once we migrate much of our middleware over, it becomes much less work to incrementally migrate individual controllers and their subsequent actions/pages over. I won't go into deep detail into this part - mostly it's fixing namespaces, adjusting features (such as converting child actions into view components), but it can go quite fast. On recent teams I was working with, we migrated easily a dozen controllers a week amongst 3-4 developers. At this point, the bottleneck wasn't the conversion, but testing to make sure the pages still worked correctly

It's essentially testing the entire application, one page at a time, so hopefully you've got some regression tests in some form or fashion. I'm not skipping the incremental controller migration because it's not interesting - it's just because our teams really didn't encounter many challenges there. There will be something that comes up, there always is, but just the controller/action/view part is not too terrible.

But in this post I wanted to focus on getting to the end - what do we do once we've migrated everything but authentication? When there's just one controller left, we're now OK to proceed with migrating the last pieces over and "turning off the lights" on the .NET 4.x application.

Migrating Last Features

The last (or next-to-last) migration typically:

Migrates the last controller, usually authentication
Turns off proxying and all remote app features

You don't necessarily need to split this into two separate units of work/deployments, as once you've migrated the last set of requests you can migrate all final features from the .NET Framework application. If the last controller is authentication, we'll also need to remove remote authentication. Our current web adapter configuration before final migration is:

builder.Services.AddSystemWebAdapters()
    .AddJsonSessionSerializer(options =>
    {
        options.RegisterKey<string>("FavoriteInstructor");
    })
    .AddRemoteAppClient(options =>
    {
        // Provide the URL for the remote app that has enabled session querying
        options.RemoteAppUrl = new(builder.Configuration["ProxyTo"]);

        // Provide a strong API key that will be used to authenticate the request on the remote app for querying the session
        options.ApiKey = builder.Configuration["RemoteAppApiKey"];
    })
    .AddAuthenticationClient(true)
    .AddSessionClient();
builder.Services.AddHttpForwarder();

With middleware:

app.UseSystemWebAdapters();

app.MapDefaultControllerRoute();
app.MapForwarder("/{**catch-all}", app.Configuration["ProxyTo"]).Add(
    static builder => ((RouteEndpointBuilder)builder).Order = int.MaxValue);

Along with migrating the authentication piece and all related middleware, we'll remove the above from our application startup, as well as the package references to all the proxy and System.WebAdapters packages. Once that's complete, our .NET application should now handle all requests. There might still be a few extra features to enable in .NET 8, such as Session:

builder.Services.AddSession();

// later

app.UseSession();

With all that complete, our .NET 8 application should now serve all requests and host all features needed to run our entire system.

Turning off the lights

While our .NET 8 application may now be "complete", we're not quite done yet. In my typical last phase we will:

Deploy the completed .NET 8 application to production
Monitor for any errors and any activity from the .NET 4.8 application
Adjust our .NET 8 application as necessary

If we don't see any issues, then the final final cleanup is:

Remove all .NET 4.8 code from the repository
Remove any shims to bridge from .NET 8 to .NET 4.8
Remove all .NET 4.8 application pipelines and deployments
Remove all .NET 4.8 production resources

And we should end with something like:

So what's next? There's still probably quite a bit to do to ".NET-8-ify" our existing system - all those architectural improvements we skipped in order to fast track migration. But most important - celebrate!

Upcoming Training on DDD with Vertical Slice Architecture in Munich

Wed, 28 Aug 2024 10:28:04 +0000

I've got another training event coming up focusing on Domain-Driven Design with Vertical Slice Architecture in Munich on October 21-23rd.

A little different than the previous times I've given this course is an option for either a 2-day or 3-day version. I had received feedback that folks were also interested in larger-scale design concepts such as bounded contexts, messaging, integration patterns, microservices, and modular monoliths. So I've included a 3rd day that covers these topics, where we look at encapsulation and cohesion at larger and larger scopes.

We'll cover:

Refactoring an existing system to leverage Vertical Slice Architecture
Applying Domain-Driven Design techniques to model complex business needs
Communication between slices
Exploring Validation and Testing (and other cross-cutting concerns) using Vertical Slice Architecture
Examining various design patterns, code smells, and refactoring techniques
Implementing the Vertical Slice Architectural pattern in various enterprise application scenarios (minimal APIs, Blazor, Web APIs, etc.)

And on the final day:

Service boundaries and bounded contexts
Communication between bounded contexts
Microservices and modular monoliths
Studying distributed systems patterns, tools, and libraries such as NServiceBus

The course pulls together my experiences building such systems for nearly 20 years now. And if you can't make the course during the day, I'm also hosting a networking event during the evening where you can meet myself and the other attendees and ask me questions. I hope to see you there!

Tales from the .NET Migration Trenches - Middleware

Tue, 06 Aug 2024 15:46:39 +0000

Posts in this series:

In the last post, we looked at tackling probably the most important pieces of middleware - authentication. But many ASP.NET MVC 5 applications will have lots of middleware, but not all of the middleware should be migrated without some analysis on whether or not that middleware is actually needed anymore.

This part is entirely dependent on your application - you might have little to no middleware, or lots. Middleware can also exist in a number of different places:

Web.config (you WILL forget this)
Global.asax (probably calling into other classes with the middleware configuration)
OWIN Startup

When choosing the first controller to migrate, I'm also looking at which controllers have the least amount of middleware, just to minimize the heavy first lift.

Let's look at our various middleware, and see what makes sense to move over, starting with our web.config.

Migrating Web.Config

I think I forget the web.config middleware mainly because I've tried to burn most things ASP.NET from my brain. But we'll find lots of important hosting configuration settings in our web.config, from custom middleware to error handling, application configuration, server configuration and more. Luckily for us, nearly all out-of-the-box configuration has a direct analog in Kestrel. We mostly need to worry about anything custom here. My sample app doesn't have a lot going on:

<system.web>                                                                                        
  <compilation debug="true" targetFramework="4.8.1" />                                              
  <httpRuntime targetFramework="4.8.1" />                                                           
  <customErrors mode="RemoteOnly" redirectMode="ResponseRewrite">                                   
    <error statusCode="404" redirect="/404Error.aspx" />                                            
  </customErrors>                                                                                   
  <!-- Glimpse: This can be commented in to add additional data to the Trace tab when using WebForms
      <trace writeToDiagnosticsTrace="true" enabled="true" pageOutput="false"/> -->                 
  <httpModules>                                                                                     
    <add name="Glimpse" type="Glimpse.AspNet.HttpModule, Glimpse.AspNet" />                         
  </httpModules>                                                                                    
  <httpHandlers>                                                                                    
    <add path="glimpse.axd" verb="GET" type="Glimpse.AspNet.HttpHandler, Glimpse.AspNet" />         
  </httpHandlers>                                                                                   
</system.web>

We only have one set of custom modules/handlers and it's the now-dead (and much missed) Glimpse project. In the rest of the configuration, we only see custom errors redirecting to an .ASPX page, which we can easily port over using custom errors in ASP.NET Core. Otherwise there's not much going on here.

In a typical application, the things I've needed to migrate over might include such settings as:

Authentication
Authorization
Cookies
Session state
Data protection
Static files
HTTP request methods
Initialization
Custom headers
Caching

Each of these has some analog in ASP.NET Core Kestrel configuration. But luckily for us, we don't have any custom handlers/modules to worry about, only porting ASP.NET runtime features to ASP.NET Core.

ASP.NET MVC 5 Middleware

Next up is ASP.NET MVC 5 middleware, which is typically set up in the Global.asax.cs file, something like:

AreaRegistration.RegisterAllAreas();                      
FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
RouteConfig.RegisterRoutes(RouteTable.Routes);            
BundleConfig.RegisterBundles(BundleTable.Bundles);

The global filters registered are:

public static void RegisterGlobalFilters(GlobalFilterCollection filters)
{                                                                       
    filters.Add(new HandleErrorAttribute());                            
    filters.Add(new ValidatorActionFilter());                           
    filters.Add(new MvcTransactionFilter());                            
}

The first filter here is a built-in one from ASP.NET MVC to provide global error handling (with no extra configuration), but the second two are custom. The first custom filter provides some customization around handling validation errors and providing a common error result back to the UI:

public void OnActionExecuting(ActionExecutingContext filterContext)                                   
{                                                                                                     
    if (!filterContext.Controller.ViewData.ModelState.IsValid)                                        
    {                                                                                                 
        if (filterContext.HttpContext.Request.HttpMethod == "GET")                                    
        {                                                                                             
            var result = new HttpStatusCodeResult(HttpStatusCode.BadRequest);                         
            filterContext.Result = result;                                                            
        }                                                                                             
        else                                                                                          
        {                                                                                             
            var result = new ContentResult();                                                         
            string content = JsonConvert.SerializeObject(filterContext.Controller.ViewData.ModelState,
                new JsonSerializerSettings                                                            
                {                                                                                     
                    ReferenceLoopHandling = ReferenceLoopHandling.Ignore                              
                });                                                                                   
            result.Content = content;                                                                 
            result.ContentType = "application/json";                                                  
                                                                                                      
            filterContext.HttpContext.Response.StatusCode = 400;                                      
            filterContext.Result = result;                                                            
        }                                                                                             
    }                                                                                                 
}

The front end does still need this, so we want to port this over. The second filter provides automatic transaction handling:

public class MvcTransactionFilter : ActionFilterAttribute                                              
{                                                                                                      
    public override void OnActionExecuting(ActionExecutingContext filterContext)                       
    {                                                                                                  
        // Logger.Instance.Verbose("MvcTransactionFilter::OnActionExecuting");                         
        var context = StructuremapMvc.ParentScope.CurrentNestedContainer.GetInstance<SchoolContext>(); 
        context.BeginTransaction();                                                                    
    }                                                                                                  
                                                                                                       
    public override void OnActionExecuted(ActionExecutedContext filterContext)                         
    {                                                                                                  
        // Logger.Instance.Verbose("MvcTransactionFilter::OnActionExecuted");                          
        var instance = StructuremapMvc.ParentScope.CurrentNestedContainer.GetInstance<SchoolContext>();
        instance.CloseTransaction(filterContext.Exception);                                            
    }                                                                                                  
}

I might not do automatic transactions like this in a normal project but because the application code expects it, we'll need to migrate this over as well. The transaction filter is interesting because it highlights the shortcomings of ASP.NET MVC 5's dependency injection capabilities - namely there wasn't anything built in for filters. Instead of migrating this filter as-is, we need to translate to the equivalent ASP.NET Core filter:

public class DbContextTransactionFilter : IAsyncActionFilter                                              
{                                                                                                         
    private readonly SchoolContext _dbContext;                                                            
                                                                                                          
    public DbContextTransactionFilter(SchoolContext dbContext)                                            
    {                                                                                                     
        _dbContext = dbContext;                                                                           
    }                                                                                                     
                                                                                                          
    public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {                                                                                                     
        try                                                                                               
        {                                                                                                 
            _dbContext.BeginTransaction();                                                                
                                                                                                          
            var actionExecuted = await next();                                                            
            if (actionExecuted.Exception != null && !actionExecuted.ExceptionHandled)                     
            {                                                                                             
                _dbContext.CloseTransaction(actionExecuted.Exception);                                    
                                                                                                          
            }                                                                                             
            else                                                                                          
            {                                                                                             
                _dbContext.CloseTransaction();                                                            
                                                                                                          
            }                                                                                             
        }                                                                                                 
        catch (Exception ex)                                                                              
        {                                                                                                 
            _dbContext.CloseTransaction(ex);                                                              
            throw;                                                                                        
        }                                                                                                 
    }                                                                                                     
}

And we register our filter:

builder.Services.AddControllersWithViews(opt =>
{
    opt.Filters.Add<DbContextTransactionFilter>();
});

Now our filter will have its DbContext injected instead of going out to a custom extension to mimic per-request service lifetimes.

Finally, let's look at the OWIN middleware.

OWIN Middleware

OWIN middleware can be found in classes with the OwinStartup attribute configured for them. Usually this is a "Startup" class but it could be anything. In my sample app, we have:

[assembly: OwinStartup(typeof(Startup))]
namespace ContosoUniversity
{
    public partial class Startup
    {
        public void Configuration(IAppBuilder app)
        {
            app.MapSignalR();

            GlobalConfiguration.Configuration
                .UseSqlServerStorage("SchoolContext")
                .UseStructureMapActivator(IoC.Container)
                ;


            app.UseHangfireDashboard();
            app.UseHangfireServer(new BackgroundJobServerOptions
            {
                Queues = new[] { Queues.Default }
            });

            ConfigureAuth(app);
        }
    }
}

Basically, it's:

SignalR
Hangfire
Authentication

Authentication might differ slightly than the ASP.NET authentication, so we'll want to port settings there. SignalR and Hangfire can be dealt with individually, but otherwise we don't have any custom OWIN middleware. This is fairly typical unless your application wholly relies on OWIN instead of say, IIS.

Middleware isn't the most exciting code to port over, but critical for ensuring our new application preserves the existing behavior of the .NET Framework application.

In our last post, we'll cover finishing up our migration and "turning off the lights".

Vertical Slice Architecture Training Course in July in the Netherlands

Mon, 22 Apr 2024 22:15:00 +0000

The last training course in Zurich was a success, in that no laptops were harmed. I think. I put a poll out on where I should do the training next and quite a few folks suggested the Netherlands. I'm happy to announce that the next VSA course will be in the Netherlands on July 17-18th.

This course approaches this topic from the perspective of refactoring an existing system to this architecture. We also look at larger and larger boundaries of cohesion, from applications to services to systems. I'm also doing something new, a Q&A at a pub where you can ask questions while we share authentic Dutch beer (Heineken).

More details here:

Vertical Slice Architecture Training

Hope to see you there!

Tales from the .NET Migration Trenches - Authentication

Mon, 22 Apr 2024 22:00:50 +0000

Posts in this series:

Of all the topics in .NET migration, authentication, like always, is the one that is most characterized by "It Depends". The solution for addressing authentication is wholly dependent on what the current authentication solution is in the current .NET 4.8 application. If you're doing external SSO, then it's likely quite simple - the new solution is simply a new client for your external SSO.

In my situation, the .NET Framework application was responsible for authentication, i.e., it had a login screen. It was a home-grown identity provider, not using ASP.NET Identity. If you're using ASP.NET Identity and all the database backing stores, you're also looking at a data migration. I'll leave that as an exercise to the reader ;)

The end result we're looking for is:

Users can log in via one of the apps (.NET 8 or .NET 4.8)
Once logged in, both apps recognize the user as authenticated and can read identical claims/roles
Users can log out via one of the apps

Our two dumbed down options available to solving this are:

Remote authentication in ASP.NET 4.8
Cookie sharing between ASP.NET 4.8 and ASP.NET Core

The cookie sharing option is intriguing but it has some limitations:

Only works with Microsoft.Owin cookie authentication
Requires shared cookie and data protection configuration between applications

Our application didn't have that first constraint so we couldn't consider it. Remote authentication works by:

Users log in and out of the ASP.NET 4.8 application
ASP.NET Core adapters call APIs in ASP.NET 4.8 to retrieve user authentication information (claims) and populates its claims identity with this data

It's very similar to the remote session story:

Except getting we're getting the claims information from the ASP.NET application. This means, however, that the login/logout endpoints will need to be migrated last. Which means if our authentication story is complicated, we'll have plenty of runway since it'll be last.

Configuring Remote Authentication

Configuring remote authentication is straightforward if we've already added the remote app server for session. We add a single line of code to the ASP.NET application to AddAuthenticationServer:

this.AddSystemWebAdapters()
    .AddJsonSessionSerializer(options =>
    {
        options.RegisterKey<string>("FavoriteInstructor");
    })
    // Provide a strong API key that will be used to authenticate the request on the remote app for querying the session
    // ApiKey is a string representing a GUID
    .AddRemoteAppServer(options => options.ApiKey = ConfigurationManager.AppSettings["RemoteAppApiKey"])
    .AddAuthenticationServer()
    .AddSessionServer();

And in our ASP.NET Core application, to add the authentication client:

builder.Services.AddSystemWebAdapters()
    .AddJsonSessionSerializer(options =>
    {
        options.RegisterKey<string>("FavoriteInstructor");
    })
    .AddRemoteAppClient(options =>
    {
        // Provide the URL for the remote app that has enabled session querying
        options.RemoteAppUrl = new(builder.Configuration["ProxyTo"]);

        // Provide a strong API key that will be used to authenticate the request on the remote app for querying the session
        options.ApiKey = builder.Configuration["RemoteAppApiKey"];
    })
    .AddAuthenticationClient(true)
    .AddSessionClient();

There's a ton of options, because of course authentication is complicated, but this also means we can turn on authentication and authorization as normal in our ASP.NET Core application:

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.UseSystemWebAdapters();

With this in place, we can access all the normal ClaimsPrincipal and IIdentity details anywhere inside our ASP.NET Core application. We can't examine the security cookie - but we shouldn't anyway, our application code should only be concerned with the principal and identity, not the underlying details of how that got populated.

If we need to add more claims, those will get added on the ASP.NET side and automatically populated on the ASP.NET Core side with those API calls back to get all the claims for the user. It's another clever shim to allow us to migrate all controllers, actions, and application code that require authentication and authorization.

In the next post, we'll look at the middleware that exists in the ASP.NET application and migrate anything we actually want to migrate, and leave the rest behind.

Upcoming Training on Modern .NET with Vertical Slice Architecture

Wed, 07 Feb 2024 21:00:59 +0000

Something new I'm starting this year is a two-day course on Modern .NET systems with Vertical Slice Architecture. It contains a lot of topics that I've consulted with organizations and built systems around for around over a decade now, and I wanted to wrap my learnings up into a single training course.

And since most of my systems I deal with are not greenfield, but dealing with existing systems, this course focuses on refactoring a system using Vertical Slice Architecture, and all the patterns, tools, and libraries that come along with it. In particular, I'll be focusing on:

Refactoring an existing system to leverage Vertical Slice Architecture
Applying Domain-Driven Design techniques to model complex business needs
Exploring various design patterns, code smells, and refactoring techniques
Using the Vertical Slice Architectural pattern in a variety of modern .NET 8 application scenarios (minimal APIs, Blazor, Web APIs, etc.)
Effective use of common libraries such as AutoMapper and MediatR
Examining distributed systems patterns, tools, and libraries such as NServiceBus

The training will be in Zurich, Switzerland on April 9-10. Use the early bird voucher code EarlyBird20 through the end of February for a 20% discount:

Register Now

I hope to see you there!

AutoMapper 13.0 Released

Tue, 06 Feb 2024 15:38:00 +0000

Today I pushed out AutoMapper 13.0 (is that too many...?):

Probably the biggest change with this release is folding in Microsoft.Extensions.DependencyInjection support directly. The AutoMapper.Extensions.Microsoft.DependencyInjection package is deprecated as a result.

Side note, the docs were messed up with this version so go to the "latest" version to see them.

Tales from the .NET Migration Trenches - Hangfire

Mon, 29 Jan 2024 18:04:40 +0000

Posts in this series:

In the last post, we encountered our first instance of shared runtime data between our different ASP.NET 4.8 and ASP.NET Core applications, in Session State. There are other mechanisms to store state in ASP.NET 4.8 (such as Application state), but Session is the most common. In this post, we'll look at another instance of shared state that isn't built in to ASP.NET, but I find quite common - Hangfire.

Hangfire is an easy way to perform background tasks/processes in a .NET web application, and it also supports persistent storage for both the jobs and queues. I use it quite a lot in applications where I don't want to introduce a separate host for processing messages, or introduce a specific queue/broker for background jobs. Hangfire supports fire-and-forget jobs as well as "cron"-based jobs. It also provides a nice dashboard where you can see completed and failed jobs, with the option of retrying failed jobs as desired.

Depending on how you're using Hangfire, it introduces a unique challenge when migrating from .NET 4.8 to .NET 6/7/8. Hangfire supports both frameworks, but as usual, the devil is in the details. We want to be able to start/consume jobs from both sides AND ensure our job executes at most once.

First, let's look to see how we configure and use Hangfire today.

ASP.NET 4.8 Hangfire Usage

In our OWIN startup in the ASP.NET 4.8 application, we find our Hangfire configuration:

GlobalConfiguration.Configuration
    .UseSqlServerStorage("SchoolContext")
    .UseStructureMapActivator(IoC.Container)
    ;

app.UseHangfireDashboard();
app.UseHangfireServer();

We can see here that we're using SQL Server for our storage (jobs and queues), and that we're using a DI container (StructureMap) for activating/instantiating jobs. We don't see it explicitly configured but our job is using the default queue, named default.

Our jobs can be enqueued anywhere really, from controllers to services to startup. For anything that migrates to ASP.NET Core that uses Hangfire, we'll have to migrate that usage as well. Here's one usage:

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Edit(Edit.Command command)
{
    await _mediator.Send(command);

    _backgroundJobClient.Enqueue(() => LogEdit(command));

    return this.RedirectToActionJson(c => c.Index(null));
}

[NonAction]
public void LogEdit(Edit.Command command)
{
    _logger.Information($"Editing student {command.ID}");
}

It's completely trivial, but let's assume the background job is actually doing something interesting, like sending emails or SMS messages.

If we were to migrate this by itself over to ASP.NET Core, we'll immediately run into an issue - Hangfire is now running in two places - ASP.NET and ASP.NET Core, and if we do nothing additional, Hangfire in each server will try to consume those jobs. Unfortunately, it might not be able to execute those jobs. In the above example, the job exists simply as a method from my controller - a perfectly valid way of using Hangfire. If this method only exists in one of the web applications, the other web application won't be able to execute the job and it will wind up failing.

Hangfire does support web farm scenarios and the competing consumer pattern, so we still know that only one side or the other will pick up the job. But it might not be able to execute it if the job code isn't there.

We could fix this by migrating all of our job code to the "shared" assembly first, but this might be a complex undertaking especially if we're using the pattern above. Instead, we can create separate queues for each host - ASP.NET 4.8 and ASP.NET Core, and ensure the job is queued to the host where that job code lives.

When both live in ASP.NET Core:

When the initiator is ASP.NET Core and the job lives in ASP.NET 4.8:

And the reverse:

And finally solely ASP.NET 4.8:

With this setup, the job initiator must "know" where the job code lives. This might seem like unnecessary coupling, but keep in mind this is transitional configuration and we won't need to have this knowledge baked in once all of the jobs and initiators are migrated.

Configuring for Multiple Hosts

Initially, we did not specify any queue in our Hangfire configuration. Now, we'll be explicit in ASP.NET 4.8:

app.UseHangfireServer(new BackgroundJobServerOptions
{
    Queues = new[] { Queues.Default }
});

And after pulling in the appropriate packages to ASP.NET Core, we configure our startup there with the other queue:

builder.Services.AddHangfire(cfg =>
{
    cfg.UseSqlServerStorage(
        builder.Configuration.GetConnectionString("SchoolContext"));
});
builder.Services.AddHangfireServer(options =>
    options.Queues = new[] { Queues.DefaultCore }
);

I could migrate the Hangfire dashboard over to ASP.NET Core, but I left it alone for now. The YARP piece will take care of that for now.

For jobs that start and stay inside of one host, there's nothing we need to do to specify a queue. However, for jobs that cross host boundaries, we'll specify the queue name in the job:

[NonAction]
[Queue(Queues.DefaultCore)]
public void LogEdit(Edit.Command command)
{
    _logger.Information($"Editing student {command.ID}");
}

This ensures that the jobs start and stop where they're supposed to, and are only executed at most once. Once all of our jobs are migrated, we can rename the queue to the default name (as long as we've drained our job queues beforehand).

So far, all of our actions don't require a logged in user. In the next post, we'll tackle authentication.

SSH on WSL

Fri, 01 Jul 2022 08:00:00 +0000

I recently set up a Windows machine to allow me to ssh into its WSL network from another box. I found a couple of useful guides here and here, but still ran into a few snags along the way, so thought I’d publish my configuration in case it might be useful to someone else (or perhaps even myself) in the future. Here are my steps:

Step 1: Install OpenSSH in WSL

Our first step will be to ensure openssh-server is installed. If not, issue the following and follow the prompts:

$ sudo apt install openssh-server

Step 2: Configure SSHD

Edit the file /etc/ssh/sshd_config to allow the desired users or groups. You’ll need to edit this with root access:

$ sudo vi sshd_config

To allow specific users, you can add the following with a list of users where is replaced with the desired user:

AllowUsers <user1> <user2> <userN>

To allow all users in one or more groups, add the following:

AllowGroups <group1> <group2> <groupN>

Note that these are users and groups known to WSL, not your windows users and groups (i.e. /etc/password, /etc/group).

Step 3: Setup Startup Script

Create a new file named startup-ssh.sh in /usr/local/bin with the following contents:

#!/bin/bash

WSL_IP=$(ip addr show eth0| grep -oP '(?<=inet\s)\d+(\.\d+){3}')
NETSH_CMD=/mnt/c/Windows/System32/netsh.exe
SSL_PORT=22
FIREWALL_RULE_NAME="SSH Port ${SSL_PORT}"

echo -n "Resetting port proxy settings ... "
${NETSH_CMD} interface portproxy reset all 2>&1 1>/dev/null

[ $? == 0 ] && echo "OK" || echo "Error"

echo -n "Forwarding port ${SSL_PORT} to ${WSL_IP} ... "
${NETSH_CMD} interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=${SSL_PORT} connectaddress=${WSL_IP} connectport=${SSL_PORT} 2>&1 1>/dev/null
[ $? == 0 ] && echo "OK" || echo "Error"

echo -n "Adding firewall rule if not present ... "
if ${NETSH_CMD} advfirewall firewall show rule name="${FIREWALL_RULE_NAME}" 2>&1 1>/dev/null
then
echo "OK"
else
${NETSH_CMD} advfirewall firewall add rule name="${FIREWALL_RULE_NAME}" dir=in action=allow protocol=TCP localport=${SSL_PORT} 2>&1 1>/dev/null
[ $? == 0 ] && echo "OK" || echo "Error"
fi

echo -n "Starting WSL ssh server ... "
sudo /etc/init.d/ssh start 2>&1 1>/dev/null
[ $? == 0 ] && echo "OK" || echo "Error"

This script does the following things primarily:

Forwards traffic going to port 22 from your Windows system to port 22 on the WSL virtual machine
Adds a Windows firewall rule to allow port 22 traffic
Starts the ssh server

We will be configuring Windows to execute this script on startup. The reason this is necessary is that WSL obtains a new IP address each time the system starts. This results in the need to reset the port forwarding and reapply with the latest WSL IP address.

Step 4: Configure Super User Execution

As indicated in step 5 of this guide, we need to allow the ssh command to be started without prompting for a password. We do this by editing the /etc/sudoers file. This can be done with the following command:

    $ sudo visudo

Note:

The purpose of editing the /etc/sudoers file using the visudo command is to validate the syntax before saving. You could edit the file directly, but if you screw something up then you could lock yourself out of gaining root access.

When I first used this command, it launched using the nano editor with which I’m not familiar. You can configure which editor is used by executing the following command:

    
    $ sudo select-editor

Alternately, you can set your EDITOR environment variable to the desired editor and use the following command:

    $ sudo -E visudo

Add the following as the last line of the file:

%sudo ALL=NOPASSWD: /etc/init.d/ssh

Step 5: Configure Windows Task Scheduler

Our final step will be to configure Windows Task Scheduler to launch our startup script when the system starts. Use the following steps:

Open the Task Scheduler app from the Windows Start Menu
Select Create Basic Task from the right panel
Create task with the following parameters:

Name: Start SSH Server

Description: Task to automate sshd startup

Trigger: Select When the computer starts

Action: Select Start a program

Program/script :%windir%\System32\wsl.exe Add arguments (optional): -d Ubuntu -e "/usr/local/bin/startup-ssh.sh"
Confirm everything is correct and click Finish

Step 7: Verify Configuration

Our last step is to verify we have everything configured correctly. In Task Scheduler, locate the “Start SSH Server” task and in the right panel click “Run”. If successful, you should be able to ssh from another machine to your Windows WSL virtual machine:

$ ssh dgreer@dgreer-pc

Perhaps Too Much Validation

Wed, 22 Jun 2022 08:00:00 +0000

Several factors have influenced my coding style over the years leaving me with a preference toward lean code syntax. I’ve been developing for quite a while, so it would be hard to pinpoint exactly when, where, or from whom I’ve picked up various preferences, but to name a few, I prefer: code that only includes comments for public APIs or to provide explanation of algorithms; code that is free of the use of regions, explicit default access modifiers, and unused using statements; reliance upon convention over configuration (both to eliminate repetitive tasks, but also just to eliminate unnecessary code); encapsulating excessive parameters into a Parameter Object, avoidance of excessive use of attributes/annotations (actually, I’d eliminate them completely if I could), and of course deleting dead code. There is one other practice I tend to see by other developers that I dislike and that’s too much validation.

Perhaps you’ve seen code like this:

public class MyService
{
    public DoSomething(IDependencyA dependencyA, IDependencyB dependencyB, IDependencyC dependencyC)
    {
        if(dependencyA is null)
        {
            throw new ArgumentNullException(nameof(dependencyA));
        }

        if(dependencyB is null)
        {
            throw new ArgumentNullException(nameof(dependencyB));
        }

        if(dependencyC is null)
        {
            throw new ArgumentNullException(nameof(dependencyC));
        }
    }

    …
}

Perhaps you even think this is a best practice. Is it? As with many things, the answer is really: It depends. One of the things that has greatly shaped my views on several aspects of software development over the years is adopting Test-Driven Development. The “test” part of the name is really a hold-over from adapting the practice of writing Unit Tests for driving design. With Unit Testing, you’re testing the code you’ve written. With Test-Driven Development, you’re constraining the design of the code to meet a set of specifications. It’s really quite a difference and one you may not fully appreciate unless you fully buy in to doing it for an extended period of time.

One of the side-effects of practicing TDD is that you don’t write code unless it’s needed to satisfy a failing test. The use of code coverage tools are basically superfluous for TDD practitioners, not to mention rendering far superior regression test suites. What, however, does this have to do with validation?

When driving out implementation through a series of executable specifications (i.e. an objective list of exactly how the software should work), we may end up writing code which technically could be called a certain way which would result in exceptions or logical errors, but in practice never is. As it relates to this topic, all code we write can be grouped into two categories: public and private. In this sense I’m not talking about the access modifiers we place upon the code artifacts themselves, but the intended use of the code. Is the code you’re writing going to be used by others, or is it just code we’re calling internally within our applications? If it’s code you’re driving out through TDD which others will be calling, then you should have specifications which describe how the code will react when used correctly as well as incorrectly and thus will have the appropriate amount of validation. If it isn’t code anyone else will be, or currently is calling (see also YAGNI), then the components which do call it will have been designed such that they don’t call the component incorrectly rending such validation useless.

Let’s consider our code again:

public class MyService
{
    public DoSomething(IDependencyA dependencyA, IDependencyB dependencyB, IDependencyC dependencyC)
    {
        if(dependencyA is null)
        {
            throw new ArgumentNullException(nameof(dependencyA));
        }

        if(dependencyB is null)
        {
            throw new ArgumentNullException(nameof(dependencyB));
        }

        if(dependencyC is null)
        {
            throw new ArgumentNullException(nameof(dependencyC));
        }
    }

    …
}

If this is an internal service that isn’t going to be called by any other code except other components within your application, we have 14 lines of code that are unneeded and are just adding noise to our code. I’ve worked in shops where every class in an application or library was coded this way, effectively adding hundreds to thousands of lines of unneeded code. Like regions, comments, or poorly factored code, this adds to the cognitive load required for reading through and understanding the code and ultimately is unnecessary. So the next time you reflexively start adding such validation, consider the possibility that perhaps you may be adding too much validation.

For Whom is this Container?

Wed, 01 Jun 2022 08:00:00 +0000

Several of the Messaging platforms in the .Net space have pretty rudimentary APIs (e.g. RabbitMq, Kafka) which require quite a bit of boiler-plate code to be written to get a simple message published and subscribed. You could turn to one of the Conforming Abstraction libraries such as NServiceBus or MassTransit, but perhaps you don’t really want a lowest-common denominator API, you don’t like something about how it creates the messages or topic/queue artifacts, or you simply want a fluent API expressed in terms of the native platform’s nomenclature and behavior. This might lead you down the road of creating your own KafkaBus, SQSBus, RabbitMqBus, etc. that feels like the API you wish the original development team had just provided for you to begin with. Ah, but now you have a dilemma: Frameworks such as this tend to require a number of components you’ll need to compose, many of which you may want to allow users to configure (e.g. serialization needs, consumer class conventions, logging, produce and consume pipelines, etc.). You could write hand-rolled factories, builders, singletons, etc. to facilitate the configuration and building of instances of your components, but you know that using a dependency injection container would make both development and long-term maintenance of your library much easier. But now you have another dilemma: Are you going to tie your project to some open-source container? If so, which one? Should you support a handful of the most popular ones? Should you just rely upon the Service Locator pattern and provide configuration for end users should they want to resolve from their own containers?

This was essentially the dilemma the ASP.Net Core team found themselves in when they set out to develop .Net Core. They had a fairly sizable framework with a lot of moving parts, many of which they wanted to allow the end user to configure. Earlier versions of ASP.Net MVC were built using a Service Locator pattern implementation which facilitated the ability to configure resolving from an open-source container of your choice. This, however, would have no doubt presented various design limitations in addition to the resulting lack of elegance to the resulting codebase, so the team decided to build the new platform from the ground up using dependency injection. They couldn’t, however, feasibly decide to couple their framework to one of the already mature and successful open source DI containers for various reasons. This prompted them to write their own.

One of the keys to understanding the capabilities offered by .Net Core’s container compared to other libraries is recognizing that they built it for their needs, not yours. There is no doubt that there was recognition of the usefulness for some developers to have an out-of-the-box DI container, but they didn’t set out to build a container to compete with the already extremely mature frameworks such as Autofac, StructureMap, or Ninject. For instance, because they weren’t developing user interactive client-facing applications, they didn’t have needs such as convention-based scanning registration, multi-tenancy support, the need for decorators, etc. Their needs pretty much were limited to known types with lifetime scopes of transient, singleton, or scoped per request.

Oddly, there is now a whole new generation of .Net developers which have never used a DI container other than that provided by the Microsoft Extensions suite which are missing out on being exposed to solutions to problems for which containers like Autofac, Lamar, and others facilitate fairly easily, largely I believe because no one has ever really told them: Microsoft didn’t really write that for you.

Pragmatic Deferral

Tue, 31 May 2022 13:00:00 +0000

Software engineering is often about selecting the right trade offs. While deferring feature development is often somewhat straight-forward, based upon a speculation about the return on investment, and generally decided by the customer; marketing; sales; or product people; low-level implementation decisions are typically made by the development team or individual developers and can often prove to be a bit more contentious among teams with a plurality of strong opinions. This is where principles like YAGNI (You’re Aren’t Going to Need It), or the Rule of Three have often been set forth as a guiding heuristic.

While I generally advise the teams I coach to allow the executable specifications (i.e. the tests) to drive emergent design and to defer the introduction of ancillary libraries, frameworks, patterns, and custom infrastructure, until you need it, there is a level of pragmatism that I employee when determining when to introduce such things.

I’ve been a fan of Test-Driven Development for some time now and have practiced it for over a decade. One of the primary benefits of Test-Driven Development is having an objective measure guiding what needs to get built. For example, if the acceptance criteria for a User Story concerns building a new Web API for a company’s custom B2B solution, your specs are going to drive out some sort of HTTP-based API. What the specs won’t dictate, however, are decisions such as whether to use an MVC framework, an IOC container, whether to introduce a fluent validation library or an object mapping library. Should we adhere strictly to principles like YAGNI or the Rule of Three for guidance here? My answer is: it depends.

Deferring software decisions comes with quite a range of consequences. Some decisions, such as whether to select ASP.NET MVC at the outset of a .Net-based Web application, could cause quite a bit of rework if you were to defer such a decision until working with lower-level components started to reveal friction or duplication. Other decisions, such as deferring the introduction of an object mapping library (e.g. Automapper) until the shape of the objects you’re returning actually differ from your entities essentially have only positive consequences. But how do we know?

The YAGNI principle is very similar to the firearm safety rule “The Gun is Always Loaded”. No, the gun isn’t always loaded … but it’s best to treat it like it is. Similarly, “You aren’t going to need it” doesn’t really mean you may not need it, but it’s intended to help you avoid unnecessary work. That is, until it causes more work.

In software engineering, the more you code, the more you’ll have to maintain. The Art of Not Doing Stuff, when correctly applied, can save companies as much or more money than building the right things. While I’m not religious these days, there’s a definition of the term “Hermeneutics” that I heard years ago from a Christian radio personality, Hank Hanegraaff. He would say: “Hermeneutics is the art and science of biblical interpretation”. He would go on to explain, it’s a science because it’s guided by a system of rules, but it’s an art in that you get better at it the more you do it. Having heard that explanation years ago, I have long felt these properties are equally descriptive of software development.

For myself, I take a pragmatic approach to YAGNI in that I make selections for a number of things at the outset of a new project which I’ve recognized, through experience, has resulted in less friction down the road; and I defer choices which I reason to have little to no cost by implementing at the point implementing a given User Story’s acceptance criteria drives the need. For example, I do start off setting up a Web project using ASP.NET MVC. I do set up end-to-end testing infrastructure. I do add an open source DI container and set up convention-based registration. These are things which I’ve found actually cause me more friction if I pretend I’m not going to need them. I don’t want to implement my own IHttpHandler and wait until I see the need for a robust routing and pipeline framework and have to go back and reimplement everything. I don’t want to be hand-rolling factories over and over and have to go back and modify code at the point enough duplication reveals the need for dependency injection, and I don’t want to edit a Startup.cs or other bootstrapper component each time a component has a new dependency. Outside of these few concerns, however, I do typically defer things until needed.

Magical Joy

Fri, 27 May 2022 13:00:00 +0000

In a segment of an interview with host Byron Sommardahl on The Driven Developer Podcast, recorded in the summer of 2021, Byron and I discussed a bit about a pattern I introduced to our project when we worked together in 2010 which Byron later dubbed “The Magical Joy Bus”

Nine Years Remote

Thu, 26 May 2022 13:00:00 +0000

A recent inquiry from a recruiter about accepting a partially-remote position prompted me to reflect upon 9 years of working remotely as a software developer.

When I first started working from home, attitudes were quite different than they are in today’s post COVID-19 world. Full time remote software development jobs were few and far between, and most employers that allowed working remotely full time did so due to factors other than a belief that it was more productive and cost-effective. Studies since have overwhelmingly shown that the majority were simply wrong.

One interesting side-effect of the previous year’s COVID-19 political entanglement is the degree to which it forced an entire generation of closed-minded, micro-managing executives to consider (through necessity) that remote work forces, especially for primarily thoughtwork-based positions, were not only viable, but perhaps even superior.

When our entire society started shutting down due to concerns over the COVID-19 virus, I actually hardly noticed at first. Having transitioned to full-time remote work in early 2014, I had long since become accustomed to working remotely by the time society started shutting down. Prior to landing my first full-time remote position, I had worked at a couple of prior companies which allowed working remotely a couple of days a week, so I had some notion of its viability even before then.

While I was already used to working remotely, the whole pandemic thing actually helped to improve the lives of remote developers by remedying many of the productivity nuciences that plagued fully-remote as well as mixed-teams. To a large extent, the primary issues that remote workers had to face prior to everything being shut down was the lack of remote workforce accommodations, namely: mature or provided collaboration tools (e.g Slack, Zoom, Miro, etc.) and equal participation of remote workers on mixed-teams. While David Fullerton, in a StackOverflow blog article written back in 2013, had proffered up the wisdom that “If even one person on the team is remote, every single person has to start communicating online”, joining any mixed team for many still resulted in the remote worker likely being marginalized in meetings as they were the only one on a call while all their co-workers debated approaches around a conference table, were forced to watch some white boarding design session over a video camera while you tried to make out what everyone was saying, or were simply being left out of key social interactions resulting in being professionally disadvantaged in key business decisions due to the formation of clicks, or simply not being present during unplanned discussions, etc. Conscientious employees working from home already knew they were far more efficient at home than in the office, as well as knowing that non-conscientious workers were just a likely or more so to screw off at work as they were at home, but it took everyone being forced to do it for an extended period of time to hammer than into the heads of many executives that felt uncomfortable with conducting business differently than they had in the 20th century.

One absolutely huge thing that goes seemingly undiscussed is the financial impact of working remotely vs. commuting. While I commuted to the office for 20 years before transitioning to full-time remote, it wasn’t until I had become accustomed to working from home and was confronted with the idea of returning back to the world of the commuting zombies that my perspective changed with respect to that commute time. Prior to accepting a full time remote job in early 2014, my commute time was approximately 1 hour one way, and that was on a good day when there wasn’t some minor traffic incident which could easily (and fairly regularly did) add an extra 20-30 minutes to my time. Once I had become accustomed to working remotely, the idea of tacking on an extra 5-10 hours a week in commute time to switch back to a job requiring you to work in the office seemed more like giving my time away for free. Prior to that, all those hours in the vehicle dealing with idiots on the road was just an assumed necessity. Driving to work was like driving anywhere else. Of course in the 20th century you had to drive to buy a new pair of shoes. Of course you had to drive to see a newly released movie. Of course you had to drive to go get a cheeseburger meal at McDonald’s. And of course, you had to drive to get to work. You didn’t think twice about it. You didn’t view commuting to work as 5-10 hours of your personal time given over to your employer for free for the privilege of employment any more than you’d have thought that McDonald’s owed you money for driving to their store to eat. Sure, you could listen to music, or talk radio, or a podcast, or an audio book. It wasn’t, however, really what you would have chosen to be doing at 6:30 in the morning. It wasn’t your time.

Prior to COVID, trying to explain this perspective to those still in the office world was very much like Morpheus trying to explain to Neo that he’s in the Matrix. Sure, recruiters or employers could understand the logic of an argument that commuting is time given to an employer essentially for free, but many would just think it ridiculous for you to go so far as to demand a higher salary for accepting a position requiring a commute (when you knew is wasn’t really required to do the job). This doesn’t even account for wear and tear on vehicles, gas expenses, or the little micro-batches of time you end up spending doing things like food prep, additional “get ready” time, more laundry, etc. that you wouldn’t otherwise do if you were staying home for the day. Moreover, just because you compensate someone for their time, there’s a threshold beyond which your standard hourly rate isn’t worth the time. Okay, you may be willing to commute if your employer is going to compensate you for the extra 5-10 hours on top of the 40 you’re going to spend sitting in their cube farm under their fluorescent lighting (“Not near a window, Jim, because those seats are reserved for managers!”). Are you, however, willing to exchange that extra 5-10 hours a week for money to sit in the office for 45 hours? How about 50 hours? 60? At some point, it isn’t about whether you’re compensated or not. Hell, 40 hours a week really is too damn many hours to begin with. Add to that the insane perspective on time off that Americans get on average compared to much of the rest of the developed world. Hell, even plumbers and HVAC workers get paid for their commute time, and their job isn’t something that can be done remotely.

Imagine if everyone actually accounted for these additional expenses when factoring in the pay they are willing to accept. If so, this would likely account for an extra 25-30% pay increase, accounting for time and travel expenses. For businesses on the fence about whether remote is better than on-site for their bottom line, this would certainly tip the scales. Currently, however, they aren’t forced to think this way. Or at least, many are still operating in a mindset that they don’t have to think this way. When it really comes down to it, a culture of requiring anyone that can do their job remotely to work in the office is really stealing from your employees. Fortunately, COVID has corrected this situation given there have been enough eyes opened to the benefits of remote work and enough businesses which have seen the waste that goes into buying or renting commercial real estate that, even after many business have begun attempting to force employees back into the office, there’s enough employers who now offer remote opportunities to give people a real choice.

User Stories

Wed, 25 May 2022 13:00:00 +0000

The use of User Stories has become fairly commonplace in the software industry. First introduced as an agile requirements-gathering process by Extreme Programming, User Stories arguably owe their popularity most to the adoption of the Scrum framework for which User Stories have become the de facto expression of its prescribed backlog.

So what exactly is a User Story? Put simply, they are a light-weight approach to expressing the desired needs of a software system. The idea behind User Stories, which was introduced as simply “Stories” in the book Extreme Programming Explained - Embrace Change by Kent Beck, was to move away from rigid requirements gathering processes in process, form, and nomenclature. Beck explained that the very word “requirement” was an inhibitor to embracing change because of its connotations of absolutism and permanence. At their inception, the intended form of stories was to create an index card containing a short title, simple description written in prose, and an estimation.

The Three-Part Template

In the late 1990’s, a software company named Connextra was an early adopter of Extreme Programming. In contrast to the distinct roles defined by the Scrum framework, XP doesn’t prescribe any specific roles, but is intended to adapt to existing roles within an organization (e.g. project managers, product managers, executives, technical writers, developers, testers, designers, architects, etc.).

The origin of most of Connextra’s stories were from members of their Marketing and Sales departments which wrote down a simple description of features they desired. This posed a problem for the development team, however, for when the time came to have a conversation about the feature, the development team often had difficulty locating the original stakeholder to begin the conversation. This led the team to formulate a 3-part template to help address friction resulting from ambiguous requirement sources. Their 3-part template is as follows:

	As a [type of user]
	I want to [do something]
	So that I can [get some benefit]

Ironically, while the 3-part template has become the defacto standard for authoring User Story descriptions, Scrum’s “Product Owner” role, most often filled by product development specialists acting as customer proxies, along with the use of software agile-planning tools such as Confluence, Planview, Azure DevOps Boards, etc., which captures who created a given story, tends to greatly diminish the need from which the template originated. This template has since become quite the de facto standard in expressing User Story Descriptions. The irony is that many teams, in caro-cult fashion, often utilize the 3-part template where the original need to identify the author of the story to start the conversation no longer exists. Change has occurred, but because many didn’t understand the underlying impetus for the 3-part template, they were incapable of adapting to that change.

Jeff Patton writes the following concerning the prevalent use of the 3-part story template in his book “User Story Mapping”:

“… the template has become so ubiquitous, and so commonly taught, that there are those who believe that it’s not a story if it’s not written in that form. … All of this makes me sad. Because the real value of stories isn’t what’s written down on the card. It comes from what we learn when we tell the story.”

Mike Cohn, author of many books on agile processes including “User Stories Applied” and “Agile Estimating and Planning” writes similarly:

“Too often team members fall into a habit of beginning each user story with “As a user…” Sometimes this is the result of lazy thinking and the story writers need to better understand the product’s users before writing so many “as a user…” stories.”

Cohn’s observations are spot on. In my experience, not only does this happen “too often”, it’s the rule, not the exception. It’s really just human nature. The moment a process becomes formulaic, teams will begin to just go through the motions without engaging their minds. This can be good for manual tasks like brick-laying, or cleaning a house, but it is detrimental to processes intended to promote communication. Sadly, many teams spend an inordinate amount of time on the trappings of things like ensuring their requirements follow the 3-part story template rather than using the story as a tool for its original intent: A placeholder for a conversation.

There and Back Again

While not explicitly stated, the original idea behind Stories in Extreme Programming was to facilitate a conversation, not to define an objective goal. The agile movement started as a way to address issues in the industry’s largely failing attempts to apply manufacturing processes to software development. In particular, Stories were intended to address the underlying motivation for requirements (i.e. how teams determine what to build), not to themselves be requirements.

In many ways, today’s User Stories have become the antithesis of what Kent Beck originally intended. Sadly, much of what is marketed as “agile” today has been corrupted by traditional-minded business analysts, product managers, and marketing agencies who never really understood the agile movement fully. User Stories have, to a large extent, become a casualty of these groups. We’ve gone from requirements to stories and back again. As described by Jeff Patton, “Stories aren’t a way to write better requirements, but a way to organize and have better conversations.”

The Better Way

Ultimately, the question companies seek to answer is: How do we determine the features which provide the best ROI for the business? While it may seem counterintuitive to some, customers aren’t generally the best source for determining what features to build. They can be a source, but they aren’t generally a team’s best source. Customers are, however, the best source for determining how customers currently work, what problems they face, and what friction is involved in any current processes. Various analysis techniques can be used to solicit customer opinions on desired features, but it’s best to rely upon such techniques merely as means to distill the problems currently faced by customers. From there, stories are best created with a simple title and a description of the customer’s problem written in prose with the intent for the description to serve as a starting point for a conversation with the team.

The best way to determine what to build is as a member of a mature agile team. The operative word here is mature. What makes for a mature team is a Product Owner with a background in the problem domain space, a Team Coach with deep knowledge of agile and lean processes, and 3-5 cross-functional developers weighted toward senior experience who have gone through a forming, storming, norming, and performing phase.

User Stories shouldn’t be feature requests, but rather a placeholder for a conversation. A conversation with whom? With your team. About what? About how to iteratively solve the problems you learned from customers in small steps with frequent feedback. Product Owners should not bring requirements to a development team. There’s great power in collaboration. A smart team of 5 to 7 individuals including a subject matter expert (what the Product Owner should bring to the table) and a coach are a far better source for what features to build than just the customer or the Product Owner.

An Example

The following is an example story which more closely follows the original intent of Stories.

Our scenario involves a company which provides a website allowing customers to create wedding and gift registries to send to others. In its current form, the site allows customers to pick from among existing vendors, but the company frequently receives requests from customers about specific products they’d like to see included. The current process involves the Sales team creating tickets for their Operations team to add new vendors to the site which involves updating the production database directly. Additionally, the work currently falls to one person whose job entails other operation tasks which often results in a delay to the timely fulfillment of customer requests.

The following represents the story:

Easily Manage Registry Products

Description

Our customers often want to add products that aren't part of our current vendor product list. This causes the sales team to constantly have to put in tickets and currently Margret is the only one that is working the tickets. We need a better solution!

Note how the description is written in prose (i.e. in normal conversational language), and doesn’t follow the wooden 3-part template. Note also, the story doesn’t prescribe how to solve the problem. It just provides background on what the problem is and who it affects. It isn’t just that the story doesn’t dictate implementation details, but that it doesn’t dictate the solution at all. This is the ideal starting point for most stories. It’s a placeholder for a conversation about how to solve the problem.

From here, the team would collaborate on the story to determine the best solution that results in the smallest feature increment which adds value to the end user. Several ideas may be discussed. The system could integrate with a 3rd-party content management system, allowing people within the company without SQL experience to update content. Alternately, the team may decide that adding a feature to allow customers to add custom products directly to their personal event registry is both easier, and scales far better than solutions requiring company employees to work tickets.

As part of a story refinement session, the team may update the story with acceptance criteria to guide the implementation:

Easily Manage Registry Products

Description

Acceptance Criteria

When the customer navigates to the edit registry view
  it should contain a link for adding custom products

When the customer clicks the add custom product link
  it should navigate to the add custom product view (note: see balsamiq wireframe attached)

When the customer adds a new custom product with valid inputs
  it should add the custom product to the customers registry
  it should display a success message in the application banner
  it should navigate back to the edit registry page

When the customer enters invalid custom product parameters
  it should show standard field level error messages
  it should not enable the save button

While an Acceptance Criteria section isn’t mandatory, it can often be valuable for helping to frame the scope of the story, a reminder to the team of the high-level plans discussed for deferred work, and/or may serve as the team’s Definition of Done. For small teams involving just a few members, or for highly adaptive and collaborative teams, it may be enough to just just write “We decided to add a feature to allow the customer to add their own products!”. The team may very well take the initial story description and rapidly iterate on a solution, deciding together when they think it’s done! (Gasp!) Of course, this level of informality probably is only best suited to highly cohesive, highly functioning teams. For inexperienced to moderately experienced teams, some denotation of Acceptance Criteria would be advisable. The key point is, the story didn’t arrive to the team in the form of requirements, but as a placeholder for a conversation.

Conclusion

As the adoption of agile frameworks such as Scrum have become more mainstream, a number of practices have become formulaic and adopted by teams via a cargo-cult onboarding to agile practices without truly grasping what it means to be agile. The User Story has all but lost it original intent by many teams who have done little more than slap agile labels onto Waterfall manufacturing processes. User Stories were never intended to be requirements, but rather a placeholder for a conversation with the development team. Let’s do better.

.Net Project Builds with Node Package Manager

Thu, 10 Dec 2020 07:00:00 +0000

A few years ago, I wrote an article entitled Separation of Concerns: Application Builds & Continuous Integration wherein I discussed the benefits of separating project builds from CI/CD concerns by creating a local build script which lives with your project. Not long after writing that article, I was turned on to what I’ve come to believe is one of the easiest tools I’ve encountered for managing .Net project builds thus far: npm.

Most development platforms provide a native task-based build technology. Microsoft’s tooling for these needs is MSBuild: a command-line tool whose build files double as Visual Studio’s project and solution definition files. I used MSBuild briefly for scripting custom build concerns for a couple of years, but found it to be awkward and cumbersome. Around 2007, I abandoned use of MSBuild for creating builds and began using Rake. While it had the downside of requiring a bit of knowledge of Ruby, it was a popular choice among those willing to look outside of the Microsoft camp for tooling and had community support for working with .Net builds through the Albacore library. I’ve used a few different technologies since, but about 5 years ago I saw a demonstration of the use of npm for building .Net projects at a conference and I was immediately sold. When used well, it really is the easiest and most terse way to script a custom build for the .Net platform I’ve encountered.

“So what’s special about npm?” you might ask. The primary appeal of using npm for building applications is that it’s easy to use. Essentially, it’s just an orchestration of shell commands.

Tasks

With other build tools, you’re often required to know a specific language in addition to learning special constructs peculiar to the build tool to create build tasks. In contrast, npm’s expected package.json file simply defines an array of shell command scripts:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "echo Clean the project.",
    "restore": "echo Restore dependencies.",
    "compile": "echo Compile the project.",
    "test": "echo Run the tests.",
    "dist": "echo Create a distribution."
  },
  "author": "Some author",
  "license": "ISC"
}

As with other build tools, NPM provides the ability to define dependencies between build tasks. This is done using pre- and post- lifecycle scripts. Simply, any task issued by NPM will first execute a script by the same name with a prefix of “pre” when present and will subsequently execute a script by the same name with a prefix of “post” when present. For example:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "echo Clean the project.",
    "prerestore": "npm run clean",
    "restore": "echo Restore dependencies.",
    "precompile": "npm run restore",
    "compile": "echo Compile the project.",
    "pretest": "npm run compile",
    "test": "echo Run the tests.",
    "prebuild": "npm run test",
    "build": "echo Publish a distribution."
  },
  "author": "Some author",
  "license": "ISC"
}

Based on the above package.json file, issuing “npm run build” will result in running the tasks of clean, restore, compile, test, and build in that order by virtue of each declaring an appropriate dependency.

Given you’re okay with limiting a fully-specified dependency chain where a subset of the build can be initiated at any stage (e.g. running “npm run test” and triggering clean, restore, and compile first) , the above orchestration can be simplified by installing the npm-run-all node dependency and defining a single pre- lifetime script for the main build target:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "echo Clean the project.",
    "restore": "echo Restore dependencies.",
    "compile": "echo Compile the project.",
    "test": "echo Run the tests.",
    "prebuild": "npm-run-all clean restore compile test",
    "build": "echo Publish a distribution."
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

In this example, issuing “npm run build” will result in the prebuild script executing npm-run-all with the parameters: clean, restore, compile and test which it will execute in the order listed.

Variables

Aside from understanding how to utilize the pre- and post- lifecycle scripts to denote task dependencies, the only other thing you really need to know is how to work with variables.

Node’s npm command facilitates the definition of variables by command-line parameters as well as declaring package variables. When npm executes, each of the properties declared within the package.json are flattened and prefixed with “npm_package_”. For example, the standard “version” property can be used as part of a dotnet build to denote a project version by referencing ${npm_package_version}:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "configuration": "Release",
  "scripts": {
    "build": "dotnet build ./src/*.sln /p:Version=${npm_package_version}"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

Command-line parameters can also be passed to npm and are similarly prefixed with “npm_config_” with any dashes (“-”) replaced with underscores (“_”). For example, the previous version setting could be passed to dotnet.exe in the following version of package.json by issuing the below command:

```npm run build --product-version=2.0.0```

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "configuration": "Release",
  "scripts": {
    "build": "dotnet build ./src/*.sln /p:Version=${npm_config_product_version}"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

(Note: the parameter –version is an npm parameter for printing the version of npm being executed and therefore can’t be used as a script parameter.)

The only other important thing to understand about the use of variables with npm is that the method of dereferencing is dependent upon the shell used. When using npm on Windows, the default shell is cmd.exe. If using the default shell on Windows, the version parameter would need to be deference as %npm_config_product_version%:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "configuration": "Release",
  "scripts": {
    "build": "dotnet build ./src/*.sln /p:Version=%npm_config_product_version%"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5"
  }
}

Until recently, I used a node package named “cross-env” which allows you to normalize how you dereference variables regardless of platform, but for several reasons including cross-env being placed in maintenance mode, the added dependency overhead, syntax noise, and support for advanced variable expansion cases such as default values, I’d recommend any cross-platform execution be supported by just standardizing on a single shell (e.g. “Bash”). With the introduction of Windows Subsystem for Linux and the virtual ubiquity of git for version control, most developer Windows systems already contain the bash shell. To configure npm to use bash at the project level, just create a file named .npmrc at the package root containing the following line:

script-shell=bash

Using Node Packages

While not necessary, there are many CLI node packages that can be easily leveraged for aiding in authoring your builds. For example, a package named “rimraf”, which functions like Linux’s “rm -rf” command, is a utility you can use to implement a clean script for recursively deleting any temporary build folders created as part of previous builds. In the following package.json build, a package target builds a NuGet package which it outputs to a dist folder in the package root. The rimraf command is used to delete this temp folder as part of the build script’s dependencies:

{
  "name": "example",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "clean": "rimraf dist",
    "prebuild": "npm run clean",
    "build": "dotnet pack ./src/ExampleLibrary/ExampleLibrary.csproj -o dist /p:Version=${npm_package_version}"
  },
  "author": "John Doe",
  "license": "ISC",
  "devDependencies": {
    "npm-run-all": "^4.1.5",
    "rimraf": "^3.0.2"
  

Conventional Options
Fri, 20 Nov 2020 07:00:00 +0000
I’ve really enjoyed working with the Microsoft Configuration libraries introduced with .Net Core approximately 5 years ago.  The older XML-based API was quite a pain to work with, so the ConfigurationBuilder and associated types provided a long overdue need for the platform.

I had long since adopted a practice of creating discrete configuration classes populated and registered with a DI container over direct use of the ConfigurationManager class within components, so I was pleased to see the platform nudge developers in this direction through the introduction of the IOptions type.

A few aspects surrounded the prescribed use of the IOptions type of which I wasn't particularly fond were needing to inject IOptions rather than the actual options type, taking a dependency upon the Microsoft.Extensions.Options package from my library packages, and the cermony of binding the options to the IConfiguration instance.  To address these concerns, I wrote some extension methods which took care of binding the type to my configuration by convention (i.e. binding a type with a suffix of Options to a section corresponding to the option type's prefix) and registering it with the container.

I’ve recently released a new version of these extensions supporting several of the most popular containers as an open source library.  You can find the project here.

The following are the steps for using these extensions:

Step 1
Install ConventionalOptions for the target DI container:

$> nuget install ConventionalOptions.DependencyInjection


Step 2
Add Microsoft’s Options feature and register option types:

  services.AddOptions();
  services.RegisterOptionsFromAssemblies(Configuration, Assembly.GetExecutingAssembly());


Step 3
Create an Options class with the desired properties:

    public class OrderServiceOptions
    {
        public string StringProperty { get; set; }
        public int IntProperty { get; set; }
    }


Step 4
Provide a corresponding configuration section matching the prefix of the Options class (e.g. in appsettings.json):

{
  "OrderService": {
    "StringProperty": "Some value",
    "IntProperty": 42
  }
}


Step 5
Inject the options into types resolved from the container:

    public class OrderService
    {
        public OrderService(OrderServiceOptions options)
        {
            // ... use options
        }
    }


Currently ConventionalOptions works with Microsoft’s DI Container, Autofac, Lamar, Ninject, and StructureMap.

Enjoy!


Picking a Web Microframework
Wed, 27 May 2020 00:23:00 +0000
I’ve had to use this at work the last couple of weeks. We had a “home grown” framework for a new application 
we’re working and the first thing I did was try and rip that out (new project so didn’t have URL and parameter sanitization
anyway to do routes, etc).

However, being that the group I was working with is pretty “anti framework” I had to settle on something that was light weight, integrated
with jetty and allowed us to work the way that was comfortable for us as team (also it had to work with Scala).

Microframeworks

The team had shown a lot of disdain for Play (which I had actually quite a lot when I last was leading a JVM based tech stack) and Spring Boot as being too heavy weight, so these
were definitely out.

Fortunately, in the JVM world there is a big push back now on heavy web frameworks so meant I had lots of choices for “non frameworks” but could
still do some basic security, routing, authentication but not hurt the existing team’s productivity.

There are probably 3 dozen microframeworks to choose from with varying degrees of value but the two that seemed to easiest to start with today were:


  Scalatra
  Javalin
  Quarkus


My Attempt with Quarkus

Quarkus has a really great getting started story but it’s harder to get started on an existing project with it, it was super trivial to add, and after a couple of days of figuring out the magic incantation I just decided to punt on it.
I think because of it’s popularity in the Cloud Native space (which we’re trying to target), the backing of Red Hat, and the pluggable nature of the stack there are a lot of reasons to want this to work. 
In the end because of the timeline it didn’t make the cut. But it may come back.

My Attempt with Javalin

Javalin despite being a less popular project than Quarkus it is getting some buzz. It also looks like it just slides into the team’s existing Servlet code base. I wanted this to work very badly but stopped before 
I even started because of this issue so this was out despite being on paper a really execellent framework.

My Attempt with Scalatra

Scalatra has been around for a number of years and is inspired by Sinatra which I used quite a bit in my Ruby years. 
This took a few minutes to get going just following their standalone directions and then some more to successful convert the routes
and account for learning curves with routes.

Some notes:


  The routing API and parameters etc are very nice to work with IMO.
  It was very easy to get json by default support setup.
  Metrics were very easy to wire up.
  Swagger integration was pretty rough, while it looks good on paper I could not get an example to show up, and it is unable to handle case classes or enums which we use.
  Benchmark performance when I’ve looked around the web was pretty bad, I’ve not done enough to figure out if this is real or not. I’ve seen first hand a lot of benchmarking are just wrong.
  Integration with JUnit has been rough and I cannot seem to get the correct port to fire, I suspect I have to stop using the @Test annotation is all (which I’m not enjoying).
  Http/2 support is still lacking despite being available in the version of Jetty they’re on, I’ve read a few places that an issue is keeping web sockets working but either way there is no official support in the project yet.


Conclusion

I think we’re going to stick with Scalatra for the time being as it is a muture framework that works well for our current goals. However, the lack of http/2 support maybe a deal breaker in the medium term.


Getting started with Cassandra: Data modeling in the brief
Wed, 05 Feb 2020 20:23:00 +0000
Cassandra data modeling isn’t really something you can do “in the brief” and is itself a subject that can take years to fully grasp, but this should be a good starting point.

Introduction

Cassandra distributes data around the cluster via the partition key.

CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));


In the above table the partition key is postal_code and the clustering column isid. The partition key will locate the data on the cluster for us. The clustering column allows us multiple rows per partition key so that we can filter how much data we read per partition.
The ‘optimal’ query is one that retrieves data from only one node and not so much data that GC pressure or latency issues result. The following query is breaking that rule and retrieving 2 partitions at once via the IN parameter.

SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code IN ('77002', '77043');


This can be slower than doing two separate queries asynchronously, especially if those partitions are on two different nodes (imagine if there are 1000+ partitions in the IN statement). In summary, the simple rule to stick to is “1 partition per query”.

Partition sizes

A common mistake when data modeling is to jam as much data as possible into a single partition.


  This doesn’t distribute the data well and therefore misses the point of a distributed database.
  There are practical limits on the performance of partition sizes


Table per query pattern

A common approach to optimize around partition lookup is to create a table per query, and write to all of them on update. The following example has two related tables both to solve two different queries

--query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';
--query by id
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;


You can update both tables at once with a logged batch:

BEGIN BATCH
INSERT INTO my_key.my_table (id, name, address, city, state, postal_code, country, balance) VALUES (7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 'Bordeaux', 'Gironde', '33000', 'France', 56.20);
INSERT INTO my_key.my_table_by_postal_code (postal_code, id, balance) VALUES ('33000', 7895c6ff-008b-4e4c-b0ff-ba4e4e099326, 56.20) ;
APPLY BATCH;


Source of truth

A common design pattern is to have one table act as the authoritative one over data, and if for some reason there is a mismatch or conflict in other tables as long as there is one considered “the source of truth” it makes it easy to fix any conflicts later. This is typically the table that would match what we see in typical relational databases and has all the data needed to generate all related views or indexes for different query methods. Taking the prior example, my_table is the source of truth:

--source of truth table
CREATE TABLE my_key.my_table (id uuid, name text, address text, city text, state text, postal_code text, country text, balance float, PRIMARY KEY(id));
SELECT * FROM my_key.my_table WHERE id = 7895c6ff-008b-4e4c-b0ff-ba4e4e099326;

--based on my_key.my_table and so we can query by postal_code
CREATE TABLE my_key.my_table_by_postal_code (postal_code text, id uuid, balance float, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code WHERE postal_code = '77002';


Next we discuss strategies for keeping tables of related in sync.

Materialized views

Materialized views are a feature that ships with Cassandra but is currently considered rather experimental. If you want to use them anyway:

CREATE MATERIALIZED VIEW my_key.my_table_by_postal_code 
AS SELECT postal_code text, id uuid, balance float
FROM my_key.my_table 
WHERE postal_code IS NOT NULL AND id IS NOT NULL 
PRIMARY KEY(postal_code, id));


Materialized views at least run faster than the comparable BATCH insert pattern, but they have a number of bugs and known issues that are still pending fixes.

Secondary indexes

This are the original server side approach to handling different query patterns but it has a large number of downsides:


  rows are read serially one node at time until limit is reached.
  a suboptimal storage layout leading to very large partitions if the data distribution of the secondary index is not ideal.


For just those two reasons I think it’s rare that one can use secondary indexes and expect reasonable performance. However, you can make one by hand and just query that data asynchronously to avoid some of the downsides.

CREATE TABLE my_key.my_table_by_postal_code_2i (postal_code text, id uuid, PRIMARY KEY(postal_code, id));
SELECT * FROM my_key.my_table_by_postal_code_2i WHERE postal_code = '77002';
--retrieve all rows then asynchronously query the resulting ids
SELECT * FROM my_key.my_table WHERE id = ad004ff2-e5cb-4245-94b8-d6acbc22920a;
SELECT * FROM my_key.my_table WHERE id = d30e9c65-17a1-44da-bae0-b7bb742eefd6;
SELECT * FROM my_key.my_table WHERE id = e016ae43-3d4e-4093-b745-8583627eb1fe;


Exercises

Contact List

This is a good basic first use case as one needs to use multiple tables for the same data, but there should not be too many.

requirements


  contacts should have first name, last name, address, state/region, country, postal code
  lookup by contacts id
  retrieve all contacts by a given last name
  retrieve counts by zip code


Music Service

Takes the basics from the previous exercise and requires a more involved understanding of the concepts. It will require many tables and some difficult trade-offs on partition sizing. There is no one correct way to do this.

requirements


  songs should have album, artist, name, and total likes
  The contact list exercise, can be used as a basis for the “users”, users will have no login because we’re trusting people
  retrieve all songs by artist
  retrieve all songs in an album
  retrieve individual song and how many times it’s been liked
  retrieve all liked songs for a given user
  “like” a song
  keep a count of how many times a song has been listened to by all users


IoT Analytics

This will require some extensive time series modeling and takes some of the lessons from the Music Service further. The table(s) used will be informed by the query.

requirements


  use the music service data model as a basis, we will be tracking each “registered device” that uses the music service
  a given user will have 1-5 devices
  log all songs listened to by a given device
  retrieve songs listened for a device by day
  retrieve songs listened for a device by month
  retrieve total listen time for a device by day
  retrieve total listen time for a device by month
  retrieve artists listened for a device by day
  retrieve artists listened for a device by month



Getting started with Cassandra: Load testing Cassandra in brief
Tue, 04 Feb 2020 20:23:00 +0000
An opinionated guide on the “correct” way to load test Cassandra. I’m aiming to keep this short so I’m going to leave out a lot of the nuance that one would normally get into when talking about load testing cassandra.

If you have no data model in mind

Use cassandra stress since it’s around:


  first initialize the keyspace with RF3 cassandra-stress "write cl=ONE no-warmup -col size=FIXED(15000) -schema replication(strategy=SimpleStrategy,factor=3)"
  second run stress cassandra-stress "mixed n=1000k cl=ONE -col size=FIXED(15000)
  repeat as often as you’d like with as many clients as you want.


If you have a specific data model in mind

You can use cassandra-stress, but I suspect you’re going to find your data model isn’t supported (collections for example) or that you don’t have the required PHD to make it work the way you want. There are probably 2 dozen options from here you can use to build your load test, some of the more popular ones are gatling, jmeter, and tlp-stress. My personal favorite for this though, write a small simple python or java program that replicates your use case accurately in your own code, using a faker library to generate your data. This takes more time but you tend to have less surprises in production as it will accurately model your code.

Small python script with python driver


  use python3 and virtualenv
  python -m venv venv
  source venv/bin/activate
  read and follow install docs
  if you want to skip the docs you can get away with pip install cassandra-driver
  install a faker library pip install Faker


import argparse
import uuid
import time
import random
from cassandra.cluster import Cluster
from cassandra.query import BatchStatement
from faker import Faker

parser = argparse.ArgumentParser(description='simple load generator for cassandra')
parser.add_argument('--hosts', default='127.0.0.1',
                    type=str,
                    help='comma separated list of hosts to use for contact points')
parser.add_argument('--port', default=9042, type=int, help='port to connect to')
parser.add_argument('--trans', default=1000000, type=int, help='number of transactions') 
parser.add_argument('--inflight', default=25, type=int, help='number of operations in flight') 
parser.add_argument('--errors', default=-1, type=int, help='number of errors before stopping. default is unlimited') 
args = parser.parse_args()
fake = Faker(['en-US'])
hosts = args.hosts.split(",")
cluster = Cluster(hosts, port=args.port)

try:
    session = cluster.connect()
    print("setup schema");
    session.execute("CREATE KEYSPACE IF NOT EXISTS my_key WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1}")
    session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table (id uuid, name text, address text, state text, zip text, balance int, PRIMARY KEY(id))")
    session.execute("CREATE TABLE IF NOT EXISTS my_key.my_table_by_zip (zip text, id uuid, balance bigint, PRIMARY KEY(zip, id))")
    print("allow schema to replicate throughout the cluster for 30 seconds")
    time.sleep(30)
    print("prepare queries")
    insert = session.prepare("INSERT INTO my_key.my_table (id, name, address, state, zip, balance) VALUES (?, ?, ?, ?, ?, ?)")
    insert_rollup = session.prepare("INSERT INTO my_key.my_table_by_zip (zip, id, balance) VALUES (?, ?, ?)")
    row_lookup = session.prepare("SELECT * FROM my_key.my_table WHERE id = ?")
    rollup = session.prepare("SELECT sum(balance) FROM my_key.my_table_by_zip WHERE zip = ?")
    threads = []
    ids = []
    error_counter = 0
    query = None
    params = []
    ids = []
    
    def get_id():
        items = len(ids)
        if items == 0:
            ## nothing present so return something random
            return uuid.uuid4()
        if items == 1:
            return ids[0]
        return ids[random.randint(0, items -1)]
    print("starting transactions")
    for i in range(args.trans):
        chance = random.randint(1, 100)
        if chance > 0 and chance < 50:
            new_id = uuid.uuid4()
            ids.append(new_id)
            state = fake.state_abbr()
            zip_code = fake.zipcode_in_state(state)
            balance = random.randint(1, 50000)
            query = BatchStatement()
            name = fake.name()
            address = fake.address()
            bound_insert = insert.bind([new_id, fake.name(), fake.address(), state, zip_code, balance])
            query.add(bound_insert)
            bound_insert_rollup = insert_rollup.bind([zip_code, new_id, balance])
            query.add(bound_insert_rollup)
        elif chance > 50 and chance < 75:
            query = row_lookup.bind([get_id()])
        elif chance > 75:
            zip_code = fake.zipcode()
            query = rollup.bind([zip_code])
        threads.append(session.execute_async(query))
        if i % args.inflight == 0:
            for t in threads:
                try:
                    t.result() #we don't care about result so toss it
                except Exception as e:
                    print("unexpected exception %s" % e)
                    if args.errors > 0:
                        error_counter = error_counter + 1
                        if error_counter > args.errors:
                            print("too many errors stopping. Consider raising --errors flag if this happens more quickly than you'd like")
                            break
            threads = []
            print("submitted %i of %i transactions" % (i, args.trans))
finally:
    cluster.shutdown()


Small java program with latest java driver


  download java 8
  create a command line application in your project technology of choice (I used maven in this example for no particularly good reason)
  download a faker lib like this one and the Cassandra java driver from DataStax again using your preferred technology to do so.
  run the following code sample somewhere (set your RF and your desired queries and data model)
  use different numbers of clients at your cluster until you get enough “saturation” or the server stops responding.


See complete example
package pro.foundev;

import java.lang.RuntimeException;
import java.lang.Thread;
import java.util.Locale;
import java.util.ArrayList;
import java.util.List;
import java.util.function.*;
import java.util.Random;
import java.util.UUID;
import java.util.concurrent.CompletionStage;
import java.net.InetSocketAddress;
import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.CqlSessionBuilder;
import com.datastax.oss.driver.api.core.cql.*;
import com.github.javafaker.Faker;

public class App 
{
    public static void main( String[] args )
    <


Getting started with Cassandra: Setting up a Multi-DC environment
Mon, 03 Feb 2020 20:23:00 +0000
This is a quick and dirty opinionated guide to setting up a Cassandra cluster with multiple data centers.

A new cluster


  In cassandra.yaml set endpoint_snitch: GossipingPropertyFileSnitch, some prefer PropertyFileSnitch for the ease of pushing out one file. GossipingPropertyFileSnitch is harder to get wrong in my experience.
  set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
  finish adding all of your nodes.
  if using authentication,  set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
  nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
  create your new keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above).
  nodetool repair -pr whatever_new_keyspace on each node in the cluster on the new keyspace.


An existing cluster

This is harder and involves more work and more options, but I’m going to discuss the way that gets you into the least amount of trouble operationally.


  make sure none of the drivers you use to connect to cassnadra are using DowngradingConsistencyRetryPolicy, or using the maligned withUsedHostsPerRemoteDc, especially allowRemoteDCsForLocalConsistencyLevel, as this may cause your driver to send requests to the remote data center before it’s populated with data.
  switch endpoint_snitch on each node to GossipingPropertyFileSnitch
  set dc in cassandra-rackdc.properties. Set to be whatever dc you want that node to be in. Ignore rack until you really need it, 8/10 people that use racks do it wrong the first time, and it’s slightly painful to unwind.
  bootstrap each node in the new data center.
  if using authentication,  set system_auth keyspace to use NetworkTopologyStrategy in cqlsh with RF 3 (or == number of replicas if less than 3 per dc) for each datacenter you’ve created ALTER KEYSPACE system_auth WITH REPLICATION= {'class' : 'NetworkTopologyStrategy', 'data_center_name' : 3, 'data_center_name' : 3};, run repair after changing RF
  nodetool repair -pr system_auth on each node in the cluster on the new keyspace.
  alter your app keyspaces for your app with RF 3 in each dc (much like you did for the system_auth step above),
  nodetool repair -pr whatever_keyspace on each node in the cluster on the new keyspace.


enjoy new data center

how to get data to new dc

Repair approach

Best done with if your repair jobs can’t be missed or stopped, either because you have a process like opscenter or repear running repairs. It also has the advantage of being very easy, and if you’ve already automated repair you’re basically done.


  let repair jobs continue..that’s it!


Rebuild approach

Faster less resource intensive, and if you have enough time to complete it while repair is stopped. Rebuild is easier to ‘resume’ than repair in many ways, so this has a number of advantages.


  run nodetool rebuild on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.
  run nodetool cleanup


YOLO rebuild with repair

This will probably overstream it’s share of data and honestly a lot of folks do this for some reason in practice:


  leave repair jobs running
  run nodetool rebuild on each node in the new dc only, if it dies for some reason, rerunning the command will resume the process.
  run nodetool cleanup on each node


Cloud strategies

There are a few valid approaches to this and none of them are wrong IMO.

region == DC, rack == AZ

Will need to get into racks and a lot of people get this wrong and imbalance the racks, but you get the advantage of more intelligent failure modes, with racks mapping to AZs.

AZ..regardless of region == DC

This allows things to be balanced easily, but you have no good option for racks then. However, some people think racks are overated, and I’d say a majority of clusters run with one rack.


MVP how minimal
Thu, 20 Dec 2018 20:00:00 +0000
MVPs or Minimum Viable Products are pretty contentious ideas for something seemingly simple. Depending on background and where pepole are coming from experience wise those terms carry radically different ideas.
In recent history I’ve seen up close two extreme constrasting examples of MVP:


  Mega Minimal: website and db, mostly manual on the backend
  Mega Mega: provisioning system, dynamic tuning of systems via ML, automated operations, monitoring a few others I’m leaving out.


Feedback

If we’re evaluating which approach gives us more feedback, Mega Minimal MVP is gonna win hands down here. Some will
counter they don’t want to give people a bad impression with a limited product and that’s fair, but it’s better than no impression (the dreaded never shipped MVP). The Mega Mega
MVP I referenced took months to demo. only had one of those checkboxes setup and wasn’t ever demod again. So we can categorical say that failed 
at getting any feedback.

Whereas the Mega Minimal MVP, got enough feedback and users for the founders to realize that wasn’t a business for them.
Better than after hiring a huge team and sinking a million plus into dev efforts for sure. Not the happy ending I’m sure you all were expecting, but I view that as mission accomplished.

Core Value


  Mega Minimal, they only focused on a single feature, executed well enough that people gave them some positive feedback, but not enough to justify automating everything.
  Mega Mega. I’m not sure anyone who talked about the product saw the same core value, and there were several rewrites and shifts along the way.


Advantage Mega Minimal again

What about entrants into a crowded field

Well that is harder and the MVP tends to be less minimal, because the baseline expectations are just much higher. I still lean towards Mega Minimal having a better chance at getting users, since there is a non zero chance the Mega Mega MVP will never get finished.
I still think the exercise in focusing on core value that makes your product not a me too, and even considering how you can find a niche in a crowded field
instead of just being “better”, and your MVP can be that niche differentiator.

Internal users

Sometimes a good middle ground is considering getting lots of internal users if you’re really worried about bad experiences. This has it’s
it’s definite downsides however, and you may not get diverse enough opinions. But it does give you some feedback while saving some face or bad experiences. I often think of the example of EC2 that was heavily used by Amazon, before being released to the world.
That was a luxury Amazon had, where their customer base and their user base happened to be very similar, and they had bigger scale needs than any of their early customers, so the early internal feedback loop was a very strong signal.

Summary

In the end however you want to approach MVPs is up to you, and if you find success with a meatier MVP than I have please don’t let me push you away from what works. But if you are having trouble shipping and are getting pushed all the time to add one more feature to that MVP before releasing it, consider stepping back and asking is this really core value 
for the product? Do you already have your core value? if so, consider just releasing it.


Collaboration vs. Critique
Fri, 18 May 2018 17:00:00 +0000
While there are certainly a number of apps developed by lone developers, it’s probably safe to say that the majority of professional software development occurs by teams.  The people aspect of software development, more often than not, tends to be the most difficult part of software engineering.  Unfortunately the software field isn’t quite like other engineering fields with well-established standards, guidelines, and apprenticeship programs.  The nature of software development tends to follow an empirical process model rather than a defined process model.  That is to say, software developers tend to be confronted with new problems every day and most of the problems developers are solving aren’t something they’ve ever done in the exact same way with the exact same toolset.  Moreover, there are often many different ways to solve the same problem, both with respect to the overall process as well as the implementation.  This means that team members are often required to work together to determine how to proceed.  Teams are often confronted with the need to explore multiple competing approaches as well as review one another’s designs and implementation.  One thing I’ve learned during the course of my career is that the stage these types of interactions occur within the overall process has a significant impact on whether the interaction is generally viewed as collaboration or critique.

To help illustrate what I’ve seen happen countless times both in catch-up design sessions and code reviews, consider the following two scenarios:

Scenario 1
Tom and Sally are both developers on a team maintaining a large-scale application.  Tom takes the next task in the development queue which happens to have some complex processes that will need to be addressed.  Being the good development team that they are, both Tom and Sally are aware of the requirements of the application (i.e. how the app needs to work from the user’s perspective), but they have deferred design-level discussions until the time of implementation.  After Tom gets into the process a little, seeing that the problem is non-trivial, he pings Sally to help him brainstorm different approaches to solving the problem.  Tom and Sally have been working together for over a year and have become accustomed to these sort of ad-hoc design sessions.  As they begin discussing the problem, they each start tossing ideas out on the proverbial table resulting in multiple approaches to compare and contrast.  The nature of the discussion is such that neither Tom nor Sally are embarrassed or offended when the other points out flaws in a given design idea because there’s a sense of safety in their mutual understanding that this is a brainstorming session and that neither have thought in depth about the solutions being set forth yet.  Tom throws out a couple of ideas, but ends up shooting them down himself as he uses Sally as a sounding board for the ideas.  Sally does the same, but toward the end of the conversation suggests a slight alteration to one of Tom’s initial suggestions that they think may make it work after all.  They end the session with a sense that they’ve worked together to arrive at the best solution.

Scenario 2
Bill and Jake are developers on another team.  They tend to work in a more siloed fashion, but they do rely upon one another for help from time to time and they are required to do code reviews prior to their code being merged into the main branch of development.  Bill takes the next task in the development queue and spends the better part of an afternoon working out a solution with a basic working skeleton of the direction he’s going.  The next day he decides that it might be good to have Jake take a look at the design to make him aware of the direction.  Seeing where Bill’s design misses a few opportunities to make the implementation more adaptable to changes in the future, Jake points out where he would have done things differently.  Bill acknowledges that Jake’s suggestions would be better and would have probably been just as easy to implement from the beginning, but inwardly he’s a bit disappointed that Jake didn’t like his design as-is and that he has to do some rework.  In the end, Bill is left with a feeling of critique rather than collaboration.

Whether it’s a high-level UML diagram or working code, how one person tends to perceive feedback on the ideas comprising a potential solution has everything to do with timing. It can be the exact same feedback they would have received either way, but when the feedback occurs often makes a difference between whether it’s perceived as collaboration or critique.  It’s all about when the conversation happens.