Dana Lawson on Kubernetes, Local Development Loops, and Constant Learning

In the sixth episode of the Ambassador Livin’ on the Edge podcast, Dana Lawson, VP of Engineering at GitHub, discusses the impact that Kubernetes has had on operations, how cloud has impacted the local development experience, and the benefits of constant learning.Be sure to check out the additional episodes of the "Livin' on the Edge" podcast.

Key takeaways from the podcast included:

• There is no single “best” software architecture. Engineers must work hard to understand the required business outcomes, the problem space, the current constraints, and future goals of an organization, and design accordingly.
• Automating as much of the continuous delivery process as possible typically provides a high return on investment. If it is difficult to deploy functionality using the official mechanisms, there is a danger that some engineers may attempt to circumvent the sanctioned approach and create their own bespoke deployment methods.
• Kubernetes provides a lot of value in running and maintaining production environments, but the local developer experience could be better.
• A local development environment should mimic production as much as possible. There should be an easy and effective way to migrate new functionality from development to production, allowing incremental testing (e.g. unit and integration testing locally, contract testing via pipelines, and incremental traffic shifting via canary releases in production)
• Tooling that bridges Kubernetes development and production environments, such as Telepresence or Ksync, can be useful when adapting developer workflows because of a move to the cloud and containers.
• Coding “in the cloud”, for example using an IDE running on a remote VM, may be useful for certain engineering teams, environments, and problems [editor's note: this podcast was recorded before GitHub announced Codespaces]
• Although cloud technologies provide engineers with many deployment options, they should aim to keep solutions simple. Adding complexity increases the opportunity for error and system failure.
• Care should be taken with cost management when moving to the cloud. Adopting cloud technologies can provide many benefits, but replicating existing development practices e.g. multiple staging environments or creating a remote development environment for each engineer, can become prohibitively expensive.
• Engineers will benefit from constantly learning about and exploring new technologies. Technical leaders should encourage this, but also channel efforts appropriately. Sometimes the latest shiny new technologies will provide value, but other times this can simply become a distraction.
• Technical leaders should keep an eye on developer communities, such as HackerNews and Reddit, and this often provides clues as to what the current hot technologies are (for better or worse!)
• Engineers will benefit from understanding the business context they are working in. Developing empathy for the customer is a vital skill (even if the customer is a fellow engineer or a dependent delivery team). This is arguably even more important when working in a startup environment, as the entire team is responsible for finding product-market fit.
• Developers will also benefit from learning more about the basics of infrastructure and operations. However, they must balance their time appropriately, and think about their current learning in relation to the needs of their current company and their own personal future careers goals.