antrix.net

Entropy Containment Engineer

2026-03-01T17:40:00+08:00

I can imagine a fairly ordinary Tuesday a few years from now.

A compliance lead notices a regulator has tightened rules around “high-risk accounts.” She opens an internal admin tool, writes what she wants in plain language, and an agent generates the change: a new check, a UI nudge, an audit record, a couple of dashboards, a feature flag. It rolls out slowly, watches itself, and keeps going.

Across the office, a support manager is tired of a refund workflow that forces three manual steps and produces inconsistent outcomes. He describes the intent, the agent threads the needle through a messy back-office system, adds guardrails, and ships the fix.

A marketer wants a different onboarding path for a specific campaign. She asks for two screens and a new event. The agent wires it up end to end.

None of this is “deep engineering,” but it’s still real engineering surface area: money, customer data, risk controls, and downstream obligations. The novelty is not that changes can be made; it’s that they can be made all day, by many people, without queuing behind a scarce engineer.

There have already been plenty of essays about this shift – domain specialists using AI to build, the impact on software company moats, on margins, on valuations, on who captures value in the economy. All of that is interesting and probably important.

One aspect I haven’t heard discussed as much is more operational and less glamorous: when this shift actually happens inside a real company, how do we stop the system from gradually degrading as the rate of change goes up by 100× or 1000×?

If you step back and look at the system as a whole, that gradual breakdown of order has a name: Entropy.

Entropy, in software terms (and why we’ve always fought it)

Entropy in a codebase isn’t a bug count. It’s the gap between the system’s intended shape and what it becomes after years of local decisions that drag it away from that ideal shape.

Every working system has an intended shape, even when nobody wrote it down. There are boundaries (what belongs where), invariants (what must remain true), and meanings (what a “customer,” a “refund,” a “risk flag” actually is). When those meanings stay crisp, the system stays legible: people can predict consequences, reuse parts safely, and make changes without constantly rediscovering hidden rules. When meanings blur, the system can still be changed for a while, but the cost shows up as bugs, performance issues, and an increasing fear of changing anything.

This is not a new problem. It’s what “software maintenance” has always been!

When we do “software maintenance”, we are reducing entropy.

Refactoring is entropy reduction. Dependency upgrades are entropy reduction. Cleaning up duplicated business rules, untangling accidental dependencies, fixing CI so it behaves predictably, keeping runbooks in shape, it’s all the same family of work: restoring the intended shape so the next change is still possible.

“Software Maintenance” has always been a part of software engineering. In healthy companies (and healthy systems), we find time to do maintenance regularly. However, it’s not the primary activity: that’s still shipping features and shipping value.

Under 1000× change, entropy management becomes the job

If domain specialists can ship changes with AI agents, the volume and frequency of modification changes character. The codebase stops evolving in bursts around sprint boundaries and starts evolving continuously, with many hands applying small, locally rational adjustments throughout the day.

With 100× or 1000× more change, treating entropy reduction as a side activity stops being viable. The curve outruns you. So the question becomes: who does the entropy work when almost anyone can generate new code and ship new behavior?

The answer is still “software engineers,” but the role changes. Engineers stop being primarily the people who implement most business features, and become the people responsible for the system’s coherence while change hits it from all sides. They still build, but their highest leverage work moves toward the machinery that keeps the system understandable, stable enough to evolve, and resistant to slow degradation.

The tools we have, and the tools we need

Some of the answer is already sitting inside software engineering. We know how to test systems, how to control releases, how to refactor, how to encode constraints. We simply have not had to apply these disciplines with the level of consistency that 1000× change will demand.

At the same time, not everything we need lives inside the traditional SDLC toolbox. If software starts to resemble other large systems under constant, loosely coordinated change, then we should expect to borrow ideas from those systems as well.

Three software engineering practices that move from optional to essential

Before looking elsewhere, it’s worth being honest about how much unused leverage already exists inside our own discipline. The practices below are familiar. What changes is not the concept, but the intensity and consistency with which they are applied.

Continuous Refactoring, industrialized

Refactoring is one of the cleanest entropy reduction tools we have. The problem is scale. As a craft activity done occasionally by humans, it cannot keep up with a world where the codebase is being modified constantly.

In a high-change environment, refactoring needs to look less like an annual cleanup and more like continuous background maintenance. Imagine an army of AI agents that engineers run and supervise: shrinking complexity hotspots, deduplicating business rules, deleting dead code, keeping dependencies fresh, and nudging the codebase back toward a small set of consistent patterns.

Engineers define what “healthy” means, constrain the agents so they do not introduce risk, and decide when automated cleanup should give way to human judgment. Entropy reduction becomes continuous rather than episodic.

Error Budgets + Progressive Delivery

Error budgets were popularized by Google’s SRE practice, but adoption remains uneven. Under 1000× change, they become central.

The mechanism is straightforward. You set a Service Level Objective. You track performance against it. When you overspend your error budget, you slow down or pause new changes and focus on restoring stability until you are back within bounds.

Pair that with disciplined release engineering: staged rollouts, canaries, automatic rollback when metrics move in the wrong direction, and monitoring that is designed alongside the change itself rather than after it. Add production-like simulation and regular chaos exercises so fragility is discovered deliberately.

The underlying idea is not to prevent change, but to control its blast radius and respond quickly when reality disagrees with the plan. Engineers build and tune this feedback system so that high change velocity does not automatically translate into high instability.

Architectural Fitness Functions + Policy-as-Code

Earlier, we talked about boundaries, invariants, and meanings. Those define the system’s intended shape. Over time, the quiet risk is drift: boundaries becoming porous, invariants being weakened, meanings diverging across modules.

Today, much of our defense against that drift is human code review. That depends on attention and memory, and it does not scale when changes arrive continuously from many sources.

If we want to preserve intent at 1000× scale, the checks must be automatic.

Architectural fitness functions encode structural intent directly: which modules may depend on which, what invariants must always hold, what kinds of coupling are forbidden. Policy-as-code applies the same principle to operational and business rules: logging constraints, audit requirements, sensitive state transitions.

The system’s constitution is expressed as executable rules. Engineers spend less time reviewing every change and more time engineering the checks that preserve boundaries, invariants, and meanings at scale.

Three ideas we need to borrow from other complex systems

There are other systems that live under constant, loosely coordinated change and would degrade quickly if left unmanaged.

Cities evolve through thousands of independent decisions each day, yet remain livable because of zoning, codes, and inspection regimes. Living organisms grow and adapt, yet remain viable because growth is paired with removal. Markets coordinate countless actors, yet avoid collapse because pricing introduces friction when shared resources are stressed.

If software systems begin to experience similar levels of uncoordinated change, it is reasonable to borrow from these mechanisms.

Stable Core, Fast Edge (zoning)

A useful architectural principle is to keep a stable core and push innovation to the edges.

The core is where the most expensive invariants live: identity, permissions, ledger rules, durable state, risk calculations. The edge is where variation belongs: UI flows, routing logic, experiments, workflow glue.

Under modest rates of change, that separation can remain informal. Under 1000× change, it cannot. Not all edge changes are equal. Some are superficial. Others reach back into core assumptions through shared data models or hidden coupling.

Cities address similar gradients of risk through zoning. Different areas carry different rules and levels of scrutiny. The same idea applies in software. A copy tweak can move quickly. A workflow change touching billing requires stronger verification. A direct change to the ledger or permission model belongs in the tightest zone.

Zoning turns an architectural idea into an enforceable structure. Engineers define the zones, specify what crosses boundaries, and integrate the checks into the delivery pipeline so variability concentrates where it is cheap and stays contained where it is dangerous.

Built-in Death (feature expiry as default)

A large share of software complexity is accumulated history. Features remain long after their context has faded. Exceptions introduced under pressure settle into the baseline.

Biological systems pair growth with removal. Cells are created and cleared out. Tissue is repaired and damaged parts are broken down. Health depends on maintaining that balance over time. Biologists call this programmed cell death, or apoptosis.

Software rarely enforces that balance. By default, it remembers everything, which means the surface area of the system only expands and each new change must navigate old intent.

A practical response is to make expiry the default. New features and experiments ship with an explicit end date. They are renewed if they prove value; otherwise they are removed. Deletion becomes part of the normal lifecycle rather than a rare, anxious cleanup.

This is entropy control in practice. Engineers make it feasible by building visibility into unused paths, safe migration tooling, reliable rollback, and release processes that treat deletion as a first-class change.

Pricing the Cost of Change

When many people can change a shared system, the tragedy of the commons appears quickly. Each change may be reasonable in isolation, yet the shared system slowly absorbs disorder if the cost of adding complexity is invisible.

Markets address this by using pricing to reflect scarcity. When a shared resource is stressed, the cost of consuming it rises, introducing friction that protects the system as a whole.

Singapore’s Certificate of Entitlement (COE) system is one concrete example. The number of cars allowed on the road is capped, and the right to own one is allocated through bidding. When demand increases, prices rise, reflecting limited capacity and preventing congestion from overwhelming the system.

Software under 1000× change faces a similar constraint. The scarce resource is system coherence. We need to price the cost of change.

That price does not have to be monetary. It can mean fewer release windows, stricter automated checks, mandatory review, or reduced access to the most sensitive zones without sponsorship. It can also influence performance evaluation. If someone consistently improves the system while shipping value, that should count. If someone repeatedly injects entropy without benefit, that should also be visible.

The point is not punishment. It is alignment. Engineers design the measurement, attribution, and enforcement so that the long-term health of the system is reflected in the day-to-day economics of making change.

The title is slightly silly, but the job is real

“Entropy Containment Engineer” is not a serious title. But I think it’s a useful one, because it points at a real inversion.

If AI agents make feature creation cheap, then the bottleneck moves. It is no longer the ability to produce change. It is the ability of the system to absorb change without losing its shape, without losing coherence: clean boundaries, stable meanings, and a system that can keep evolving without turning into a mess.

Engineers become the people who build and operate the machinery that resists drift. That fight the entropy.

Continuous refactoring that runs in the background so complexity does not quietly accumulate. Error budgets and progressive delivery that treat stability as a managed resource. Fitness functions and policy-as-code that encode boundaries, invariants, and meanings so they do not drift. Zoning that recognizes gradients of risk rather than pretending everything is either core or edge. Built-in expiry so growth is paired with removal. Pricing mechanisms that make the tragedy of the commons visible in day-to-day decisions.

In the agent-assisted era, the hard part will not be producing change. It will be keeping the system coherent as change becomes abundant.

The engineers who matter most will be the ones who design and operate that entropy fighting machinery.

The System of Record Fallacy

2026-02-16T20:01:00+08:00

I was having dinner with a VC friend and we ended up talking about the meltdown in SaaS valuations. When discussing SaaS, I often hear this logic: AI will commoditize a lot of software. But the Systems

I was at dinner recently with a VC friend and we ended up talking about the meltdown in SaaS valuations. When discussing SaaS, I often hear this logic: AI will commoditize a lot of software. But the Systems of Record are safe. If you are the SoR, you have a moat & valuations will certainly recover.

I’m very skeptical of this logic: System of Record → Defensible valuation.

What’s a System of Record

A SoR is one where the authoritative write happens. The primary store of truth for a domain. The ledger, the CRM master, etc.

Yes, those are critical and risky to replace but that’s not the only reason why they are valuable!

Look at your own company. Your actual SoR is probably a transactional database. Postgres, SQL Server, etc.

But no serious architecture stops there. You replicate data, reshape and repurpose it. You build warehouses and lakes because your OLTP database is not designed for aggregations, ML feature extraction, or complex reporting.

This is why Databricks exists at its current scale. Databricks isn’t anyone’s SoR. It sits downstream, consumes replicated data, and clearly commands a massive valuation. The value is in participation, not custody.

Which suggests the story is more complicated.

A SoR is one node in a broader graph of systems. Analytics engines. Workflow tools. They all exist because the primary store cannot and should not serve every read pattern or operational use case.

Now add AI.

If AI agents become the orchestrators of workflows, they will sit above many of these systems and interact with them programmatically. From the outside, that can make the SoR look like a commoditized state backend.

Commoditized! That’s a word inversely correlated with valuation multiples. :)

If you’re a SaaS vendor and you sense that risk, a defensive reaction is understandable. Tighten APIs, constrain access, and rate limit aggressively.

Essentially make autonomous agents harder to integrate. Preserve control over the primary interface so you’re not reduced to dumb store. Some vendors are already doing some of this.

But as we just discussed, the value customers extract from a SoR rarely sits inside the SoR box itself. It emerges when that data participates in a broader system. When it flows into the other nodes.

The whole is greater than the sum of the parts.

If you restrict access in order to defend a moat, you are shrinking the surface area through which that excess value is created. And customers will notice that. They will feel that the system is less flexible, less compounding, less future-proof.

At that point, you are not just defending a moat. You are forcing your customers to ask whether the box is still worth the constraints.

None of this means systems of record are unimportant. They clearly matter.

It just means that “is it a system of record or not” is the wrong question.

The right question: does this system become more valuable as the graph around it grows? Or does it depend on restricting that graph to preserve its position?

Agent Context Graphs: Color Me Skeptical

2026-01-26T20:10:00+08:00

Over the past few weeks, I’ve heard repeated mentions of this idea of “enterprise context graphs” paired with AI agents.

One representative description goes like this:

”.. the missing layer that actually runs enterprises: the decision traces – the exceptions, overrides, precedents, and cross-system context that currently live in Slack threads, deal desk conversations, escalation calls, and people’s heads. […] Once you have decision records, the ‘why’ becomes first-class data. Over time, these records naturally form a context graph: the entities the business already cares about (accounts, renewals, tickets, incidents, policies, approvers, agent runs) connected by decision events (the moments that matter) and ‘why’ links.”

What’s interesting, and perhaps somewhat telling, is that the only people I’ve seen talk about these “context graphs” are VCs, not engineers.

I’ve seen firsthand three attempts to create a unified data & domain model across just one slice of an enterprise fail after years of effort. Many CTOs I know have similar stories. We all have scars from this.

The reason is simple. It is genuinely hard to unify the complexity of a real company into a clean data model. Hard enough to capture end-to-end what a large company does, let alone why!

So I remain skeptical that such a “context graph” can be built in any clean or durable way.

But wait. The argument is that AI agents change the equation.

Because agents sit directly within business workflows, the claim is that they are naturally exposed to the “why,” and can capture decision traces as decisions are made, allowing a context graph to emerge organically over time.

To quote again:

“When an agent triages an escalation, responds to an incident, or decides on a discount, it pulls context from multiple systems, evaluates rules, resolves conflicts, and acts. The orchestration layer sees the full picture: what inputs were gathered, what policies applied, what exceptions were granted, and why. Because it’s executing the workflow, it can capture that context at decision time – not after the fact via ETL, but in the moment, as a first-class record.”

I find this logic suspect, because the hard problem here is not data capture. It is representation.

Capturing traces and rationales is one thing. Representing them in a structured graph of entities and relationships is where things repeatedly break down. That modeling problem doesn’t disappear just because an agent is in the loop.

And if the claim is that structure doesn’t really matter because AI can figure it out anyway, then it’s not clear why we need to talk about a “context graph” at all! AI should already be able to infer what it needs from Slack threads, deal desk conversations, escalation calls, and people’s notes.

I can’t help but think that this is less about a real, unmet need, and more about naming and inflating a new startup category that glosses over some very old, very hard problems.

Sravya, the Genesis

2025-08-24T14:01:00+08:00

A Spark at the Stanford Executive Program

June 2025 was a turning point. I found myself in the thick of the Stanford Executive Program, six weeks of full-time classes on the beautiful Stanford campus. Strategy, leadership, culture, communications, AI, and more. It was like a compressed MBA, taught by some of the best minds in the world. It was intense and exhilarating.

And the reading. My god, the reading. Every class came with loads of it! Case studies. Book chapters. Sometimes more than one per session. Enough to bury anyone.

But here’s the thing. I don’t really “read” anymore. I used to read a lot of books and long form content. But then life got too busy, and I found it increasingly hard to sit down and read at length. These days I listen instead of reading. Podcasts, audiobooks, interviews, even research papers if I can get them in audio. I listen while jogging, commuting, making coffee.

So staring at this mountain of reading at Stanford, I wondered, what if I didn’t read any of it? What if I could listen to it instead? Could I turn every case study and assigned chapter into a personal podcast feed?

I had been deeply immersed in the evolution of Generative AI since 2023 and I knew that building an app that did exactly this was possible using the current state of AI. But where would I find the time to develop this app?

Vibe Coding

A few weeks before Stanford, back at work, a colleague showed me Google Firebase’s App Prototyping agent. You simply describe what you want and it generates a working app. It’s called vibe coding, a term coined by Andrej Karpathy in early 2025:

“There’s a new kind of coding I call ‘vibe coding,’ where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.”

The idea is to let the AI write the code, test it, fix it, and iterate till it works. You guide it with prompts, course-correct where needed, and accept the messiness. You never read the actual code!

I had been thinking about vibe coding since then. Could this really work beyond toy demos? Were the critics right about fragile architectures, spaghetti code, security holes waiting to happen? I wanted to know for myself which made this the perfect experiment; kill two birds with one stone. Try vibe coding, and maybe solve my reading overload at Stanford!

The Vision: Auto-Podcastify My Stanford Readings

The idea took shape. Build an app that converts assigned readings into a podcast. Drop in a PDF or a chapter, get back a narrated audio file. Stitch them together, and I’d have my own SEP podcast feed to queue on a morning run.

This was the genesis of Sravya: an app that let’s you Read Less, Listen More.

It sounded simple, but building this would be anything but trivial. I didn’t want the app to create dumb read-aloud versions of text. It couldn’t be your basic text-to-speech. It needed to be smart to not read boilerplate headers and footers. It needed to know how to handle inherently visual artifacts like charts and illustrations. It had to sound good - like it was really meant to be heard as audio.

AI could probably get me there, but only if I told it clearly what I wanted. With these systems, vague instructions give you unusable results. Garbage in, garbage out.

Writing the Spec

So I needed a product specification first. I began by talking to ChatGPT, using voice dictation to dump my messy, half-formed ideas. The AI turned my ramblings into a rough product spec, something that started to look like real requirements. It listed features, inputs and outputs, a user flow, even some edge cases. I was vibe speccing.

Of course it wasn’t perfect. I had to push back, refine, add detail, re-explain things I hadn’t been clear about. We went back and forth. Each round improved the spec until it started to feel like something I could actually build from. When it felt close, I asked ChatGPT to break it into milestones. I knew from experience that if you throw a huge, complex project at these AI tools all at once, they choke. But if you build step by step, they have a fighting chance.

Armed with a spec and a set of milestones, I opened up Firebase Studio and got to vibe coding.

Coming Next

In the next post, I’ll walk through what actually happened inside Firebase.

What did the tool build?
What surprised me?
Where did vibe coding shine, and where did it trip?

To be continued …

Forever Thirty

2025-01-11T18:01:00+08:00

Aurora adjusted her e-lenses, the protest below snapping into sharp relief. From the hundred-and-twentieth floor, the chanting was a low thrum, the signs held aloft like desperate pleas to indifferent gods. LONG LIFE FOR ALL. STOP THE MONOPOLY. The same demands, echoing daily against the mirrored facades of the Big Four, a quartet of megacorporations whose tendrils wrapped around every facet of civilization. Helion, the largest, loomed dominant, its name a byword for both innovation and ruthless control – particularly when it came to GeneLife’s anti-aging therapy.

A sigh escaped her lips, misting briefly in the recycled air of her private balcony. Inside Helion’s walls, time was a commodity, carefully rationed. Employees received their annual rejuvenation cycle, a precise cocktail of gene therapies that arrested aging, granting them extra decades, even centuries. Veterans, some well over two hundred, moved with the practiced efficiency of seasoned professionals in their prime. Decades, even centuries, spent within Helion had forged an unrivaled depth of institutional memory, a fortress of knowledge. No upstart competitor – constantly losing talent both to the constraints of a natural lifespan and to the irresistible draw of the Big Four’s life extension – could hope to overcome such accumulated expertise. Smaller companies withered, their brightest minds inevitably drawn to the gilded cage of extended life, leaving behind skeletal remains of forgotten industries. Venture capital dried up for any enterprise not blessed by a Big Four partnership – who would invest in a future measured in mere decades when centuries were on offer?

Ten years ago, a ripple of hope had spread among the masses barred from the therapy, those condemned to the brevity of a normal lifespan. A group of independent scientists, fueled by ethical outrage, had dared to pursue their own research. Helion’s legal eagles descended like harpies, the ensuing lawsuits protracted and brutal. The scientists, denied the very therapy they sought to democratize, now lived their dwindling, accelerated lifespans in frustrated obscurity, a stark warning to any would-be challengers. Whispers of alternative research now vanished like smoke in the wind, snuffed out by the sheer economic gravity of the Big Four.

Aurora retreated inside, the balcony doors hissing shut, muffling the distant cries. She filed her daily reports – the same incremental progress, the same layers of bureaucratic approval. A quiet rebellion flickered within her: the yearning to build, to create something beyond Helion’s meticulously controlled ecosystem. But the thought felt as flimsy as the protesters’ cardboard signs against the monolithic power she served. Who would trade a guaranteed 200-year lifespan for the uncertain trajectory of a dream?

She caught her reflection in the polished chrome of a passing service bot. The same smooth skin, the same untroubled eyes she’d had since her thirtieth birthday, four decades ago. “Forever Thirty,” she murmured, the old joke tasting like ash in her mouth. It was a promise, a perk, a prison. The chants outside seemed to grow louder in the sterile silence of the corridor, each syllable a tiny hammer blow against the walls of her carefully constructed reality. As she walked towards her next meeting, the pervasive hum of Helion a constant companion, the unspoken truth settled heavy in her chest: “forever” wasn’t a gift; it was a leash, and the protesters, for all their perceived naiveté, might be the only ones seeing the cage for what it truly was.

Large Language Models: Changing the Future of Communication

2023-11-04T08:01:00+05:30

Every significant stride in human history has hinged on one key factor: communication. Our world, our societies, and our cultures have been shaped by our ability to connect and share ideas, stories, and knowledge. The advent of computers and the internet supercharged this exchange, facilitating unprecedented levels of global interaction.

But these interactions have largely been human-driven. As computers can’t independently process or understand ambiguous human language(s), they need precise instructions to ‘communicate’. This gave birth to programming, a complex language for ‘talking’ to machines.

The stark contrast between human communication, with its richness and adaptability, and the rigid structure of machine communication became more evident as our dependence on technology grew. But the emergence of Large Language Models (LLMs) is gradually bridging this gap, making computers capable of adapting their communication in a more human-like way.

Humans learn by experiencing, observing, and replicating. Early AI research sought to emulate this process but faltered due to the paucity of data and computational power. The AI domain then shifted focus to task-specific learning models, such as fraud detection and image recognition. However, the surge of data from the internet and the advent of GPUs – initially designed for gaming, but exceptionally apt for data processing – revived the ‘learning by example’ approach, leading to the creation of LLMs.

LLMs, fundamentally, are software capable of interacting through various mediums - text, voice, images, and video - mimicking the ambiguity and adaptability characteristic of human communication. As humans use foundational communication skills to learn, garner feedback, and improve upon other skills, LLMs can be directed to do the same. Instead of creating task specific AI, we can take a LLM that is capable of generalised communication and teach it task specific skills. This opens up significant opportunities to streamline and automate many tasks and processes that currently require extensive custom software development.

This paradigm shift heralds a revolution in communication: human-to-computer, human-to-human, and computer-to-computer.

Human-to-Computer Communication

LLMs promise a future where we don’t need to meticulously instruct computers on every task, thereby eliminating the necessity of programmers for each task. Picture using Excel: remember the thrill of employing a function to automate a process? Now imagine applying this excitement to every part of your life. By describing your needs in plain language and providing a few examples, an LLM assistant can become your digital butler, ready to carry out your requests. It’s an era of human augmentation that we’re only beginning to explore. This is also the area with the most investment today, be that in the form of better chatbots for customer service, or AI assistants that speed up data analysis and content creation.

Applying these technologies to the wealth management space, Endowus is actively pursuing several LLM initiatives to enhance our services and provide our customers with an increasingly seamless, digitally-enabled experience. This will be accomplished through the design and launch of a variety of AI-driven Chatbots, and a Fund Recommendation Engine that adopts a natural conversational style to capture customers’ goals and provide sophisticated, personalised recommendations.

Human-to-Human Communication

LLMs are fundamentally about understanding human communication in all its forms and across all mediums. This understanding is not language-specific – it merely requires an abundance of examples.

Consequently, LLMs are poised to catalyse a revolution in language translation, rendering high-quality translations imbued with the original text’s nuance, idiom, and metaphor. Imagine reading ‘Anna Karenina’ or ‘The Three-Body Problem’ in any language, with no delay for translation and no compromise on quality.

As this technology matures, translations will be done in real time, enabling us to speak with each other in our native languages & yet understand each other completely. The fabled Babel Fish from ‘The Hitchhiker’s Guide to the Galaxy’ is on the verge of becoming a reality.

For businesses, this represents a significant leverage opportunity for efficiently translating all forms of marketing and business communications to support regional aspirations and expansion into new markets.

Computer-to-Computer Communication

As someone whose background is that of a software developer, I find the potential of LLMs in computer-to-computer communication particularly fascinating. The dream of autonomous software agents, previously restrained by their inability to adapt to new scenarios, now appears within reach.

Do you recall that cinematic moment in Independence Day when Jeff Goldblum’s character audaciously infects an alien spacecraft with a computer virus? As a software programmer, I always found that scene a bit hard to swallow. I mean, here on Earth, we face hurdles in getting a simple Windows app to run seamlessly on a Mac, and vice versa. Yet in the realm of Hollywood, we somehow managed to implant our human-made software, a computer virus, into an entirely alien tech!

With LLMs, software agents could negotiate a shared language (a protocol in software terms), enabling them to interact without needing human intermediaries. That scenario from Independence Day is completely in the realm of possibility now!

As we look to the future, I foresee software development focusing on two key areas: the foundational, involving making LLMs more efficient and cost-effective, and the applied, involving leveraging LLMs in diverse contexts. Traditional programming might still exist, but it will morph into a specialised dialect, akin to doctors or physicists communicating using their domain specific terminology.

The Future

Large Language Models signify a profound shift in the communication paradigm, offering a path towards a more intertwined human-digital world. It’s about harnessing practical solutions that enrich our lives, transforming our reality by infusing it with elements of a previously imagined futuristic sci-fi world. As custodians of this technology, we must handle it with care, ensuring that it aligns with our values and serves our collective needs.

As we navigate this new frontier, it’s crucial to embrace the possibilities while acknowledging the accompanying challenges. By treating this progress with thoughtful curiosity, we can orchestrate a future where technology becomes an integral and harmonious part of our daily interactions and communications.

An edited version of this post was first published here: https://www.smehorizon.com/large-language-models-5-takeaways-for-smes/

Android Work Profile

2021-11-13T10:10:10+08:00

Android has a neat feature called Work Profile that sandboxes work apps & data from personal apps & data. If needed, your company can (remotely) manage and/or wipe just the Work related data without touching your personal data. This makes it really easy to use personal devices for work usage, while staying compliant with data security requirements.

However, Android doesn’t include any built-in user interface to create and manage a Work profile. This feature is only exposed as APIs that 3rd party Mobile Device Management (MDM) vendor products leverage. Examples: VMWare, Google.

Thankfully, there’s a neat OSS app called Shelter that implements a UI on top of the APIs, making it possible for you to create a Work profile right on the phone.

With Shelter, you can create a Work profile and manage apps within it, all from your phone. This is incredibly useful if your company uses Google Workspace since it keeps your personal Google account separate from the Google Workspace account that is used for work. You get two different instances of GMail, Calendar, Drive, etc. No cognitive overhead of making sure that you are using the right account!

It’s a complete experience! Links opened from a Work profile app open in Chrome (or another browser) within the Work profile. Clipboard & filesystem contents are separated so there’s no accidental leak of data from Work to Personal or vice-versa. Very nicely done & highly recommended!

One slightly unintuitive aspect of setting up Shelter was enabling Widgets. I wanted to have the calendar widget on the Android homescreen and the widget wasn’t available at first. The solution was to open the Shelter app, navigate to the Calendar app entry on the Shelter profile, click on it and then enable the Allow Widgets in Main Profile setting for the app.

I really wish Windows and MacOS also implemented something like a Work Profile for our desktop environments as well. In this post-covid world where work-from-home is the norm, Bring Your Own Device (BYOD) is not limited to phones any more!

Migrating to AWS Amplify

2021-09-30T18:30:00+08:00

In a recent blog post, I wrote:

this site runs on a shared host provided by Dreamhost. I know that sounds awfully outmoded in this day and age of containerizing everything! But I’ve had antrix.net on shared hosting for more than fifteen years now and if it ain’t broke…

I guess it was time to fix it!

Being in between jobs and unable to travel, I found myself with a few days of downtime on hand. I spent some of that time evaluating a few of the newer jamstack/serverless/heroku-inspired hosting services, live tweeting along the way. In the end, I’ve settled on AWS Amplify for the moment and migrated devdriven.by and antrix.net to it.

This post documents my setup on Amplify, i.e, a prime example of Infra-as-blog-post.

Build Setup

As I’ve described previously, I use Pelican as the static site generator to build this site. Pelican provides a Makefile that encapsulates the entire site generation process. Thus, replicating it in Amplify’s app build specification was straightforward:

version: 1
frontend:
  phases:
    build:
      commands: 
      - pip3 install -r requirements.txt
      - cd site
      - make publish
  artifacts:
    baseDirectory: site/output
    files:
      - '**/*'
  cache:
      paths: []

I’ve setup Amplify to build two branches:

Every commit to develop branch is built & deployed to https://next.antrix.net/
Every commit to main branch is built & deployed to https://antrix.net/

I used the Environment Variables facility of Amplify to setup a variable named STAGING which is active in the develop branch build.

The Makefile uses the STAGING variable to make some changes in the build, e.g., use a different robots.txt with a global deny rule for the staging website.

Domain Setup

Amplify uses S3 for storage and Cloudfront as the CDN for static content. But these services are not exposed to you and are managed behind the scenes within Amplify’s service accounts. The end result is that Amplify creates a <some-id>.cloudfront.net fqdn and also makes your site available on <id>.amplifyapp.com. To use a custom domain such as antrix.net, we need to setup DNS records to point to the cloudfront fqdn.

While I’ve now migrated hosting away from Dreamhost, I continue to use them for DNS management. Following Amplify’s instructions, I setup four DNS records:

@        ALIAS    <id>.cloudfront.net
www      CNAME    <id>.cloudfront.net
next     CNAME    <id>.cloudfront.net
<id2>    CNAME    <id3>.acm-validations.aws

That last one is to provide AWS proof of domain name ownership.

Rewrites & Redirects

Amplify provides flexible URL rewrite/redirect features that addressed all of my requirements. You can create these using their web console and changes appear to take effect instantaneously. Here’s my configuration:

Performance

The few builds I’ve observed so far trigger almost immediately after a push to Github and then complete deployment in a minute. This is quite good already and I suspect can be improved further using build caching.

On to the page load performance, this was the site’s Lighthouse score prior to migrating:

And this is the score post migration to Amplify:

As expected, the performance score improved thanks to edge caching enabled by cloudfront.

Résumé writing: Focus on Outcomes

2021-09-19T15:40:00+08:00

Take a look at this snippet taken from a resume:

Senior Engineer in the Sales Trading development team within Citi Equities Technology. In this role, my responsibilities include:

Working with desk traders and market connectivity teams on several features.

Software development primarily in Java on Linux with some Perl, Solaris as well.

Ensuring high quality code using unit testing and other practices.

Driving quality improvement & cost reduction initiatives.

As a hiring manager reading this resume, it’ll be really hard for me to get a good sense of the candidate based on the above snippet. The entire snippet reads like a job description. Isn’t that good? If the description of the job that the candidate claims to have done matches the description of the job that I’m hiring for, isn’t that all I need as a hiring manager? Not quite.

The reality is that there’s only so many ways to describe a software engineering role. So if resumes are nothing more than descriptions of the job, every resume starts to look the same and there’s nothing to differentiate one candidate from another. Thus, a resume written like a job description is failing at its primary function: getting the hiring manager to pick it out of a stack and shortlist the candidate for an interview.

Now take a look at this snippet:

Senior Engineer in the Sales Trading development team within Citi Equities Technology. Worked in a team of four that developed PTE – the primary trading system used by Citi’s Program Trading desk globally.

On-boarded the Program Trading desk onto two new exchanges, collaborating with desk traders and the market connectivity teams.

Built a FIX to Proprietary messaging adapter that allowed 70MM$ notional daily order flow to be routed to smaller markets and also serve as a failover route to the larger markets.

Migrated dozens of Java services and Perl/Shell scripts from Solaris to Linux yielding infra cost savings of $90K/year.

Spearheaded the Development Maturity Model, a development efficiency improvement program within Global Equities. Raised team’s score by 54%.

It’s for the same role but is now re-written to focus on tangible achievements and outcomes. There’s still a job description but it’s limited to the first couple of lines to provide context. It’ll be hard to come across another resume that looks exactly like this!

Not only does such an outcomes focussed resume stand out in front of a hiring manager, it also provides them with data that they can use to make a rough assessment of the level at which this person performs. This makes it easier for them to determine if the candidate would be a good fit for the scope/level of the role being hired for.

Writing resumes like this isn’t very hard to do. While the earlier example is from my own resume, let me share another example from the resume of a friend whom I recently helped with this exercise.

Here’s a section of his resume from before:

As a senior technologist in the Private Banking Technology group at {Large Bank}, I am responsible for:

Development of pre-trade disclosure automation platform which is part of digitization strategy of the bank to ensure all risks are disclosed accurately.

Heading the architecture and implementation of price discovery and execution platform for Foreign Exchange Options trading.

Participate in wider code and design review meetings to ensure adherence to clean coding and other agile practices.

Screening and interviewing software engineers for the division.

It mentions some specifics of the role but by and large, the entire section still reads like a job description.

Now here’s the same role, re-written to be outcomes oriented:

As a senior technologist in the Private Banking Technology group at {Large Bank}, I have:

Owned the greenfield buildout of FX Options price discovery and execution platform which enabled sales traders to efficiently get the best deal across the street and also reduced the manual booking efforts of Middle Office team by 80-90%

Onboarded majority of FX Vanilla options to new automated flow supporting a monthly trading volume of nearly $125M

Provided valuable data insights to senior management on key metrics related to platform usage and influenced future strategy

Mentored software engineers to help them establish best practices for software development and automated testing to ensure fast time to market

Hopefully, you will agree that the re-written version is much more differentiated and impactful.

A great piece of career advice that I received more than a decade ago was to update my resume every six to twelve months regardless of whether I was looking for a new job. The act of updating the resume provides the catalyst you need to reflect on your own career growth. If in six months or an year you have nothing meaningful to add to your resume, then clearly you aren’t growing in the role and it’s time to explore new options.

Obviously, implementing this advice would be rather difficult with a resume that reads like a job description! An outcomes oriented resume forces you to take stock of your career and be honest about your own progression.

So go ahead and take a good look at your resume. If it doesn’t mention what you’ve actually achieved, spend a little time and make it outcome oriented today!

Building a Documentation Culture

2021-08-08T15:10:00+08:00

“How do you build a good documentation culture in a company?” — that was the question posed by a friend who’s the VP of Engineering at a fast growing startup.

As we do in most situations, more so when talking about culture, we start with the People.

People

People model behaviours that they observe are being practiced by others and that are being rewarded. Those behaviours become your culture.

Good documentation starts with a good writing culture. So you want to make good, clear writing the norm in your organization.

There are plenty of opportunities to model & reinforce a good writing culture. Are your emails well written? When you see someone writing an ineffective email, do you take the time to coach them on writing clear, actionable emails? Are your JIRA tickets well written or just one liners that would be inscrutable to anyone, including the ticket writer after a month? How about presentations that your team creates? Are they being thoughtful in adapting their writing style to the medium?

Writing is a skill and like any other skill, it improves with practice & training. So invest in writing training for your staff! Amazon is known to have a strong writing culture and there’s plenty of internal training that’s provided for employees to learn to write well. Do note that writing well is contextual. What’s well written prose for a period novel is probably not well written for a business document. The training needs to reflect that. In my first job at a scientific research institute, I was sent to a training on writing good scientific papers. It was immensely useful and I put that training to good use in the two journal papers that I subsequently wrote during my tenure there.

Finally, as I mentioned earlier, culture is also what gets rewarded. If you want a good documentation culture, then do factor in documentation work into performance appraisals. While documentation in general should be considered part of the job (and culture!), there are opportunities to call out & reward specific activities, e.g. when a junior employee writes a new piece of documentation or when a senior employee creates/improves a documentation related process.

Process

Which is a good segue into looking at process improvements that you can make to create a good documentation culture.

The goal of a process is to ensure you get the desired outcomes in a reliable & repeatable manner. In the case of documentation, the outcome to strive for is great Quality of documention, i.e., documentation that’s Accurate, Useful and Complete.

Accuracy means that your documentation reflects reality. So you want processes that help you catch & fix errors in your documentation.
Useful documentation helps the reader accomplish their goal. So you want feedback mechanisms that inform you whether readers managed to do that.
Completeness means that there aren’t any gaps in your documentation. Documentation could be useful but not complete (e.g. some features aren’t documented) and vice-versa (e.g. everything is documented but is incomprehensible to the reader).

The processes will vary depending on the quality bar that you are striving for. Given limited time that can be spent on documentation alone, you’d want a higher quality bar on customer visible documentation than internal documentation.

With that mental model, here are some ideas for processes that you can implement:

Make it easy to create new documentation by creating templates. More on this in the next section below.
Make it easy to add & update documentation. E.g., don’t prevent other teams from updating your team’s docs - make them world writeable. The risk of incorrect changes is small and easily manageable with version control.
Pull Requests must include doc changes. If you have a PR review checklist, add “docs updated” to that list as a reminder. Don’t tolerate a “docs will be updated in the next PR” culture.
Use a written process (e.g. Amazon’s PR/FAQ, Python’s PEP, Oxide’s RFD) to discuss and agree upon major architecture, process or product changes.
Add automation to check broken links within your documentation, especially for customer facing docs.
Add analytics and feedback button mechanisms to your doc pages and look at the metrics to assess quality.
In your customer support system, ensure there’s a “documentation related” tag that can be applied as the root cause when resolving tickets. Measure this ticket count to check progress on your documentation efforts.
Improve the previous process by ensuring such a support ticket isn’t closed till a related doc fix ticket is created.

Hopefully, these bullets have given you a few jump off points to consider when making process changes to improve documentation.

Product

Finally, we look at the product which in this case, is the actual content of your documentation.

The most important thing you can do here is to figure out how best to organize the content and then create templates around that organizational framework. Your documentation may be Complete, but if it’s poorly organized, it’ll be perceived as neither Useful nor Complete by your readers who’ll struggle to find what they need. Moreover, an organizational structure also significantly reduces friction in creating new documentation since you are not repeatedly spending cycles just figuring out where every new piece of content should live (cf. bike-shedding). This is applicable both for external documentation as well as internal docs.

A good organizational framework is The documentation system which organizes documentation by purpose:

Tutorials - step by step docs that aid learning for a new comer.
How-to guides - problem oriented docs that help in accomplishing specific tasks.
Conceptual - docs providing detailed explanations that aid in building a strong understanding of your system.
Reference - Informational documentation that provide all the nitty-gritty details.

I often cite the Django project’s docs as an example of high quality documentation and you can see how it follows the above framework.

Interestingly, this framework isn’t limited to just software or technical products. You can easily imagine how you could apply it to say, a podcasting tool or a stock trading platform.

For internal documentation, an area which benefits immensely from investment is organizing operational documentation. This documentation covers aspects like where’s a particular service’s source code, the requirements queue, design docs, deployment processes, monitoring dashboards, SLAs, alarm configurations, on-call rotation, etc. When I was at JPMorgan Chase, one of the first problems that the SRE chapter in my team tackled was to standardize the service documentation so that on-call engineers could quickly and reliably find what they needed when responding to incidents.

Spotify’s Backstage attempts to standardize all of this information so that it’s discoverable at scale. But you don’t need to start with such a system; just take the concepts and then create a few template wiki pages that make it easy to instantiate internal docs for every new microservice that your team spins up.

Architecture documentation is often overlooked in a fast moving environment and struggles to keep up with an evolving system. Here, I recommend the C4 Model as a light-weight system that can help your team maintain high quality architectural documentation. Every four to six months, get your entire team in a (virtual/physical) room for a couple of hours sketching your system’s architecture on (digital/physical) whiteboards or flipcharts using the C4 model. For maximum impact, split your team into two and get each half to document the system independently and then compare their diagrams to identify gaps. Once done, just upload the diagrams as-is to your internal wiki. This exercise not only keeps your docs up to date but is also a fantastic way to deepen your team’s collective understanding of the system and serve as a great onboarding resource for those new to the team.

Being intentional

Good intentions never work, you need good mechanisms to make anything happen. — Jeff Bezos

To create a culture of documentation, communicate your intention and back it up with investments in your people and processes.

Don't ignore Chaos testing!

2021-05-23T21:35:00+08:00

In March this year, I left JPMorgan Chase to join Amazon Web Services. I was at JPMC for close to eight years and my last role there was to lead the product development & delivery of observability services. This included passive observability services for aggregating & alerting over operational metrics, events, logs, and traces; as well as active observability services like synthetic transaction testing & chaos engineering.

That last bit around chaos engineering was one of the more memorable, forward looking, leading edge, I-can’t-believe-a-bank-does-this, pieces of work that I’m especially proud of incubating & delivering. From that vantage point, I thought I’ll share a bit of my perspective on this field.

I’m not going to delve into explaining chaos engineering in this post. The principlesofchaos.org site does a great job of explaining the fundamentals in 10 minutes and is well worth a read. If you have more time, then consider watching this talk on chaos engineering that a colleague and I gave. It’s a great talk, I promise. 😊

One of the ideas we touch on in that talk is the distinction between chaos testing and chaos experimentation.

In essence:

Experimentation leads to learning new things about the system while testing validates our existing understanding of the system.

In a complex system, it’s very hard, if not impossible, to predict the behaviour of the system based purely on an inside out understanding of the system’s components and internal design¹. Thus, we run chaos experiments with failure injection to understand how the system behaves under turbulent conditions. The results of these chaos experiments provide us with new knowledge about the system².

On the other hand, chaos testing also involves failure injection but with the goal of validating that the currently known properties of the system (related to failure modes) haven’t changed. The properties are either known because they were a design goal (e.g. fallback to cache when source is unavailable) or were added to the system in response to a failure mode discovered through chaos experiments (e.g. circuit breakers).

I feel that in the chaos engineering community, chaos testing is sometimes given the short shrift.

Chaos testing is an important new tool to add to your arsenal of unit, integration, and performance tests. Chaos tests can be automated and added to your CI/CD pipeline as part of your regression test suite. In an enterprise and/or regulated context, automated chaos tests provide always available evidence of conformance to BCP standards.

While there’s immense value to be unlocked through chaos experimentation, you shouldn’t ignore chaos testing! After all, what good is the knowledge accrued from a chaos experiment if the next big feature release or refactoring leaves your system vulnerable to the same failure mode due to a regression?

Hence, your chaos engineering adoption journey must include both chaos experimentation and chaos testing.

The challenge adopters face today is that the tooling for automated chaos testing is still in infancy. While significant progress is being made by the likes of Chaos Toolkit, Verica, Gremlin, AWS, and others, I believe we are still at the beginning of this promising journey³. As a point of comparison, consider how pervasive unit testing is today. However, this wasn’t always the case. JUnit was created in 1997 and Kent Beck’s XP book came out in 1999. Evidently, it took decades for unit testing to become pervasive, requiring a combination of greater collective experience and better tooling.

We are in that same primordial state with chaos engineering and I’m convinced that we’ll see immense progress in the tooling landscape as well as our understanding of the field in the coming decade.

Systems theory makes this explicit in that the internals of a system are incidental and what matters are the system’s inputs & outputs. ↩
In fact, Chaos Engineering emerged because complex systems’ failure modes couldn’t be understood by traditional inside-out methods. ↩
At JPMC, my team built our own internal chaos service because none of the tools met all of our needs. ↩

AWS SA - Associate

2021-05-07T20:25:00+08:00

To help with my new job, I’ll be taking a few AWS certifications. First on that list is the AWS Certified Solutions Architect – Associate certification exam which I passed yesterday.

As a reminder, this is my personal blog and whatever you read here are my personal views and not those of any past, current or future employer!

The AWS SA Associate exam tests foundational knowledge AWS services and the ability to use them to solve requirements with secure & robust solutions. The exam’s duration is 130 minutes & it consists of 65 multiple choice questions with either one or two right answers.

Preparation

These are the resources that I used to prepare for the exam:

Stephane Maarek’s Udemy Course: This course is great. It covered all the stuff needed for the exam. But this course can be overwhelming if you don’t have some prior experience building software systems. If that’s the case, you should find another foundational course before coming to this.
Exam Readiness: AWS SA Certified - Associate: This free course is a nice short resource to get oriented to the exam. The key value of this course is that it explains clearly what factors to consider when an exam question asks for scalability vs resiliency vs HA vs fault tolerance.
Whizlabs practice tests: These were just alright. While the level of the actual exam is probably 20-30% harder than the questions here, the good thing about the test collection is that they cover a wide variety of questions. For each question that I’d get wrong in practice, I would read the linked documentation, any related documentation as well as the FAQ pages.
On the day before the exam, I reviewed all the slides from Stephane’s course as a final refresher. This is essential since the amount of material covered in the exam is quite vast and reviewing everything a day before helps considerably with information recall.

The exam

I took the exam at a PSI center instead of at home. The exam interface is quite straight forward and allows you to navigate between questions as well as flag questions for review later. The count down clock is always visible on the top right and gives a five minute warning as well. I had sufficient time to complete all the questions and then review about 30-odd questions again, before running out of time.

I was informed of the result (pass!) as soon as the exam ended. The screen also informed me that I’ll receive an email within five business days with the score. However, when I logged on to the AWS training website later in the night, the score certificate was already available.

One weird trick

If you are Singaporean, you can use your SkillsFuture credits towards Stephane’s Udemy courses. I did just that for this course and plan to buy his other courses as well for the next certification exam that I attempt.

Good luck if you are planning to take the exam as well!

Designing an IT organization from scratch

2021-02-22T17:35:00+08:00

A few months ago, the Monetary Authority of Singapore awarded the first set of digital banking licenses. As the press release mentions, there were 14 applicants out of which 4 were granted licenses.

As it happens, I had interviewed with one of those applicants. While I did clear the interviews, it didn’t result in a job offer since they didn’t get the license. To be honest, this was a bit surprising since they are a well known brand and were thought to be a strong contender. Nevertheless, there’s not much point in hiring people without a banking license!

The role that I interviewed for was called the Head of Production Services, reporting into the CTO:

Responsible for overall technology and operational support (front to back) of the bank’s digital platforms residing on multi-cloud environment, which includes operating the total cost ownership of the bank’s technology platforms efficiently, managing the life-cycle of the platforms systems, complying to regulations, enabling risk & controls, to deliver a high standards bar in users’ customer experience.

In effect, this role would’ve been responsible for delivering the entire IT platform capabilities that the bank’s business, operations and software engineering teams would rely upon. While a bit out of my wheelhouse, the scope and impact of the role was meaningful and challenging enough for me to go for it.

During the interview process, I was asked to prepare and present how I would structure this Production Services organization. This was a very interesting thought exercise that I enjoyed digging into immensely. I drew on my own experience, received insight & feedback from a few trusted friends, and read numerous resources. One of the resources that I found quite helpful was Truth from the Valley: A Practical Primer on IT Management for the Next Decade, a book by Mark Settle. I highly recommend it to anyone in an IT leadership role.

Here’s the final deck outlining my design of the IT services organization at a hypothetical Blue Bank. I hope you find it interesting!

GoAccess on Dreamhost

2021-02-09T17:10:00+08:00

In the most recent reboot of the site, I had ripped out all JavaScript, including Google Analytics. To be honest, it wasn’t a big loss since I can’t even remember when I had last looked at the Analytics data.

But it’s good to have some understanding of what’s happening, if only to detect and weed out badly behaved crawler bots.

As I have occasionally mentioned here, this site runs on a shared host provided by Dreamhost. I know that sounds awfully outmoded in this day and age of containerizing everything! But I’ve had antrix.net on shared hosting for more than fifteen years now and if it ain’t broke …

Anyway, Dreamhost provides web site statistics out of the box that rely on server side access logs to generate reports. While it works, it is based on Analog which, true to its origins, looks like something from the 90s.

Could I replace Analog stats with something a bit more modern? Could I do it while ignoring the irony that I’ll be fixing something that isn’t broken?

GoAccess appears to be the new kid on the block of open source web log analyzers. It seemed simple enough to configure and use that I decided to give it a Go.

The rest of this post describes how I set it up on Dreamhost. If you want to jump ahead, here’s the final result: stats.antrix.net

The first step was to compile and install goaccess:

$ mkdir ~/goaccess
$ cd ~/goaccess/
$ wget https://tar.goaccess.io/goaccess-1.4.5.tar.gz
$ tar xvzf goaccess-1.4.5.tar.gz
$ cd goaccess-1.4.5
$ ./configure --prefix=$HOME/goaccess/ --enable-utf8
$ make
$ make install

Next, I created the stats.antrix.net subdomain using Dreamhost’s domain management UI, making sure to use $HOME/stats.antrix.net/ as the web server root.

After that, I created a simple script that will call goaccess to parse the most recent access logs and generate an html report in the web server root directory.

$ cat $HOME/goaccess/gen-access-report.sh
#!/bin/bash

${HOME}/goaccess/bin/goaccess --db-path ${HOME}/goaccess/data/ --persist --restore --log-format=COMBINED --anonymize-ip --keep-last=90 --real-os ${HOME}/logs/antrix.net/http/access.log.0 -o ${HOME}/stats.antrix.net/index.html 1>${HOME}/goaccess/cronjob.log 2>&1

if [[ $? -ne 0 ]]; then
    echo "goaccess execution failed"
    echo "========================="
    cat ${HOME}/goaccess/cronjob.log
    exit 1
fi

The script does a few things:

It instructs goaccess to use a database to store parsed data
The data is retained for 90 days
The access.log.0 file is parsed, which is the most recently rotated log file
The client IPs are anonymized because my stats are now public
Execution logs are printed out only in case of any failure

Finally, I setup a cronjob to run the script once a day.

$ crontab -l
MAILTO="xxxxxxxxxxx@xxxxx.com"
0 7 * * * /bin/bash ${HOME}/goaccess/gen-access-report.sh

If there’s any error, the failure output printed by the script is emailed to the MAILTO address.

That’s about it. Relatively painless to setup and it seems to be working well so far: stats.antrix.net

CKAD

2021-02-02T20:25:00+08:00

I passed the Certified Kubernetes Application Developer exam recently. As is apparently a requirement, I must now blog about it!

In case you are not aware, the CKAD is a 2 hour CLI based hands-on examination that tests how well candidates can apply their Kubernetes knowledge to perform numerous tasks in a real kubernetes environment. In case you are not aware, Kubernetes is a highly popular container orchestration system that hits the sweet spot between being too complex for web applications and too simple for anything else.

As the search link above indicates, there’s no shortage of advice on how to pass the CKAD. I’ll refrain from repeating those and just share what I think hasn’t been discussed so far.

At the risk of stating the obvious, preparing for the CKAD is a two step process:

learn Kubernetes
practice for the exam

Learn Kubernetes

I learnt Kubernetes mainly through the material from learnk8s.io. My employer had arranged for their in-person training which was delivered by Daniele Polencic, whom I had met earlier at Kubecon Shanghai through my friend James Buckett. Daniele is a great instructor who knows Kubernetes inside out. And their material is top-notch. It’s focussed on giving you a strong understanding of the building blocks of Kubernetes instead of just helping you prep for CKAD.

As just one example, I found that quite a few resources don’t explain the relationship between Services and Endpoint objects. This relationship is really important since it’ll help you debug and diagnose various Service issues. The learnk8s material explains this in a very cogent way. There are many more such fundamental concepts that are glossed over in CKAD focussed material which are explained well by Daniele and team.

Practice for the exam

During my CKAD, I received 18 problems that I had to solve within 2 hours. That’s not a lot of time if you don’t have some hands-on fluency with Kubernetes. The best way to build fluency is practice!

I practiced with just these two resources:

dgkanatsios/CKAD-exercises: This is widely recommended and I agree. I went through these exercises twice.
CKAD Practice Challenges by Liptan Biswas: this gives a feel for what the real exam is like.

Note that the actual exam was more difficult than these practice problems but not too far off.

The other thing I did was to have a set of bookmarks ready to go during the exam. Since it is allowed to refer to the kubernetes.io documentation during the exam, having a set of bookmarks to the relevant material was a big time-saver. With the amount of yaml boilerplate involved in kubernetes, you definitely want quick access to yaml templates that you can copy from the website into your exam terminal.

One weird trick

Speaking of yaml, you must set up your vim editor for yaml editing. At the start of the exam, I created a ~/.vimrc file with the following:

set ts=2 sts=2 sw=2 et
set list
syntax on
filetype plugin indent on

It is absolutely worth memorizing the above because you don’t want to be fighting with yaml editing during the exam. I haven’t seen set list mentioned in CKAD prep resources but I highly recommend it. That instruction makes vim render tab characters as ^I so that they stand out. The last thing you want is an invalid yaml due to a stray invisible tab character that you’ll struggle to find and fix!

The other vim tip worth remembering is to :set paste before pasting a bunch of yaml from the documentation. This prevents vim from indenting the already indented code that you paste from elsewhere. I realized this during the practice sessions where I’d spent way too much time just fixing indentation of copy/pasted samples.

One more trick worth sharing is to use kubectl replace --force when you need to delete and recreate a pod or other object. Saved me quite a few minutes!

Closing

Due to the format, this exam was more interesting compared to the usual multiple choice certification examinations. That however, does not change my view of certification examinations in general. I think IT certifications are great as a structured approach to learning a new subject. But they are not very useful as signifiers of subject knowledge that an employer can supposedly use in making hiring decisions.