AI makes your team faster. It may also be quietly overwhelming the people responsible for making sure it all holds together.

Two years ago, something shifted. The developers I work with stopped staring at blank files waiting for inspiration. They started shipping. Fast. A feature that once took a week arrived on Friday afternoon in a pull request. A proof-of-concept that used to require a research spike was running in a sandbox by lunchtime. On the surface, it looked like a productivity miracle. And in many ways, it was.

But somewhere in the middle of all that acceleration, I started feeling something I didn't expect: exhausted. Not from doing less work — from doing a fundamentally different kind of work that nobody had really prepared me for. I was no longer just a senior engineer. I had quietly become the human quality gate at the end of an AI-powered production line.

This is a post about that experience — and a reflection on what it means to lead an engineering team in the age of AI-generated code, AI-generated documentation, and AI-accelerated expectations.

The Acceleration Is Real. So Is the Bottleneck.

There's no point pretending the tools aren't remarkable. I've watched a non-technical executive describe a rough idea in plain English and receive a working, deployable application within 48 hours. I've used AI myself to cut research time from hours to minutes — skipping through documentation, comparing architectural options, generating functional scripts I could validate immediately.

The productivity gains are real, measurable, and genuinely exciting.

But here's what we don't talk about enough: when five developers on a team can each generate production-quality code in a matter of hours, the bottleneck doesn't disappear. It moves. It moves to the one person who has to review all of it, understand all of it, and be accountable for all of it — the lead engineer.

"I used to occasionally receive a large pull request. Now I regularly receive pull requests with more than 10,000 lines of code. That's not a small shift — it's a different job."

Ten thousand lines. Generated in a day or two. Each line potentially containing a perfectly reasonable decision that doesn't fit the codebase, the organisation's standards, or the architectural direction we've been building toward for months.

The Problem Isn't the Code. It's the Context.

When I review AI-generated code, the issue is almost never that the code is wrong. It often works. Sometimes it's elegant. The problem is that it works in a vacuum — without awareness of the larger system it's being dropped into.

A Tale of Two Auth Flows

Real scenario: A developer is building a new service that requires user authentication. They prompt their AI tool: "Implement OAuth 2.0 authentication." The AI, being genuinely helpful, implements a secure, well-structured OAuth 2.0 flow. It chooses the Client Credentials flow — reasonable for certain service-to-service contexts.

The problem: every other application in the organisation uses the Authorization Code + PKCE flow. The new service is technically secure. But it doesn't fit. It can't be managed the same way. It creates a maintenance burden, an onboarding headache, and a future security audit question.

None of this is visible in the code itself. You only know it's wrong if you know the organisation.

The same pattern repeats across every layer of a codebase: pagination strategies (cursor-based vs. page-based), API versioning conventions, logging formats, error response shapes. An AI tool has no way to know what your organisation decided in a design review six months ago. It makes a reasonable choice. The lead engineer has to catch it, explain it, and redirect it — across dozens of pull requests, week after week.

Over-Engineering by Default

There's another pattern I've noticed: AI tools tend to gold-plate. Ask for a simple data endpoint and you may receive a fully instrumented, horizontally scalable service with a caching layer, a retry queue, and an event-driven architecture. These are often impressive. They're also sometimes completely unnecessary.

I've seen caching layers added to services that receive ten requests a day. I've seen event-driven patterns introduced into workflows that run once a week. The AI isn't wrong, exactly — these patterns are useful in the right context. But context is precisely what AI doesn't have. And the lead engineer has to not only identify the over-engineering, but explain why simplicity is the right call — a harder conversation than it sounds when the code in front of you looks sophisticated and well-intentioned.

Junior Engineers, Design Patterns, and the Invisible Gap

One of the trickier dynamics I've observed is the gap between what junior engineers ask for and what the codebase actually needs.

Senior engineers develop pattern instincts over years of reading code, making mistakes, and reviewing others' work. They know when to reach for a Strategy pattern versus a Factory. They recognise when a new module is inadvertently duplicating an abstraction that already exists three directories away. They feel the shape of a codebase.

Junior engineers — even talented, hard-working ones — haven't had time to build that instinct yet. And now they have a tool that can generate a thousand lines of working code from a single prompt. The code may function perfectly and still be architecturally alien to the rest of the project.

The cognitive load here is subtle but real: as a reviewer, I have to hold the existing architecture in my head, understand what the AI has generated, identify where they diverge, and then explain the divergence in a way that's instructive rather than demoralising. Every time. For every PR. On top of everything else.

"The bottleneck used to be 'can we build it?' Now the bottleneck is 'can we integrate it?' — and that second question is entirely on the human."

The Documentation Deluge

The challenge isn't confined to code. It extends upstream — into the documents, diagrams, and design artefacts that define how we build things before a single line of code is written.

AI tools are extraordinarily good at generating documentation. A single prompt can produce a 15–20 page architecture document, complete with database schema breakdowns, relationship diagrams, data flow descriptions, and edge case analyses. This is genuinely useful. It's also genuinely overwhelming.

The problem is that generating a document is not the same as understanding it. When I hand a stakeholder a comprehensive 20-page requirements document, they reasonably expect me to be able to speak to every page of it. That means I need to have read it carefully, verified it against our actual constraints, and caught any places where the AI has been confidently wrong — which happens more than I'd like to admit.

The Revision Loop

You generate a 20-page document. You identify sections that need changing. You prompt again with your revisions. The new document comes back. You read all 20 pages again to verify the changes were applied correctly — and that nothing else shifted in the process. Sometimes the AI has quietly reintroduced the very content you asked it to remove, just rephrased slightly. You catch it, or you don't.

Multiply this loop by two or three documents a day, every day, and you start to understand the weight of it.

There's also a subtler issue: stakeholders and senior leadership are using AI too. They can now arrive at meetings with polished documents, detailed proposals, and rapid-fire ideas that used to require days of preparation. The pace of conversation has accelerated. The expectation of response time has compressed. Everyone can have a fully articulated perspective on everything, almost instantly.

The lead engineer in the middle of this — responsible for translating between leadership's AI-accelerated vision and the development team's AI-accelerated execution — is now doing a kind of cognitive dance that simply didn't exist before. And it's tiring in ways that are hard to quantify.

What's Actually Missing: Governance at the Prompt Layer

The honest answer is that we don't yet have good tools for governing how AI is used within a development team. We can add linting rules, we can write coding standards documents, we can include architectural decision records in our repositories. But none of these things reach back to the moment when a developer is sitting at their keyboard, typing a prompt, and an AI tool is making a dozen invisible architectural micro-decisions.

Some teams are experimenting with prompt templates, shared context files, and AI coding guidelines — and these help at the edges. But for genuinely novel problems, for the parts of a system where there's no prior art to reference, the AI still has to make a call. And someone human still has to evaluate that call after the fact.

Right now, that someone is almost always the lead engineer.

A Note to Other Leads in the Same Position

If you're a senior engineer or engineering lead and any of this feels familiar, I want to say: you're not imagining it. The cognitive load is real. It's structural, not personal. It's a consequence of a genuine and rapid shift in how software gets built — and the tools, processes, and norms for managing that shift at a human level are still catching up.

The goal of this post isn't to argue against AI tools. They've made me faster, made my team more capable, and opened up possibilities that didn't exist two years ago. The goal is to name what's happening honestly — because the first step to solving a problem is admitting it exists.

We need to have a serious conversation with our leaders about what it means to lead a team when the output volume has multiplied but the human review bandwidth hasn't. About how we set expectations with stakeholders who are also using AI and have lost their intuition for how long things actually take. About how we invest in the judgment and context-building that no AI tool can replace.

The bottleneck has moved. Now we need to figure out how to move with it.

Offline and Offsite backup and disaster recovery is often confused as the same thing. Both terms describe data locality but are designed to address different risk in an organization's disaster recovery posture.

What are offline backups?

Offline backup involves storing data on a storage medium disconnected from the network such as a tape drive or an external HDD. It is also called “air-gapped backup” because it’s not reachable by any running software.

What are offsite backups?

Offsite backups on the other hand is data stored at a different physical location than the primary data. Offsite backup copies help provide redundancy and improve availability of data. They provide protection against disasters like fires, power outage or natural disasters in the primary site.

The overlap

These concepts can overlap:

• A backup can be offsite but still online — for example, a cloud replica that is network-accessible.

• A backup can be offline but still onsite — for example, a tape stored in a cabinet at the primary site.

To maximize resilience, enterprises must focus on combining both Offline and Offsite backups.

Example: a weekly backup is copied from the primary site to a secondary site over the network. The secondary site then writes that copy to an air-gapped medium (e.g., tape) and retains it in a physically separate location. This approach protects against both localized site failures and network-based threats.

Upgrading the resiliency of your 3-2-1 backup strategy

The 3-2-1 backup strategy requires three copies of data, on two different media types, with one copy stored offsite. The most resilient way to implement 3-2-1 is to make the third copy both offline and offsite. The primary data lives on production storage. The second copy is a disk-based backup kept onsite for fast restores. The third copy is a weekly backup sent to a secondary site and written to a tape drive or external HDD. After the backup is completed the tape drive or HDD is disconnected from the network.

Threat 1: Hardware and Media Failure

Hardware failure is the most common threat to data. Drives fail, storage controllers malfunction and tapes degrade over time.

Mitigation: An offline copy is protected from cascading hardware failures. A firmware bug that bricks an entire storage array cannot reach a backup that is disconnected. An offsite copy adds further protection. If the failure is caused by a local issue like overheating or unstable power a copy at a different location is not affected.

Threat 2: Human Error and Data Corruption

A misconfigured script can overwrite production data. An application bug can corrupt a database silently over weeks. These failures are dangerous because they propagate. A corrupted database gets replicated to every connected backup target.

Mitigation: Offline backups break this chain. They preserve the state of the data at the point the backup was taken. If corruption has been spreading for days the offline copy from last week may be the only clean version that exists.

Threat 3: Ransomware and Cyber Threats

Modern ransomware does not just encrypt production data. It actively searches for backups. Network-attached backup shares and cloud-synced repositories are all targets.

Mitigation: An air-gapped copy such as a tape in a vault or a disconnected drive is not reachable by malware. There is no network path to it. Offsite backups add protection against insider threats. An employee with administrative access to local systems cannot destroy backups at a remote facility they cannot physically access.

Threat 3.1 Correlated Failures

The real risk is when a single event compromises multiple copies at once. Ransomware encrypts production data and the network-attached backup. A fire destroys the server room and the backup drives next to it.

Mitigation: Offline breaks the network link so cyber threats cannot propagate to every copy. Offsite breaks the geographic link so physical events cannot affect every copy. Together they create a backup that is digitally unreachable and physically distant.

Threat 4: Site-Level Disasters

Fire, flood or prolonged power outage can destroy everything in a single location. It does not matter how many backup copies exist if they are all in the same building.

Mitigation: Offsite backups solve this. A copy at a different geographic location ensures a localized disaster cannot wipe out all copies of the data. When combined with offline storage there is an added benefit. A disaster that takes out network connectivity can leave cloud-based offsite backups unreachable. A physical offline copy at a remote facility is available regardless of network conditions.

The 3-2-1-1-0 strategy and beyond

It is beneficial to extend this to build the 3-2-1-1-0 strategy where the extra "1" explicitly requires one copy to be offline or immutable and the “0” explicitly requires a zero errors copy. This makes the air gap a formal part of the backup policy. Backup tools and policies must be configured to ensure that only a backup with zero errors is moved to the offline copy.

Conclusion

In my experience making customers understand the difference between offline and offsite backups has been very critical while also explaining the importance of implementing both. Achieving whatever I have mentioned in this blog for both offsite and offline backups is very challenging in large organizations. I will continue to explore the world of backup and disaster recovery and come back with more practical insights in the future.

I thought it would be good to write more about what I have seen continue my train of thought around Engineering with AI. I see there is a lot of value in me spending time to express my thoughts and experiences on this blog series. Trying to write an essay without AI tools has never been harder.

Keeping up

It’s exciting and challenging to keep up. I feel like I am gonna miss the AI train if I don’t keep up. Every few weeks/months there is big leap in LLM tools. Each model claims to be more powerful than the previous one with lots of graphs and tables explaining the tests and evaluation metrics. One of the major focuses in these evaluations are around how much faster or better the model is in code generation, image generation or video generation tasks.

Some times it feels like it’s little but other times it feels like AI can solve any given problem in the world. Apart from tools, there are new ways to interact with LLM tools, including MCP servers and Agentic AI. As of writing this blog I haven’t explored a whole lot of these tools.

Being spoilt for choice

AI tools for the software development industry all the way from hands-free development websites like v0.dev by Vercel, lovable.dev, bolt.ai to IDE and CLI applications like Cursor IDE, Windsurf IDE, Claude Code & Gemini CLI.

All the tools either have free versions with limits this is to increase the user base of the tool. Some users will understand the usefulness and novelty of the AI tool enough to make a decision to upgrade to the paid version while others try to understand the best way to use the free version of the AI tool in question.

Which brings me to the next question.

Is spending money on paid versions opportunity cost?

The FOMO (Feeling Of Missing Out) of not utilizing the latest and greatest models has me more often than not deciding which paid/Pro/Plus version I must purchase? I have seen times where I have purchased ChatGPT Plus, Claude Pro, Cursor and a HuggingFace subscriptions all at the same time.

The decision to upgrade to a paid version of an AI tool is difficult for various reasons. The pricing of most of these tools are around the 20$ per month mark which translates to roughly 2000 INR a month. I always trick myself to think that this is an opportunity cost I am paying. I say to myself I will learn something new or be able to solve some complex problem more easily than other people using the free models. In fact I use some of these paid AI tools more just because I have paid for it; and have explored some cools features before my peers can. A good example of opportunity cost for me is getting to use Claude Code earlier than my peers who use the free model.

Should you pay for AI tools?

Paying for AI models is a commitment and for most users in a country like India, it’s difficult to commit to a subscription. The logic is simple if you spend 20$ per month per tool on more than one tool, you will end up spending close to 240$ to 480$ a year or 21,000 INR to 42000 INR a year. People I work with are surprised to know that I pay for Plus / Pro variants of AI tools, some times even multiple paid tools. Their argument is that there are ample free models available which give comparable results to paid versions. Some of the bright minds that I work with also have figured out prompting strategies etc. to be able to make the most of the free models and in some cases give better results than the free version. I am not that smart.

I have switched between paying for ChatGPT Plus and Claude Pro multiple times. Depending on what I was doing I would cancel my subscription and move to the other. Notable examples of when I switched is when the “Ghibli Art” trend caught along and when “Claude Code” was released. I have always found that the Plus/Pro versions gave slightly better results than the free version; this might be a confirmation bias, but I tend to be more confident with the outputs I get from the paid version. I have been able to the use the latest/greatest models and see the benefits of higher token limits, context windows and a variety of output options. Models like o4-mini-high and Claude Opus 4 really helped me increase my trust and skill in AI assisted problem solving and development.

Trying out these paid tools helped me recognize the value of these tools in the software development space and also stay ahead of the curve. Since you don’t hit the limits frequently, you are encouraged to use the tools more effectively by starting more prompts and experimenting.

AI is becoming more accessible

I believe that AI is becoming more accessible to people. Free models are getting good and there are more entry-level paid plans for people.

Cheaper Plans

A very good recent example is Perplexity’s partnership with Airtel in India to give away Perplexity Pro’s 1 year subscription for free. As users utilized the free subscription they realized that it was the same as the free version with just a few more capabilities. Perplexity here might be trying to tap in to a market of users who will build dependencies.

ChatGPT also release a Rs. 399 per month plan so that students and individuals wanting to utilize AI more effectively can claim that slight edge over people who don’t use it. To OpenAI it is a game of numbers in India and monetizing a niche market of young developers and working class who might not be able to pay 20$ a month but can spare 9$.

Open Source Models

There is also a wave of free open source models which are getting good and becoming accessible for free through Ollama. Some recommended models to try is Mistral, Llama by Meta and Gemma by Google. These models perform well enough with basic compute and memory. I have tried running Mistral 7B and Llama3.1 8B while trying to build local chat bots for some side projects. I see these models are good for most general purpose workflows like email reformatting, basic bash scripting and content generation. The only limiting factor here is the need for GPU memory and fast storage.

Projects like Open Web UI are also making it easier for organizations to deploy the above open source models on a chat interface to introduce their employees to AI in high compliance environments like finance and healthcare.

My recommendation

Keeping up with AI has been tough. The benefits of using AI tools have never been better. Paying for an AI tool can benefit you in some ways while choosing to use the free versions help you understand best-practices and prompting strategies to make most of the free models. Open source models will help you start learning more about the use cases of AI, but you’re limited by GPU memory.

Get one paid plan if you can. Experiment deploying local models using tools like Ollama. Explore more powerful models for specific tasks using platforms like HuggingFace which allow you to pay for the compute/memory and GPU for only the resources you have used on an hourly basis. This will help you understand the benefits and challenges of using AI.

In early 2021 I was tasked to focus on the security aspects of my backup project including working on securing various secrets and the encryption architecture of files being backed up. With my traditional sense of software design and development, I added dataflow diagrams, high-level design diagrams and low-level design diagrams. I thought I went the extra mile by adding the class diagrams, database schema structures and other details into my document.

I happily scheduled a design review with the architect (my mentor) for the file encryption design. In the design review my mentor asked me a number of questions which I had no answers to. “Where is this encryption key stored?”, “What will you do if someone takes a memory dump and the encryption key is leaked?”, “If the encryption key is leaked, will you re-encrypt all the files?”, “How many encryption keys will you store?”, “How will you encrypt the encryption key, and who will encrypt that key?”. All these questions were difficult to answer, I had focused only on one part of the problem which was encrypting the file and storing the encryption key, I had missed all the other points.

My mentor then asked me to do something called as threat modelling and come back. This blog pretty much sums up the theory about what I learnt from there on. There is information about what I learnt, and how I built my own methodology for threat modelling.

Threats and Vulnerabilities

When I started to do some research about the topic, I realised that there is a difference between a what is considered to be a vulnerability and what is a threat. I had interchangeably used these terms before and had seen others do it too.

What is a Vulnerability?

A vulnerability is any weakness of a computer system which can be exploited to negatively impact the security policy of the system.

Example 1, Role Based Access Control is not implemented on the database containing user data. Any database user can create, update, or delete user records.

What is a Threat?

A threat is any negative impact on a computer system which can be introduced through a vulnerability.

Example 2., A rogue developer accesses the database to delete all data from the user's database.

If you notice here, the above threat is a result of Example 1’s vulnerability. Since, there was no RBAC implemented on the database there is a threat for a rogue developer to delete all data.

So now, how do I find out all off these vulnerabilities and corresponding threats, I thought that it was not humanly possible to do that by my next design review call. I find out later that there is a way to categorize common threats and vulnerabilities using something called as a STRIDE model.

What is Threat Modelling?

Threat modelling is a pre-emptive process to identify, communicate, and understand threats and mitigations. The result of threat modelling is a threat model.

What, why and who?

What is a Threat Model?

A threat model is a structured representation of all the information that affects the security of an application.

In essence, it is a view of the application and its environment through the lens of security.

Why Threat Model?

When you perform threat modeling, you begin to recognize what can go wrong in a system.

In an ideal scenario, threat modeling should take place as soon as the architecture is in place. However, modelling all threats at this time might not be ideal.

No matter when you end up performing the threat model, understand that the cost of resolving issues generally increases further along in the SDLC.

Who Should Threat Model?

You. Everyone. Anyone who is concerned about the privacy, safety, and security of their system.

This intrigued my interests and did a little more research to find out some theory behind threat modelling and what are the best practices. Since I was in the early part of my career, I used to spend a lot of time doing theoretical research and analysis for any task assigned to me. I found the threat modelling manifesto on my research journeys (pre-ChatGPT era)

Threat Modelling Manifesto

This was formed by a group of security researchers and CISOs across the globe. More information in the references. I learnt Threat Modelling can be as simple as asking 4 questions while to define a design or solution.

Ask Four Key Questions

1. What are we working on?

2. What can go wrong?

3. What are going to do about it?

4. Did we do a good enough job?

At that time I was working on securing different secrets so, here is an example.

Example

1. What are we working on?

• The User-Agent will upload files to an S3 bucket designated to that user using AccessKeyID and SecretAccessKey of that user.

1. What can go wrong?

• A memory dump of the user-agent can yield the AccessKeyID and SecretAccessKey which can be maliciously used by the user to get free storage on Wasabi.

1. What are going to do about it?

• Create a wrapper on the remote API to fetch an STS token for the S3 client instead of storing the AccessKeyID and Secret Access Key on the user-agent.

1. Did we do a good enough job?

• We have moved the threat away from the user-agent to the Remote API.

• Securing the deployments from remote access and raising alerts on memory dumps or root commands can help us identify malicious behavior and remediate issues

The next section is a little interesting and I have designed it based on my experience as a security engineer. These steps are a mix of various sources stitched together with my experience.

3 Step Approach for Threat Modelling

Step 1: Decompose the Application

On a high-level we do the following things:

• Identify entities, entry points, exit points and interactions between various components of the system.

• Create a Dataflow diagram (DFD) highlighting different interactions and privilege boundaries.

Tool recommendation: Microsoft Threat Modelling Tool

Activities to decompose the application:

1. Start with a three-tier architecture

2. Identify external entities and interactions

3. Identify internal entities and interactions

4. Identify boundaries of the application.

5. Identify the trust zones (network layer, domain layer, interaction layer)

6. Identify 3rd-party components, integrations and zones.

7. Create a data-flow-diagram with the above information.

It can be as simple as a diagram like this:

In the above diagram you can see the dataflows, network boundaries and interactions between different components of the application.

Step 2: Categorize and Rank Threats

Step 2.1 Threat Categorization

A threat categorization such as STRIDE is useful in the identification of threats by classifying attacker goals such as:

• Spoofing

• Tampering

• Repudiation

• Information Disclosure

• Denial of Service

• Elevation of Privilege

Step 2.2 Threat Ranking

Risk = (Probability of Threat) x (Cost to organization)

To be holistically analyzed with stake holders and understand the risk appetite.

This is a very deep topic, I haven’t understood this yet fully, but from what I know you must present the inherent threats and the impact of the threat (cost) to the business and the business decides what is a threat worth fixing.

An example is, there is a vulnerability that the master key of the application is stored with a single admin user. The threat is that the admin user can be malicious and delete the master key or use that to decrypt all secrets.

The business might consider calculate the risk to be low if the admin user is the CEO of the company or the business owner himself. The probability is near zero so we might not need to worry about engineering to fix the threat immediately. Now, there are other risks here “what if the CEO is not too secure with how he manages secrets?”, so the CEO will accept the risk. There are lots of scenarios for something as simple as a password. I will try to write another blog sometime later to speak about another topic called Shamir Secret Sharing which can solve the problem of distributed secrets. Before I digress more let’s move to step 3.

Step 3. Identify Countermeasures & Remediations

Common Threat types and Counter Measures

At the end of the threat modelling activity, you will have a list of threats which you now have to sit and categorize. This is a difficult task since you need to see if your design fully mitigates the risk or not. For some threats, the cost of development might be high and they might eventually go into the bucket of non-mitigated threats. It’s good to document it so that there is a record of what were the threats identified and what were the mitigations put in place.

Threat Profiling

• Non mitigated threats: Threats which have no countermeasures and represent vulnerabilities that can be fully exploited and cause an impact.

• Partially mitigated threats: Threats partially mitigated by one or more countermeasures and can only partially be exploited to cause a limited impact.

  • Note: A threat is categorized as partially mitigated only when the risk level is considerably reduced from its state as a non-mitigated threat.

• Fully mitigated threats: These threats have appropriate countermeasures in place and do not expose vulnerabilities.

This is the theory behind threat modelling. In a future blog, I will take an example of threat modelling and walk through the 3 steps to help understand threat modelling better. I will also show the usage of tools like Microsoft Threat Modelling to build a threat model.

References

[1] Threat Modeling | OWASP Foundation

[2] Threat Modeling Process | OWASP Foundation

[3] Download Microsoft Threat Modeling Tool 2016 from Official Microsoft Download Center

[4] Threat Modeling Manifesto

[5] https://online.visual-paradigm.com/diagrams/templates/threat-model-diagram/threat-modeling/;VPSESSIONID=729E30DEF0017DF4ABDA751E05F09972

With this article I want to see if I still have it in me to write an article without the help of AI tools. I will take the good old approach of using Google Search and Microsoft Word Review / Grammarly for grammar. I must say, I have spent more than 6 months adding my thoughts slowly in to this article. I am reminiscing my days in high school where I had to write an essay of about 500-700 words in less than 30 mins. Little did I know that I would be so spoilt with AI tools to write a simple article about how AI is taking over my job.

The more I think about it the harder it gets for me to write this article. I started writing this blog on December 31st 2024, so whenever this blog is published, you will know how bad AI has affected my ability to write a blog without the help of AI tools.

If you find any grammatical errors in this content it probably means that I have been successful in not using any AI tools to write this content. XD I may have used it to review the content, but not change any content, but guess what? You will never know.

Early 2023 - Don’t trust AI, Learn to Validate information

In the early months of 2023 I read a HackerNews article about ChatGPT and how it can write poems for you, create articles and summarize content provided to it. Others in my office also catch the wind about ChatGPT. Every technical discussion in my workplace ends with “Have you tried to ChatGPT it?” or “What does ChatGPT say about this?”.

I am annoyed since I hadn’t caught on to ChatGPT for technical research and every point I made in a design discussion would be challenged with a quick summary obtained from ChatGPT. With about 2 years of experience it feels like everything I know about software engineering, cloud and other things skills are being challenged. My mentors, stakeholders could simply prompt ChatGPT and validate whatever I am saying. It is super easy now for junior engineers to challenge me and others.

Given enough context, anyone could propose a solution or provide a rebuttal with reasonable confidence. ChatGPT in 2023 was in its initial stages and was not very accurate, it would make lots of mistakes and some points were easy to dismiss with a quick Google search or documentation link. I was skeptical about using the results for any productive work.

This one time my colleague (you know who you are) presented information output by ChatGPT as a fact and in the end it turned out to be quite the opposite.

This was the first major lesson for me. Never trust AI blindly, validate all information provided by it.

2024 I start using AI for everything; building a knack for prompting and validating

In 2024 ChatGPT, Microsoft Copilot and Claude AI improved quite a bit in terms of technical accuracy. There were occasional glitches and misinformation which were easy to deal with by providing additional context and simple Google searches about the given topic.

Intrigued by AI tools, I start to explore different versions of the paid subscriptions. In the last half of 2023 up to the mid of 2024 I bought the Pro version of ChatGPT for longer conversations, the latest model and priority access in times of high demand on OpenAI servers. Post that, every minor inconvenience to write a piece of code slowly changed into a ChatGPT prompt followed by a Ctrl + C and a Ctrl + V. Creating complete packages in Golang with passing unit tests became a breeze. A task for writing utility libraries for encryption and decryption with unit tests took about 10 mins. This is compared to the time when creating an encryption library in .NET Core was one of my first tasks when I joined as an Associate Software Engineer. The estimate on the task was 6 weeks. The process involved doing the research, design, discussion, approvals and security review before actual development. Now, all of this was done with documentation and unit tests. You could also ask ChatGPT to do a security review and give back a threat model and it would do it for you.

Suddenly I deliver 2x-5x more work than without these technologies. It’s surprising that I am not using 100% of the skill sets I needed a year back for the same job role. I spend more time understanding the problem and the use case to craft the right prompt to help me solve the problem. The focus is more on what the business logic must be rather than what are the utility libraries we need, the interfaces etc. Validating information provided by AI is the most important skill I am building now.

Claude for scripting

In the second half for 2024 I start using Claude AI. I use it for absolutely everything, including generating code snippets and bash scripts as I have to work with Linux and Windows environments. I start working directly with customers in the sales engineering team, I realize the true power of AI in software engineering. I understand the customer request and convert it into a prompt.

I often need a simple bash script to install something, or a bash script to automate a whole lot of commands that the customer would otherwise have to manually run. A simple example is about when we containerized our back up application. We now had to tell the customer to install docker, then change directory to the location where the docker compose file was and multiple steps before the container was deployed. The first customer we deployed this for without AI’s help, took around 1 hour and 30 mins.

After this call, my team and I sit together and craft a prompt with Claude AI describing the issues in our first customer deployment and what were the steps that would need to be automated with a simple bash script. Claude built the initial script which we incrementally modified with additional prompts.

The next time we had to do a customer deployment, we were ready with a setup script. The time for deployment in customer environments reduced to less than 15 mins; this includes the time it takes to download the binaries and the docker image. Whenever the customer faced an issue with the bash script, it’s easy to feed the error back to Claude and ask it to fix it.

Building the same enhancement a year ago (prior to 2022) would have been estimated to take way more time; 3 staff months at the least.

God mode development with Claude + Cursor

In Q2 of 2024, while I was improving my ability to use Claude effectively, my colleagues pick up an AI powered IDE called Cursor. The editor prompts the next logical piece of code in the application based on the context of what you’re typing. It is the best thing to happen to me as a software engineer. Writing golang code is suddenly effortless and some of the cases you would have missed is just a Tab away.

Our design discussions and ideas streamlined into code super easily. I boilerplate architecture and code structures in Claude and give it to my team, and they use Cursor to fill the business logic. Everything is so simple and effortless.

Our goal is to backup a 25 TB database. Where do we get this database was the question. You guessed it right!

“I want to test the back up of a 25 TB database, I want to generate 25 TB of random data and then load this data as rows to my database. Give me a combination of bash and SQL scripts to generate and load this data in the least time possible. I have enough CPU, RAM and Storage available. I want to be able to build this test database in the next 2 days.”

Boom a script is ready. My mentor and I make some more minor tweaks to the script and run it overnight in parallel; we actually ran the script to load 50 TB by mistake.

Oh God! What do we do now? We decide to go for the backup test, and start the backup tool. The database backup takes 17 hours (we do a local test with two 72 cores, 1.5 TB of RAM machines connected physically using 10 GBe ethernet) and the restore about the same time. The 36 hours to verify the contents turn out to be the most nerve wrecking. With Claude and Cursor we did it! We beat the expectation by 2X.

All of this was done in less than 2 months, building the capability to back up a 50 TB database would have been estimated for 6-12 months otherwise.

2025 the path to become a 10X developer with AI

In late 2024 Claude, ChatGPT and other tools introduce the ability to create projects. My team starts using it and we get a enterprise account for all the team members. It is super easy to create a shared context and chats using projects. All projects have a shared context and information regarding requirements, answering styles etc. AI models are more advanced and we find new ways to ingest software development projects.

What I have understood.

I understand the need of effective prompting to derive the best results out of the AI tool. The AI is trying to answer your prompt with the least number of tokens possible; which means there is a lot of data it might not consider while providing you with a response. It’s important to be clear in your prompt what you expect from a single query. You must be able to build context and ask the AI tool solve the problem in parts similar to how you would interact with humans in the real world. Do not rely on it solely yet, use your own judgement and experience to review the results. I know this might not be the case in a couple of months.

As a leader in my organization says the next big programming language is going to be english; which I agree with. Prompt engineering for specific domains where R&D has never explored might see new light with the cost of experimenting with code and research reducing significantly.

Let me see if I can keep this up and write a few more with my experiences; share some best practices etc. I will try not to use AI tools to generate content, so that this experience is as authentic as possible and I don’t forget how to write sentences and paragraphs in English.

In this blog, let's explore what a disaster recovery plan is, why we need it, and how it contributes to an organisation's broader business continuity plan.

The objective is business continuity

Business continuity is the capability of an organisation to continue operations of products and services following any disruptive events. These disruptive events include natural disasters like earthquakes, floods and solar storms; other events like terror attacks, arson, security breaches and data breaches. Disruptive events can also be socio-economic factors like global recession, trade deficits, and loss of customers.

Organisations plan and create systems to prevent and prepare for recovery from any of the above mentioned threats. The plan enumerates a range of disaster scenarios and lists the roles, responsibilities and steps to recover regular trade. The key focus is on preparedness, protection, response and recovery strategies.

While business continuity focuses on the overall resilience of an organisation, disaster recovery planning is a crucial subset that specifically addresses the restoration of IT systems and technology operations.

Disaster recovery planning is a subset of Business Continuity

Business continuity (BC) and disaster recovery (DR); often collectively referred to as BCDR, are two related but distinct approaches to ensuring an organisation's resilience. BC is a proactive strategy that focuses on minimising risks and ensuring the organisation can continue to operate and deliver products and services during a disaster event by defining ways for employees to continue their work. On the other hand, DR is a reactive subset of BC that concentrates on the specific steps needed to resume IT systems and technology operations after a disaster occurs, and is implemented only when a disaster actually strikes.

To develop an effective disaster recovery plan, it is essential to first understand what the business needs to protect. This involves creating an inventory of critical assets, identifying stakeholders, and cataloging procedure and business process documents.

Understand what the business needs to protect

Create an Inventory

What do we need to protect?

• IT Equipment: Catalog all hardware, software, and network infrastructure, including servers, workstations, routers, and storage devices.

Who are we dependent on?

• Contractors: List all external contractors and their roles in maintaining or supporting the organisation's IT systems.

• 3rd Party Vendors: Identify all vendors providing essential services, products, or support to the organisation's IT operations.

Where do we keep data and where do we backup to?

• Primary Sites: Document the main locations where business operations and IT systems are housed.

• Recovery Sites: Identify alternative locations that can be used to resume operations in the event of a disaster.

Identify Stakeholders and key responsibilities

Who are the decision makers?

• Executive Management: Support, approve, and communicate the disaster recovery plan; make critical decisions during disasters.

• Business Unit Leaders: Collaborate with IT to identify critical processes; provide input for BIA; develop business continuity plans; ensure team awareness; participate in testing and training; coordinate during disasters.

Who will implement and maintain it?

• IT Department: Develop, implement, and maintain technical aspects of the plan; prioritise IT system recovery; ensure data integrity and availability; conduct testing and updates.

Catalog Procedure Documents and Business Process Documents:

• Procedure Documents: Create and maintain a comprehensive set of documents detailing the step-by-step procedures for critical IT operations, such as system backups, data restoration, and emergency response protocols.

• Business Process Documents: Document all essential business processes, including their dependencies on IT systems, to ensure a clear understanding of how technology supports the organisation's operations.

Once the critical assets and stakeholders have been identified, it is crucial to collaborate with the business to determine the potential impact of a disaster. This involves conducting a Business Impact Analysis (BIA), Threat and Risk Analysis (RA), and Impact Analysis.

Collaborate with Business to decide what is the impact of a disaster

Business Impact Analysis (BIA):

• Identify and prioritise critical business functions and processes.

• Determine the potential impact of disruptions on each function or process, including financial losses, reputation damage, and regulatory consequences.

• Establish recovery time objectives (RTOs) and recovery point objectives (RPOs) for each critical function or process.

Threat and Risk Analysis (RA):

• Identify potential threats to the organisation's IT systems and operations, such as natural disasters, cyber-attacks, and equipment failures.

• Assess the likelihood and potential impact of each threat.

• Develop strategies to mitigate or eliminate identified risks.

Impact Analysis:

• Evaluate the consequences of disruptions on the organisation's overall operations, including the impact on employees, customers, and stakeholders.

• Identify the interdependencies between various business functions and IT systems.

• Determine the resources required to maintain critical operations during a disaster event and to recover from disruptions.

With a clear understanding of the potential impact of a disaster and the critical assets that need protection, the organisation can finally build a comprehensive Disaster Recovery Plan (DRP). The DRP should outline the scope, objectives, and general contents necessary for effective disaster recovery.

Finally build the Disaster Recovery Plan

A disaster recovery plan is a document with information about how to resume operations from any disruptions. The disaster recovery plan mainly consists of an organisation’s IT infrastructure. The goal of the disaster recovery plan is to minimise data loss, recovery time and ensure system integrity and availability is returned to an acceptable level.

With the BIA, RA and impact analysis reports, we can identify the the impacts of disruptive events and sets the context for RPO and RTO objectives.

Scope of a DRP

The scope of a DRP encompasses all critical IT systems and infrastructure that support the organisation's core business processes.

Objectives of the DRP

1. Minimise downtime and data loss during a disaster event

2. Ensure the timely restoration of critical systems and applications

3. Maintain the integrity and availability of data and systems

4. Provide clear guidance and direction to staff involved in the recovery process

5. Comply with regulatory requirements and industry best practices

General contents of a disaster recovery plan

1. Introduction

   • Purpose of the DRP

   • Scope of the plan

   • Objectives of the plan

2. Roles and Responsibilities

   • DRP team members and their contact information;

   • Roles and responsibilities of each team member

3. Incident Response

   • Incident detection and reporting procedures

   • Incident classification and prioritisation

   • Communication plan (internal and external)

4. Inventory

   • Inventory of critical IT systems and infrastructure

   • Identify backup tools and mechanisms for different workloads and infrastructure

   • Primary Site information

   • Secondary Site information

   • Off-site Backup Location

5. Business Impact Analysis (BIA)

   • Identification of critical business processes

   • Recovery Time Objectives (RTOs)

   • Recovery Point Objectives (RPOs)

   • Prioritisation of recovery efforts

6. IT Systems Recovery Procedures

   • Backup and data replication procedures

   • Step-by-step recovery procedures for each critical system

7. Vendor and Third-Party Coordination

   • Contact information for key vendors and third-party service providers

   • Procedures for coordinating with vendors during a disaster

8. Testing and Maintenance

   • Schedule for regular DRP drills, testing and exercises

   • Procedures for updating and maintaining the DRP

   • Post-incident review and lessons learned

9. Appendices

   • Contact lists (employees, vendors, stakeholders)

   • System and network diagrams

   • Copies of critical documents and agreements

By developing a well-structured Disaster Recovery Plan that encompasses all the essential elements discussed in this blog post, organizations can significantly enhance their ability to recover from disruptive events and ensure business continuity.

Conclusion

Disaster Recovery Planning is crucial to an organisation's business continuity plan. Understanding the business's needs, risks, and potential impacts is key before creating such a plan. The overall cost and complexity of the disaster recovery plan may vary depending on these needs.

The first step is to document the disaster recovery plan. Regular testing and DR failover during maintenance windows can help the IT team identify gaps in the plan and suggest improvements.

When major infrastructure is changed or added, the DR plan must be updated with the latest backup procedures and recovery mechanisms. A regular test plan should also be implemented.

Businesses should regularly audit their Disaster Recovery plans and DR drill documentation.

References

Business continuity planning

What Is Disaster Recovery? - Features and Best Practices

What is a Disaster Recovery Plan (DRP) and How Do You Write One?

My career began in 2020 amidst the lockdown, a challenging time for remote work adaptation. Initially, the learning curve was steep due to the lack of in-person interactions. Resolving simple queries involved cumbersome steps: checking availability, sending messages, and hoping for timely responses. Often, the wait for guidance led to either self-resolution or forgotten issues.

The most challenging part was the early morning standup meetings. After working late nights, waking up at 9 AM to share updates was exhausting. I often struggled to articulate my progress and questions. To address this, I started noting down my daily activities and queries using my phone's notes app. This practice allowed me to efficiently communicate during standups, even when partially awake.

Adding some structure

By mid-February 2021, while working on a complex component, I faced the challenge of effectively sharing and recording vast amounts of research, ranging from articles and forums to research papers. Initially, I shared these resources via chat threads, but as discussions expanded, it became difficult to track and retrieve information.

I attempted to organise this information in a Notepad file, but soon faced issues with chronological order and relevance. This led me to explore other tools.

Starting with Notion

I revisited Notion, an app I had previously registered for but seldom used. I created daily pages titled with the date and sections like “What did I do yesterday?” and “What will I do today?” This format provided clarity and ease in tracking my activities and action items. However, after two months, I encountered difficulties in retrieving specific information from the daily records.

Calendars and databases on Notion

To overcome this, I utilised Notion’s calendar database feature, organising daily updates and research into a calendar grid. I adopted descriptive titles for each entry, facilitating easy retrieval through the search function. This system proved highly effective, and I began creating a dedicated calendar database for each month. It became an invaluable resource for recalling solutions and sharing knowledge with colleagues.

My daily routine journaling

I dedicate 15 minutes daily to reflect on my accomplishments and pending tasks. Summarizing discussions and my contributions helps maintain mindfulness about my work and ensures no task is overlooked. This practice has streamlined my morning routine, offering a clear starting point each day.

Sharing knowledge about previously encountered problems and their solutions has become effortless. I am always just a search away from finding and sharing relevant articles or commands with my teammates.

Conclusion

Keeping a work journal is like having a secret work buddy. It's your daily roadmap, highlighting what needs doing and celebrating what you've done. It saves you from those awkward "I forgot" moments in meetings, acting like your own work highlight reel. Plus, it's a fantastic way to see how much you're growing in your job. And when work gets too much? Jotting things down is like a mini stress-buster session. So, in a nutshell, a work journal is your go-to for staying organized, being on top of your game, and keeping cool under pressure.

The first level of disaster recovery is to back up data to a magnetic tape or disk drive stored off-site. In this blog, I will provide a quick overview of off-site backups.

Off-site Backup

The goal of offsite backup is to store data in a different location from its origin. Make a copy of the data over the network to an off-site storage media. Off-site storage media includes tape, network-attached storage (NAS), or a cloud storage solution. The storage of off-site data is also known as vaulting.

Data is crucial for any organization, but what happens when disaster strikes the main data center? This is where off-site backups become essential. They provide a secure and accessible alternative to keep your data safe. Let's explore how off-site backups work and why they're so important for protecting your data.

Core Propositions for Off-site Backups

Protection against complete system failures

If a disaster or a cyber attack hits the main office, we can use off-site backups to get everything back up and running. This way, we keep the business going no matter what happens.

Data can be lost not just because of system crashes, but also if the hardware, like hard drives, breaks down. Hard drives don't last forever and can stop working, causing data loss and expensive delays. Drives with moving parts break down faster than those with flash memory, but all types have a limit on how much they can be used. Off-site backup in a different location is a good way to protect against losing data if a hard drive fails.

Geo-redundancy

The geographical separation of data in off-site backups is a critical aspect of a robust data protection strategy. By storing data in a location physically distant from the primary site, off-site backups provide a vital safeguard against local disasters such as fires, floods, earthquakes, or even man-made events like theft or vandalism. This separation ensures that, even if a catastrophic event were to completely compromise the primary business location and its on-site backups, the off-site data remains unaffected and secure.

Better security

In the case of tape backups and optical disc backups, storage media can be kept in isolation with physical security in place. Tapes are physically durable and immune to cyber threats, making them reliable for disaster recovery, especially when stored off-site. Cloud storage solutions on the other hand provide features like immutability (write once read many - WORM) which prevent data from being overridden or deleted.

For example, in the case of a ransomware attack on the on-site backup of an organization; off-site backups provide the required levels of immutability and physical isolation to ensure that business can continue.

Optimization of Space

Off-site backups ensure that your primary disks are not being utilized by backups. This saves space and would require less space upgrades to the storage array. Storing critical files of the operating system and applications ensures that the primary server runs smoother. Off-site backups using cloud storage and tape offer cost-effective solutions compared to traditional on-site backups.

Storage Destinations for Off-site backups

Network Attached Storage (NAS)

Network Attached Storage (NAS) is a popular storage solution. Linux and Windows operating systems by default allow network file storage mounting capabilities. Protocols like Network File Share (NFS) and Storage Message Block (SMB) are used to mount storages. Backup applications either mount the storage on to a local file system or session mount for data transfer. It's relatively easy to setup and is cost effective since no additional infrastructure is required. The network connection between two environments between different locations is the only requirement. The costs associated with network peering and management might be high.

Cloud Storage

Cloud storage stands out for its minimal upfront costs and flexible scalability. Users pay only for the storage they use, with the ability to easily adjust as needs change, avoiding the high initial investment in hardware and infrastructure required for on-site backups. Additionally, cloud providers handle maintenance, security, and infrastructure management, significantly reducing operational and maintenance costs.

Tape Storage

Tape backups, while requiring some initial investment in tapes and drives, are more affordable than establishing a full on-site data center. They are particularly cost-effective for long-term archival storage due to their low cost per unit of storage and long shelf life. Tapes also don't require energy for data storage, which further reduces ongoing costs. Tapes are physically durable and immune to cyber threats, making them reliable for disaster recovery, especially when stored off-site.

Conclusion

Implementation of off-site backups is more than just a safety measure; it's an essential component of the disaster recovery plan. By off-site data storage, organizations can safeguard against a spectrum of risks, from natural disasters to sophisticated cyber threats.

The utilization of cloud and tape storage options not only enhances data security but also offers a cost-effective and scalable solution to traditional on-site methods. This approach not only ensures business continuity in adverse scenarios but also contributes to the overall efficiency and resilience of the IT infrastructure.

References

What Is Off-Site Backup? | Definition from TechTarget

Off-site data protection - Wikipedia'

Offsite Data Backup [Top 5 Reasons for Data Security] | Liquid Web

Tape Storage vs. Disk Storage: Getting the Facts Straight about Total Cost of Ownership Calculations - Fujifilm Data Storage

In this blog, I will provide an overview of disaster recovery and introduce the seven levels of disaster recovery, setting the stage for a deeper exploration in future blogs.

Disaster Recovery

Disaster recovery is a crucial aspect of maintaining or re-establishing vital infrastructure and systems following a natural or human-induced disaster, such as a storm or cyberattack. It's essential for keeping all critical aspects of a business functioning despite significant disruptive events. Effective disaster recovery requires well-thought-out policies, procedures, and tools to ensure business continuity.

Measuring Data Loss and Recovery Time

In the event of a disaster, an organisation's primary goal is to restore all systems rapidly while minimising data loss. These objectives are quantified as Recovery Time Objective (RTO) and Recovery Point Objective (RPO):

• Recovery Time Objective (RTO): This is the duration required to restore infrastructure and data to resume business operations.

• Recovery Point Objective (RPO): This represents the acceptable amount of data loss, measured in time, from the point of the disaster.

The Need for a Secondary Site

A secondary location equipped with comparable infrastructure—like computing resources, storage, and networking—is necessary, particularly when the primary site is not immediately recoverable. The data restored at this secondary site is crucial for continuing business operations.

States of Infrastructure and Data Layers

The secondary site can be either active or passive. For instance, while the computing, network, and storage might be active, if the site lacks the necessary data (or state) to function as the primary site, data restoration is needed. In this scenario, the data layer is in a passive state, which impacts the RTO during disaster recovery.

Considerations for Your Disaster Recovery Plan (DRP)

To effectively establish a DRP, businesses must discuss their domain-specific needs to determine appropriate RPO and RTO requirements. For example, banks typically require very low RPO and RTO, aiming for minimal downtime, whereas a university or research organisation might tolerate some data loss and a longer recovery period.

From Backups to Continuous Data Replication: The 7 Tiers of Disaster Recovery

Achieving desired RPO and RTO goals involves understanding the different levels of disaster recovery, ranging from level 0 to level 6. Each level offers varying degrees of data protection and recovery speed, with increasing cost and complexity.

1. Level 0 - No Off-Site Data: This basic level involves storing data exclusively on-site, without off-site backups. It's the most cost-effective but carries the highest risk of total data loss in case of on-site disasters. Ideal for small, non-critical setups.

2. Level 1 - Backup Tapes Off-Site: Involves backing up data to magnetic tapes stored off-site. It's a more secure option than Level 0 but can be slow in data recovery. Suited for institutions where data recovery speed is not a critical factor.

3. Level 2 - Disk Backup Off-Site: Faster recovery is possible as data is backed up onto disk-based systems off-site. It’s more expensive than tape backups but allows for more frequent backups. Suitable for medium-sized businesses prioritising recovery speed.

4. Level 3 - Electronic Vaulting: Data is sent in batches to an off-site location at regular intervals. It strikes a balance between backup frequency and costs, ideal for organisations with moderate data-change rates.

5. Level 4 - Point-in-Time Copies: Offers frequent snapshots of data, providing multiple recovery points. This level is storage-intensive and ideal for businesses with high transaction rates or those maintaining critical systems.

6. Level 5 - Transaction Integrity: Ensures all transactions are captured up to the point of failure, offering high data integrity. It's technically complex and ideal for setups where transactional consistency is crucial, like financial institutions.

7. Level 6 - Zero or Near-Zero RPO: Provides continuous data protection with almost instantaneous recovery and minimal data loss. It's the most sophisticated and costly solution, suitable for large enterprises or critical government systems.

Conclusion

In disaster recovery planning, accurately defining Recovery Point Objective (RPO) and Recovery Time Objective (RTO) is crucial for business resilience. These objectives dictate how quickly and effectively a company can bounce back from disruptions. However, implementing these objectives through appropriate disaster recovery tiers involves a careful balance of costs and capabilities. A successful DR plan aligns with the organisation's risk tolerance and budget, ensuring that the level of investment is proportional to the potential risks and impacts. In essence, a well-crafted DR plan not only protects critical business functions but also aligns with the organisation's financial strategy, ensuring long-term stability and growth.

References

High Availability vs Disaster Recovery: What's the Difference and Why it Matters for Your Business

High availability

7 tiers of disaster recovery

Basic Steps for your Business Continuity & Disaster Recovery plan