Search for a command to run...
As a software engineer with 5 years of experience, I work on the core backup & recovery features of Zmanda, an enterprise backup and recovery product. I have strong skills in software design, cloud-native development, and delivery. I also foster effective communication and collaboration among the development team, architects, product owners, and business owners. I contribute to some open-source projects and share my technical insights on my blog.
No comments yet. Be the first to comment.
Are you a reverse centaur?
The next skill is not just coding. It is knowing what problem is worth solving.
Cognitive Overload for Engineering Leads
They both solve different problems Offline and Offsite backup and disaster recovery is often confused as the same thing. Both terms describe data locality but are designed to address different risk in
The feeling of missing out
ChatGPT, Claude, Gemini, Meta AI and many other tools have made a significant impact on the way we interact with technology and source information on the internet. As I type this article, I have an itch to make a prompt to Claude AI to complete this article for me. I know that it is gonna become harder and harder for me in the future as this technology advances. I will try to write this article fully without the help of AI tools.
With this article I want to see if I still have it in me to write an article without the help of AI tools. I will take the good old approach of using Google Search and Microsoft Word Review / Grammarly for grammar. I must say, I have spent more than 6 months adding my thoughts slowly in to this article. I am reminiscing my days in high school where I had to write an essay of about 500-700 words in less than 30 mins. Little did I know that I would be so spoilt with AI tools to write a simple article about how AI is taking over my job.
The more I think about it the harder it gets for me to write this article. I started writing this blog on December 31st 2024, so whenever this blog is published, you will know how bad AI has affected my ability to write a blog without the help of AI tools.
If you find any grammatical errors in this content it probably means that I have been successful in not using any AI tools to write this content. XD I may have used it to review the content, but not change any content, but guess what? You will never know.
In the early months of 2023 I read a HackerNews article about ChatGPT and how it can write poems for you, create articles and summarize content provided to it. Others in my office also catch the wind about ChatGPT. Every technical discussion in my workplace ends with “Have you tried to ChatGPT it?” or “What does ChatGPT say about this?”.
I am annoyed since I hadn’t caught on to ChatGPT for technical research and every point I made in a design discussion would be challenged with a quick summary obtained from ChatGPT. With about 2 years of experience it feels like everything I know about software engineering, cloud and other things skills are being challenged. My mentors, stakeholders could simply prompt ChatGPT and validate whatever I am saying. It is super easy now for junior engineers to challenge me and others.
Given enough context, anyone could propose a solution or provide a rebuttal with reasonable confidence. ChatGPT in 2023 was in its initial stages and was not very accurate, it would make lots of mistakes and some points were easy to dismiss with a quick Google search or documentation link. I was skeptical about using the results for any productive work.
This one time my colleague (you know who you are) presented information output by ChatGPT as a fact and in the end it turned out to be quite the opposite.
This was the first major lesson for me. Never trust AI blindly, validate all information provided by it.
In 2024 ChatGPT, Microsoft Copilot and Claude AI improved quite a bit in terms of technical accuracy. There were occasional glitches and misinformation which were easy to deal with by providing additional context and simple Google searches about the given topic.
Intrigued by AI tools, I start to explore different versions of the paid subscriptions. In the last half of 2023 up to the mid of 2024 I bought the Pro version of ChatGPT for longer conversations, the latest model and priority access in times of high demand on OpenAI servers. Post that, every minor inconvenience to write a piece of code slowly changed into a ChatGPT prompt followed by a Ctrl + C and a Ctrl + V. Creating complete packages in Golang with passing unit tests became a breeze. A task for writing utility libraries for encryption and decryption with unit tests took about 10 mins. This is compared to the time when creating an encryption library in .NET Core was one of my first tasks when I joined as an Associate Software Engineer. The estimate on the task was 6 weeks. The process involved doing the research, design, discussion, approvals and security review before actual development. Now, all of this was done with documentation and unit tests. You could also ask ChatGPT to do a security review and give back a threat model and it would do it for you.
Suddenly I deliver 2x-5x more work than without these technologies. It’s surprising that I am not using 100% of the skill sets I needed a year back for the same job role. I spend more time understanding the problem and the use case to craft the right prompt to help me solve the problem. The focus is more on what the business logic must be rather than what are the utility libraries we need, the interfaces etc. Validating information provided by AI is the most important skill I am building now.
In the second half for 2024 I start using Claude AI. I use it for absolutely everything, including generating code snippets and bash scripts as I have to work with Linux and Windows environments. I start working directly with customers in the sales engineering team, I realize the true power of AI in software engineering. I understand the customer request and convert it into a prompt.
I often need a simple bash script to install something, or a bash script to automate a whole lot of commands that the customer would otherwise have to manually run. A simple example is about when we containerized our back up application. We now had to tell the customer to install docker, then change directory to the location where the docker compose file was and multiple steps before the container was deployed. The first customer we deployed this for without AI’s help, took around 1 hour and 30 mins.
After this call, my team and I sit together and craft a prompt with Claude AI describing the issues in our first customer deployment and what were the steps that would need to be automated with a simple bash script. Claude built the initial script which we incrementally modified with additional prompts.
The next time we had to do a customer deployment, we were ready with a setup script. The time for deployment in customer environments reduced to less than 15 mins; this includes the time it takes to download the binaries and the docker image. Whenever the customer faced an issue with the bash script, it’s easy to feed the error back to Claude and ask it to fix it.
Building the same enhancement a year ago (prior to 2022) would have been estimated to take way more time; 3 staff months at the least.
In Q2 of 2024, while I was improving my ability to use Claude effectively, my colleagues pick up an AI powered IDE called Cursor. The editor prompts the next logical piece of code in the application based on the context of what you’re typing. It is the best thing to happen to me as a software engineer. Writing golang code is suddenly effortless and some of the cases you would have missed is just a Tab away.
Our design discussions and ideas streamlined into code super easily. I boilerplate architecture and code structures in Claude and give it to my team, and they use Cursor to fill the business logic. Everything is so simple and effortless.
Our goal is to backup a 25 TB database. Where do we get this database was the question. You guessed it right!
“I want to test the back up of a 25 TB database, I want to generate 25 TB of random data and then load this data as rows to my database. Give me a combination of bash and SQL scripts to generate and load this data in the least time possible. I have enough CPU, RAM and Storage available. I want to be able to build this test database in the next 2 days.”
Boom a script is ready. My mentor and I make some more minor tweaks to the script and run it overnight in parallel; we actually ran the script to load 50 TB by mistake.
Oh God! What do we do now? We decide to go for the backup test, and start the backup tool. The database backup takes 17 hours (we do a local test with two 72 cores, 1.5 TB of RAM machines connected physically using 10 GBe ethernet) and the restore about the same time. The 36 hours to verify the contents turn out to be the most nerve wrecking. With Claude and Cursor we did it! We beat the expectation by 2X.
All of this was done in less than 2 months, building the capability to back up a 50 TB database would have been estimated for 6-12 months otherwise.
In late 2024 Claude, ChatGPT and other tools introduce the ability to create projects. My team starts using it and we get a enterprise account for all the team members. It is super easy to create a shared context and chats using projects. All projects have a shared context and information regarding requirements, answering styles etc. AI models are more advanced and we find new ways to ingest software development projects.
I understand the need of effective prompting to derive the best results out of the AI tool. The AI is trying to answer your prompt with the least number of tokens possible; which means there is a lot of data it might not consider while providing you with a response. It’s important to be clear in your prompt what you expect from a single query. You must be able to build context and ask the AI tool solve the problem in parts similar to how you would interact with humans in the real world. Do not rely on it solely yet, use your own judgement and experience to review the results. I know this might not be the case in a couple of months.
As a leader in my organization says the next big programming language is going to be english; which I agree with. Prompt engineering for specific domains where R&D has never explored might see new light with the cost of experimenting with code and research reducing significantly.
Let me see if I can keep this up and write a few more with my experiences; share some best practices etc. I will try not to use AI tools to generate content, so that this experience is as authentic as possible and I don’t forget how to write sentences and paragraphs in English.
