Creating software is my passion, and I’ve spent a lot of time reflecting on why it resonates with me so deeply. I spent far too long thinking about this, but I think I’ve narrowed down the reasons why I like creating software as much as I do. This document explores the various aspects that drive my love for software development.
Solving Problems
Fundamentally, I like solving problems. Real problems, faced by real people. It just so happens that the tool I find myself using to express this is often building software. Building websites allows me to solve problems like educational, or promotional, or organisational or analytical, for large businesses, or for individuals with a dream. The solutions I implement impact people. The Python scripts I use to automate things on computers solve problems for me, and also for other people (like Autoclass). I’d like to think that most pieces of code I’ve written are fundamentally designed to solve real problems, which, in turn, helps someone in their life.
This is why I find little reason to do or "grind" Leetcode problems. They're "problems" by definition, but there's no heart, no feeling behind these problems. There's no reward at the end except some green text saying all of your tests are passing. There’s no appreciation for what was accomplished because no one sees it. It’s just a number, reduced to a mere statistic which, to me, means nothing. I'm sure, in fact, I know it helps you improve as a developer, but in my career so far, I've had little reason to solve meaningless problems for numbers, when I can spend my time helping real people with problems that have more real world implications.
Technical Challenge
The technical challenge is another one I find particularly enjoyable. It's the process of figuring out the best way to solve a problem with the skills I have, but also thinking just a bit beyond my skills, as this will allow me to continue to expand my skills and to learn new things.
Knowing about which solution is the "best" comes from experience, and from being aware of what's going on in the industry. You may not know how something like Rust works, you might not understand how to use a particular library, but you know from your general knowledge that Rust is for low level applications, or x library is used to chart lots of data in a performant manner. This way you can use this high-level knowledge and use it to your advantage when you have a problem to solve.
Learning by doing is the only way to do it in the software space. Watching tutorials, or watching edutainment videos, or even watching someone build something is never, ever going to be as helpful as building something yourself. You'll hit bugs and issues that the developer in the video knew to avoid, but you didn't. You'll find a solution to that problem, and you can be proud of yourself for building something that wasn't spoon-fed to you by a wizard dev. Anyway, I digress.
Optimisations
You've now written a solution to a problem. It's bodged (thanks Tom Scott) together unorganised pieces of code, the program is slow, with only the happy path implemented (no error handling). You can think of a hundred ways of improving it. Even with all these issues, it works. It solves a problem.
Now's the time for all of the optimisations. Pick an issue with the biggest performance impact on execution time and solve it. Some things are inherently slow like making network requests in serial when they can be parallelised, or opening and closing files in a loop that runs 40000 times. These are usually my first things to check and improve. Once most, if not all, of these issues are resolved, one can think about the outcome of the entire process.
Personal Satisfaction
Your program is now doing something, anything, so fucking fast that nothing can compete. You utilised all the tools available to you to solve a problem in a few seconds which would have otherwise taken days, if not have been completely impossible to solve. The end result of all of this is a program that people can depend on for their work, and you can be proud of yourself for solving all the problems you solved along the way to making the best possible software you could.
Web Scraping
Being a web developer, I understand the process of creating and maintaining a website. Because of this knowledge, I'm also well suited for scraping data (read: exploiting) these websites. Web scraping is a domain that uniquely encapsulates many of the elements I love about creating software. It involves automating the extraction of data from websites, often circumventing anti-scraping measures put in place by these sites. This practice requires a blend of problem-solving, technical know-how, creativity, and optimisation.
Handling Anti-Scraping Measures
Many websites implement measures to prevent automated scraping. This is usually done as rate limiting, and obfuscated HTML structures. Overcoming these obstacles is part of what makes web scraping so intriguing. For example rate limiting can be managed by carefully managing the requests to avoid raising any suspicion.
Optimisation & Reliability
Optimising a web scraper is a key part of the development process. Efficiently managing requests, handling retries for failed requests, and ensuring data accuracy are all essential. Parallelising network requests is a common strategy to enhance performance. All of this must be done keeping in mind all of the possible failure points in the system. The website may change, the requests might be slower, or faster. There's a lot that can go wrong, which means there's a lot to ensure completely dependability and confidence in software you write.
Data Processing and Storage
Once the data is scraped, the next step is processing and storing it. This can involve cleaning and transforming the data into a usable format. For instance, JSON data might need to be parsed and converted into a format for a CMS somewhere, or uploaded to a CRM a client is using. There's hundreds of possibilities, and that's always something new to do.
Conclusion
In conclusion, my passion for creating software comes from solving real-world problems and tackling technical challenges. From building websites to automating tasks with Python scripts, with my work, I aim to make life easier for others.
The joy lies in the process: optimising code, overcoming obstacles, and pushing boundaries. Web scraping, in particular, embodies these aspects with its unique challenges and opportunities for innovation. Creating software is about making an impact, whether through automation, valuable websites, or data-gathering tools. The continuous learning and problem-solving involved keep me motivated.
Ultimately, my journey in software development is about building meaningful solutions and enjoying the creative process. As I continue to grow, I look forward to new challenges and making a positive impact with my work.