This content originally appeared on DEV Community and was authored by Kalob Taulien
Lots of people talk about tech taking their jobs and automating humans out of a job.
And while that's sometimes true, there are a few subjects where people can't be replaced.
This is an article about how we use tech to automate big tasks at Arbington.com
But, once again, not everything can be automated (or can be automated but very poorly).
Automating big tasks
At Arbington.com we deal with a lot of scale. We have over 49,000 videos, over 4,500 hours of content, over 1,500 courses and lists of 10's of thousands of businesses, teachers and students to reach out to.
If I asked you to manually gather a list of 40,000 businesses in North America, could you compile that list with contact details in less than 3 days?
The answer to the above question is obviously no. People just can't operate that fast.
But computers can.
Using Python and less than 50 lines of code I was able to find business listing sites all over the web, scrape them, and collect data from each page.
This is all free information that's accessible to everybody. It's just A LOT of information.
And computers are good at doing one task over and over, for as long as they have reason to continue doing it. Whether that's scraping, checking if a site is down, or serving a website to thousands of views per minute.
Tools we use
Instead of hiring a small army of people to collect all the data we wanted, we wrote a small Python script that uses requests and Beautiful Soup 4 to get the page and parse the HTML. Then we used simple for loops to loop through all the data, and a handful of if statements to validate some basic data.
Then we put all that finalized data into a huge Python list and used airtable-python-wrapper,
Python and Airtable to bulk insert data into one of our tables.
The whole process took about 20 minutes to write code (and solve little bugs that came up with inconsistent data) and about 3 hours to run.
Once the code was written, we pointed it at a website and said, "Fly my beautiful butterfly, fly".
And then we waited.
If there's enough demand, maybe I'll post one of our scraping scripts to show you how we created a bot to gather and parse data, and put it into a useable table on Airtable. Leave a comment below if that interests you.
Learn Python. It's easy, and fun!
As a side note, if you're not familiar with Python you might want to consider learning it. It's the worlds #1 most popular coding language now.
And at Arbington.com we have over 35 python courses. Take them all for free with a 14 day free trial and then it's just $15/month after that. For 35 courses on Udemy that's going to be over $350. I'll let you do the math and comparison on that 😉
Human data is messy though
People, AKA: humans, are messy. We don't format phone numbers or email addresses the in a standardized way. People write:
test at gmail.com, test @ gmail . com, test [at] gmail [dot] com, and about a zillion other ways of trying to avoid spam bots from collecting their email addresses, phone numbers, business names, addresses, and so on.
First, we had to find a few directories that had relatively clean data in a consistent format.
Then we put all the data in Airtable and found the outliers with a simple filter and search. That looks like filtering for " " (a space) in an email address, for example.
Then, manually, we went and cleaned up the data so it was consistent for the entire team. From there, it became a people-focused task: emailing prospects one by one and answering questions.
We literally have hundreds of thousands of contacts now. And 95% of the data was collected through automated methods.
What would have taken a team of people MONTHS to achieve, took us a matter of days.
We still need people
Once we had all the data from our automated tasks, we needed to fill the gaps.
Not every business lists a phone number, email address or physical address. Not every portfolio website has an email address. And when they do, it's formatting incorrectly.
This is where people come in.
Instead of trying to code an automated tool that could account for every type of email address (including images) we simply said, "humans are better for this."
Humans have unparalleled pattern recognition. We can match an email address just by looking at it without reading it. Even with all the variations of email addresses out there.
So we went and manually collected the missing data.
People are simple better at tasks that involve creativity or advanced pattern recognition.
Could we have automated this? Absolutely. But it'd have taken us a long time to create a cool tool that would ultimately be thrown away.
Opportunity happens when people connect
The next big hurdle that tech couldn't solve was a simple conversation with our potential business partners.
We couldn't just email 40,000 businesses with one message. Businesses are unique, like people, and we needed to cater to each business differently.
Again, this is where people come in.
People can handle objections, consider situations, understand when to sell and when to back off, and customize each response to be friendly and helpful.
Automation can't do that.
Yes, of course it's a lot more work to hire a team of people to do this, but the end result is a much higher ROI than if we automated everything.
Automation can't replace people - that's the point I'm driving here.
When it comes to communication, creativity and empathy we simply cannot automate most of that (some exceptions apply, of course)
Should you fear losing your job as a dev, or being automated out before you land a job as a developer?
Nope. Absolutely not.
Sure, there are lots of no-code solutions out there. And when starting a startup, definitely take that approach! It'll save you a lot of pain of coding your own solutions, dealing with bugs and edge cases that users bring up.
But at the end of the day, your app/website/business has custom logic that likely can't be completely automated and will absolutely require human input.
Maybe in 2100 we'll have to open this conversation again, but for now you're safe as a developer.
For now, you can learn all your automation coding needs on Arbington.com.
- 💴 $15/month
- 🏫 14 day free trial
- 👩🏫 1,500+ courses
- 🔥 Certificate of Completion included in every premium courses
- Check out Airtable.com - we love this service so much we don't even want to get paid for referring them. ♥️
What's next? API's!
In the next article I'll talk about the API's we use, how we use them, and why we use them. It'll be pretty dev-heavy - so if web development is your cup of tea, stick around for that one.
This content originally appeared on DEV Community and was authored by Kalob Taulien
Kalob Taulien | Sciencx (2021-11-23T01:43:44+00:00) Automate everything… until you can’t.. Retrieved from https://www.scien.cx/2021/11/23/automate-everything-until-you-cant/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.