One of the hardest things about software engineering is evaluating alternatives. "Which package or technique will work the best here?" is a question I'm constantly asking myself before, during, and after tasks.

Which X is Best?

In the before phase, all you have to go on is the hype, your experience, and that of those around you. Solid hype can mean solid solutions or just a lot of money and effort behind it. It can still be an ok guide if you're looking to fit in or something. As far as how much effort it will take, that is harder.

Another thing in the pro column of being in the trendy space is that out of date documentation and blog posts won't exist... because there's no old version that exists. Maybe this suggests that if you truly break core APIs in your project you should rename it entirely. On the other hand, newer things may have a lot of breaking changes before committing to 1.0, so, shrug.

Anyways, I'm not #buildingathing using the flashiest new stuff. I don't love the flashiest new stuff and [old man rant you don't want to hear].

Anyways, when evaluating the new solutions to file uploads, I was overwhelmed, and had I decided to commit fully to the trendy solution, I would have been miserable.

"File uploads to AWS S3 in Rails" seems to just flow together so well, I spent a lot of time looking into solutions to that problem. Also, it was a problem I'd solved before. And tempted by Heroku's siren song to also do it using direct upload, I spent a day comparing plugins, configuring XML on AWS, which itself takes a bunch of recursive defining of terms in their documentation to even know what's going on.

This is about the time when I say...

"Hey, wait a second"

I was looking for "X" (a decent way to direct upload files to S3) because it seemed like a good way to accomplish "Y".

I don't actually care about X. I care about Y.

And back to the process of finding the "best x", this is actually something I do a lot. I head down one path, and it starts to suck, so I back track and head down a different one. And the possibilities and tweaks and configurations get smaller and smaller to switch... that is until the whole thing is taking forever.

Ok. So direct file uploads to S3 in Rails with Heroku sucks. Yes, there's a simple way to do it, but it's buried beneath years of searches pumping up solutions that don't work anymore, not to mention Heroku's own documentation is not awesome. I think we were excited about microformats and science and verified information at one point, and I think it had something to do with this. But then platforms realized they didn't need to accommodate a decentralized web and stopped caring (I guess?).

If I was focused on X (the method), I would be able to do it. It would suck, and maybe taken a couple more days (?), but when the search for alternatives to solve your problem is not fruitful, expanding the search might be the answer (although, you might find more content/library rot there too...).

In my case, this worked out great. I don't need S3 or consequently, direct upload to a data store. I need files... somewhere, through a not too shitty experience.

On not Heroku, this would probably mean opening up the file system. On Heroku (which is maybe why this branch of search was so fruitful at one point, and so rotten now), the solution IS S3 because it IS NOT the filesystem (on Heroku), or so it seemed.

So the biggest jump I could make here would be dropping Heroku. I didn't want to go that far, so I started looking into other ways to store files. If it's not the filesystem on your app server, and it's not a remote file storage system (like S3), well, the database is looking pretty obvious. So why not the database?

Old Reasons

The old reasons for why S3 for file uploads have something to do with Heroku's intentional limitations. But also, what were we doing with files during this time that this S3 remote filesystem became so important? We were uploading profile images. We needed a file system to serve images that everyone could see. Image-sized files had to be ok. AND WE WERE GOING TO SCALE SCALE SCALE OUR SOCIAL NETWORKS.

That's not at all what I need. Yes, I need "file" storage, but no, I don't need a million files, resized and duplicated and cropped to support everyone using this thing. 5000 people. Maybe 10 documents each (they're for uploading resumes and cover letters). I'm expecting it would take an actual million (I estimated) documents before I would blow through the $50/mo hosted postgres as far as size goes. I'd probably hit the row limit first, and it would probably be from one of the other tables.

Also, not only don't I need these files to be readable and writeable (to the web), I don't want that at all. These are resumes. I don't want them spread across the web. Additionally, I've seen enough "Whoops, our S3 bucket was public" type mistakes in the wild to not trust myself in configuring it properly without a lot of attention (yes, I can, but so could have these other people I've seen screw it up).

So database it is. It wasn't that hard, and it's changed much less than the gem/AWS stuff has.

The Lesson

My first guess, as well as all of the most immediate information for how to solve the problem, were both bad. But not insisting on the particular solution and revisiting the outcome led me to something much easier to implement and maintain.

You could interpret this as "laziness," one of the "three great traits of a programmer." Or maybe this is some kind of "Iterative Deepening Depth-First-Search" as applied to finding a solution at the application level.

Maybe the heuristic to know is just "if it hurts, stop."

By The Way

The titular "thing" is getting very close. If you're looking for a job, it can probably help you quite a bit. If you want to hear more, I'm always available at evanburchard.com/contact.