My ramblings on the tech industry, life, and random experiences.

Wedge Martin

On Full Stack

(or: Your Full Stack is Whack)

April 14, 2016

Not to wax too cynical on the subject, but I'm often cautious when I hear the term 'full stack developer' or see it on a resume.

Now, I'm fully aware that the world has evolved beyond having to schlep gear to a colo and register your own AS number in order to be a presence on the Internet, and I'm fully aware of the amount of mileage that you can get out of a pure cloud implementation without the need to really support ( or even understand much of ) the underlying infrastructure. However at some point, if you're in the SaaS business, you'll need to scale up. And scaling up means understanding more of the stack than the pile of third party libraries/SDKs/frameworks that make up your 'front end' and your 'back end'.

So let's take a look at what makes the mythical 'two halves' of our SaaS platform tick.

The Front End

I've been accused before of being 'anti-framework', but this is not the fairest description. I find frameworks useful in many cases, and I certainly use a few, but it's important to use them for the right reasons. To load 200kb of JavaScript on a page simply to use the jQuery selector is a pretty good indication that you should be getting by on a couple of hand written functions that can get the job done with far fewer bits.

I overheard a conversation a few years ago between two front end developers in which one asked the other which 'framework' his team at YouTube was using for their front end. When the YouTube front end engineer replied that they "didn't use one" and that their team lead required that all of their JS code fit into a single TCP packet, the other stood staring as if some mechanical malfunction had triggered an error inside of his exception handler. This made me smile.

TCP is over 40 years old, and even though the 64k maximum window size seemed generous in the 70s, today it means that data transfer over latency when you're trying to stuff half a meg of minified JavaScript over a connection can get ugly. Modern day networking has a few different provisions for handling this, such as SACK, but it requires that both ends support it, which isn't always going to be the case. For this reason, it's often not a great idea to have all of your front end code packed into a big monolithic file, but broken into a few files, ideally segmented by functionality so that not all files are required for every page load on the site. This ensures that while a user at home may have a 100mbit pipe to the internet, you're not limited to the effective 10mbit throughput you'd get over large latency. Granted, serving your static assets up via a CDN can help to mitigate the latency issue by ensuring that a replica of the data is closer to the edge where the user is connected, but it's better to design for the worst case scenario.

I asked a full stack developer a couple of years ago during an interview what 'Transfer-Encoding: chunked' meant in an HTTP header and he couldn't even produce a guess. My point is not to say that he wasn't a particularly good developer, but shouldn't the networking stack be considered part of the 'full stack', considering it is the artery that delivers the content to the consumer?

What also worries me is when a front end developer insists that the back end do work that the front end can easily do, such as sorting, transforming, or merging data when having the back end do the work could involve creation of temporary tables for buffering sorts, or creating extra overhead for manipulating the data when the data transforms are only necessary for a particular view of the data. Client CPU is essentially free. Unless you are going to impact user experience - use it. Keep your API endpoints lean, fast, and efficient as possible, or you'll be paying the price.

A while back I saw a tech blog post where a team had used a pool of async workers to dynamically generate images for icons which were essentially colored boxes with a couple of text characters over top of them. Figuring out why this seemed like a better option than just storing the characters and a hex string for color for each object and having the client browser visualize the icon will require that I take a trip to the pub and marinate my brain in a few gin and tonics.

The Back End

Back here is where I generally get irritated.

There's a lot more to building a scalable API than knowing a high level language and using a popular ORM to talk to a database, but it's a start. Picking up a good understanding of cache engines, service oriented architectures ( and knowing when/why/where to use them ), load balancers, storage systems, IP and layer 2 networking will get you a long way as well. Some other concepts I find a lot of full stack developers fall short in knowing include:

I/O. Data store I/O is a killer. What may very well be the first rule of thumb in designing scalable back ends is: if at all possible, do not touch disk. At the very least, fetching data from a physical disk or SSD should by far be the exception, and not the rule. The slowest point in your system where bottlenecks are most likely, is your persistence store. There are wonderful memory based data stores to be used for cache, so use them. And when a file/document/object/rhinocerous/whatever is in cache, and an update to it occurs, write the update to cache, don't rely on the cache expiration to update what's there. Otherwise you wind up with a stampede of app servers all trying to populate the missing data at the same time, and your whole platform can go straight to hell in a hand basket.

Understand your app server. You should know intimately the pros and cons to having a pool of single threaded app servers vs a multithreaded app server.

If you can do it, process asynchronously. I'm not talking about asynchronous app servers such as NodeJS, but rather the concept of data processing via API calls. Unless you absolutely need the processed data in the payload in real time, you should probably consider dispatching it to a farm of workers and using a message bus or similar concept to send notifications when the processing is done. One of my current favorites is RabbitMQ + WebStomp. Free up your app server threads to handle more requests. Sure your user_details API endpoint only takes 4ms on average to respond, but if the client request for it has been dispatched to one of your single threaded app servers which is currently processing an API call which takes 2 seconds, the browser that is requesting that user data is going to wait, and the effective response time for the user_details call will be wahtever's left of the 2 second API call plus 4ms. NewRelic uses a simple header injection technique to measure request queueing. If you're a customer ( and I can't see why you wouldn't be ), then I highly recommend using it.

Know how to pick the right database, and understand extremely well how your app will need to talk to it. Throwing a NoSQL-based platform into the mix without being able to answer why it's a good fit is a good way to get your resume placed at the bottom of the paperweight stack. Is your app write-heavy? Read-heavy? These important details should dictate what kind of database you use, how to layer out your storage devices, logical volumes etc, what types of indexes you will need, and which types of systems to install to begin with. If you find yourself creating indexes on tables because your ORM decided to generate queries a particular way, you may be fighting a losing battle. You, the app developer, should be dictating how your app talks to the database and determining what indexes you will need throughout your application.

The Other Stuff

If you are truly a full stack developer, in addition to understanding storage, networking protocols, application layer protocols, system tuning ( which I didn't even touch on, but may rant about in the near future ), and all of the fun stuff that is required to conquer the front end and back end, you're probably also familiar with NTP, SMTP, and a flavor or two of monitoring systems, SNMP or otherwise. You're also likely comfortable using a network analyzer to debug communication issues and identify bottlenecks.


The idea is not to say that anyone who claims to be a full stack developer needs to have built entire SaaS platform from cabling wires to building out massive storage systems, but one should fully understand how an application talks over a network, and when and why it talks to its data store, in order to be able to construct scalable systems and brand oneself 'full stack'.

Stop Following

September 13, 2015

I don't often wax entrepreneurial, mostly for distaste for the word and partly because I see so many people offering their advice that it seems like the room is a bit too noisy for a conversation to be had. However, this particular topic I've not seen touched on, and seems to me to be important enough to write on.

In this day and age when everyone is an entrepreneur and startup founder ( or more gag-inducing a 'serial entrepreneur' ), it's odd to me that so many articles I see posted on LinkedIn are telling people how Steve Jobs ran his meetings, or how tech superstar John Smith runs his middle management team to achieve X, Y, and more importantly, Z. Seems like everywhere people are offering up or sharing what they think the blueprint is to being a succesful startup owner, and too often ( in my humble opinion.. but hey, this is my blog, so you're obviously going to be getting my slant on things ) it's by example.

There are two main types of perils here: The 'Follow Ideology X' problem, and the 'Do things like Y does them to achieve wild success' problem.

Firstly, the 'Do Things Like Y does' problem: Thinking that you can replicate what Steve Jobs did as a successful CEO may not work at all with your personality type, the chemistry of your team, the goals of your company or its culture. Chances are, whatever it is that Steve Jobs did to manage his meetings wasn't something he found in a book he read by Guy Kawasaki or any other piece of tech literature. He likely tried a few different approaches, figured out what bits worked the best, and left the ones that didn't behind. Isn't that what being an entrepreneur is about? If you don't experiment, and figure out what works best for you and your team, you may be heading down the wrong path, regardless of how well it worked for Him. So stop following, and go out and do what works best, and if you succeed at it, take pride in the fact that you did it your own way.

Secondly, let's think about the Follow Ideaology X problem: What I mean is, as with anything, following some concept or suggestion to an extreme is dangerous. I know a wildly successful mobile application company which has followed The Lean Startup to the point where it's now hurting them. Not that Eric Ries doesn't have a plethora of valid points and great ideas in his book, but there comes a time when a company needs to realize that it's not a startup anymore, and 'doing one thing really well' isn't going to work when you're in a crowded space and many of your competitors are doing that one thing really well too. Worse yet, they've found something that's supplemental to what they're doing that leads them into breaking new ground, or an adjacent space where they can execute on a successful revenue strategy. I'm of course not saying that you as an entrepeneur shouldn't read books written by individuals that have blazed the trails before you - what I *am* saying is these books should be reference materials to draw ideas from, or to help you form the types of experiments that will allow you to find your own success and make it yours.

Bumming a Quarter

(or: Because of Sheep)

August 15, 2015

I suppose it happens in more than a few peoples' lives but despite my humble origins, asking a passerby on the street for change wasn't a regular occurrence.

At the age of 17, with no money and having barely made it through high school, my dream of going to art college was obviously out of the question. And as if by chance, an Air Force recruiter had begun pestering me. I did have a secret desire of becoming a pilot for most of my youth, and this was good enough ammunition for the recruiter to convince me to take the Air Force placement test (ASVAB) which I did surprisingly well on. After that, all I needed to do was to pass the physical exam and get sworn in, and I was on my way. My father was somewhat pleased with this, having been in the Marines himself and we both agreed that it was a reasonable route for me, given the situation I was in and the potential career development that could come out of it.

This was the summer of 1992. Having worked several odd jobs ( starting at the age of 13 ) I was prepared to work and scrape to get by. I had alread had a roommate and lived on my own, but when times got really hard, my father reluctantly let me move back home for long enough to know that I needed to find another way out into the big brave world, so the recruiter's timing seemed fortuitous.

I suppose I was like a lot of other kids my age that didn't have the way or means into university, with one small exception. I had already been introduced to and had fallen madly in love with the Internet. Not what you know and experience today, but a pre-web, pre-e-commerce internet driven largely by curiosity, DARPA and some smatterings of grant money. A good friend ( and to this date one of the coolest people I've ever known ) introduced me to it by explaining it to me on a road trip we had undertaken to deliver him back to the university one winter. I was instantly mesmerized by the possibilities, and knew immediately that this would shape our collective future. I just had no idea what exactly that future looked like. Regardless, I proceeded to spend as much time as I could at his university ( Duke ) and conveniently enough, we looked a lot alike (we'd been mistaken to be brothers on more than one occasion) and this little fact would grant me access to all of the compute clusters at Duke where I became reacquainted with computers after a several year hiatus.

I had first been introduced to computers while in a special program at the elementary school I had attended in Hiawassee, Georgia where we were given access to a TRS-80 ( lovingly referred to by us old schoolers as the Trash Eighty ) where we had been taught the BASIC programming language and were more or less set free to see what we could do with ourselves. As a long time Dungeons and Dragons aficionado, text based adventure games fascinated me, so I wrote a few of those. When Jason sat me down in front of a DEC Ultrix machine at Duke, I realized that these computer things had evolved quite a long way. Consequently, it was a year or more later when I was shown a Windows 3.11 machine and it felt so uncomfortably primitive that I couldn't imagine that there would be any use for one at all, and was dumbfounded by the fact that they didn't incorporate the IP stack in their OS. Even the Mac seemed to be on its way and leaps and bounds ahead of the Windows platform with regards to networking in general.

So by the time I approached the medic at the recruiting location in Charlotte, North Carolina where I was then supposed to be sworn in and swept off to basic training, I already knew my way around a unix shell, some basic IP networking, some scripting, and a bit about the workings of the Internet. Yet I had absolutely no idea that this information would ever become valuable, and had been ridiculed a fair amount for being so caught up in such nonsense to begin with. So when the recruiter asked me that fate-laden question, 'Are you allergic to anything?' it hadn't dawned on me that despite all of the recruiter's recommendations about what to say in response to any of the physical examiner's questions ( "If he asks you if you've ever sneezed, you say 'no'" ), that my being allergic to wool would have any impact on my ability to join the Air Force and blow up people and things with expensive weaponry. Yet impactful it was, as his immediate response to me was 'Alright. You're out.' He made a quick gesture for the next kid in line to approach his podium, and I managed to throw in a question, which likely sounded something like, "I'm sorry.. What?" To which he replied - and on this I'm a bit more certain of - "All of our shit is wool." Now, being able to rationalize this in my head had no affect on the fact that in just a few moments, I would find myself on the streets of Charlotte looking for a quarter to use for the payphone to call my father to pick me up. My father was furious, as I had finally been close to some semblance of a future, and had blown it with complete ignorance.

But Dad was angry for a reason, and although it makes perfect sense today, being upset at your son for not giving any indication of trying to do and be better befuddled me at the time. However, it was a good motivator for me to get out on my own again and figure something out. Two years later, at the age of 19, I would start my first company with a group of really bright kids around the same age as me, and although we would ultimately fail, the experience set the stage for the rest of my life.

What Bums Me Out About the Tech Industry (today)

(or: Kids Stay Off My Lawn)

January 6, 2015

I consider myself somewhat of a technologist, and it is true that technology is a passion of mine. When I came to Silicon Valley in October of 1995 from South Carolina, it was more than I ever could have dreamed of. I had no idea what was going on out here, and that there was a whole valley full of people as passionate as I was about this giant network that I'd fallen in love with. I'm not sure if I had even heard of Silicon Valley - I just knew that I had a job waiting for me working at an Internet Service Provider that had been far more successful than mine had been ( though granted, I had been trying to convince Carolinians that the Internet was going to be more exciting than re-runs of MASH ). Here, the whole place was buzzing with talk about new ideas, startups, new programming languages, routing protocols, and people trying to get their heads around the Internet and what it meant for our future. People weren't drawn to it because they saw the next get-rich-quick scheme, or because it was trendy to call themselves entrepreneurs. Hell, people rarely even used that word back then, even though we were all starting companies, or working in startups. In those days, a startup wasn't a guy who paid some overseas software shop to crank out an MVP to run on a couple of cloud instances hoping to be the next WhatsApp. We were in it because the technology itself was fascinating and we wanted to be part of it. If you wanted a serious web presence, you paid for rack space or a cage in one of the colo providers in MAE-WEST or the PAIX where you would schlep big expensive (yet elegant, beautiful, and reliable) Sun gear to host your site or service on. You had to really want it, and fully believe in what you were building. You bought and configured big pieces of networking gear, load balancers and storage systems to scale your environment out - and it was expensive. To host anything of significance you needed to understand BGP ( the backbone protocol of the Internet ) and have an ASN, an IPv4 block, and know a fair amount about power, cooling, and systems. The concept of a 'web developer' was very new, and most of them even knew how to build out systems, and a good bit about networking and lower level programming. You didn't call yourself a programmer or any sort of engineer if you didn't know a fair amount about most or all of these things.

Sure, I stay current. I still spend a lot of time keeping up to date on what programming practices or design methodologies are out there. I create projects for myself in new languages so that I can understand their approaches to solving age old problems and new ones. I can't say that I relate with their developer communities though. I can't subscribe to the flawed philosophy that a developer shouldn't have to know how an application is talking to his database, or the fine details of what goes on in the underlying system or storage cluster. Those people are like ticking time bombs for some company to hire to build out their platform. They'll get your prototype out the door at light speed, but put any traffic on it and prepare yourself for a bill the size of a Pirates of the Caribbean movie.

But worse than the brogrammers, I think it's the 'entrepreneurs' that bug me the most. The word feels so tainted now. A friend showed me a profile recently of a guy who claims to have started more than 100 successful companies during the span of a 20 year career or so. My guess is that his criteria for having 'started' a company is forming a cohesive vision for a product for a few minutes before lunch. And of course this individual had about as much technology under his belt as Barney Fife. So not only did he not start and run these alleged successful companies, in the couple of months he would've supposedly been with each of them, he wouldn't even have been able to build the products either.

In contrast to those golden days, the tech industry today seems to lie at this horrible intersection of the mysteriously entitled generation Y, the millenials, and the extremely cheap and available resources for getting a product to market that the cloud and inexpensive overseas outsourcing shops have created. The latter in and of itself isn't necessarily a bad thing, but it's allowed for the scene to be polluted with the same get rich quick types that ran out and became real estate agents during the real estate bubble. Of course there's nothing wrong with someone figuring out how to make money to support their family, but there's no soul in doing something purely for cash either. Sure your cousin Joe really took a shine to real estate. Now tell him that tomatoes have gone to 20 dollars a piece and see what he does to his front yard. The tech industry was never a place for those people - we were tinkerers and researchers at heart that would have been doing the exact same thing were there not even a promise of a paycheck - just because we wanted to. We were the kids that took every one of our electronic toys apart to see how they worked. We made blue box dialers out of parts we bought at Radio Shack and not because we could save money on making phone calls from payphones - we did it because we could and because we were hungry for whatever piece of technology we could get our hands on. For us, the Internet was this vast sea of geek mystery and we wanted to explore it all and be part of its amazing expansion. We did it because there was an inexhaustible quantity of information to be learned about a subject that was dear to us. We used archie and gopher to transfer open source software around and share knowledge. We snuck into computer labs at neighboring universities to get our hands on computers that we otherwise would have no access to. And there was altruism within the Internet community.

Those golden days are long gone, and it doesn't feel like there will be another wave of innovation that will bring us back to those magical times when such an earth shattering revolution of technology will be solely in the hands of those that love it for what it is. But then again, had it not gone the direction it did, I wouldn't be able to sit here and wax nostalgic about having been a part of what I feel was like witnessing the very birth of rock and roll. ░

Why I Sleep On The Sofa

December 8, 2014

It’s important to know a little bit about my life for this to make any sense. My wife is a 5 foot tall, beautiful brazilian woman. We’ve been married for about 14 years, and have vastly differing senses of humor. Most of the time, and yes, often when inappropriate, i’m trying to be funny or make people laugh. this works on her almost never. When it does, she replies with her subtle portuguese accent ‘you are so stupid.’ She told me a few years into our marriage after i’d heard that from her at least a dozen times that she means more like ‘silly’ but then gave me that look she does when she’s trying to figure out if i've caught on to her sarcasm.

At any rate, it's important to have that bit of background to truly appreciate conversations like the one below. It's also important to know that she has an inordinate number of unique facial expressions, all of which can speak volumes at the right moment, and it also helps to paint the picture understanding that her default facial mode is an impossible combination of angry, and extremely cute.

And without further adieu, I give you one of the hundreds of reasons I wind up sleeping on the sofa:

[11/26/14, 9:18:35 AM] andrea: check your phone
[11/26/14, 9:24:19 AM] andrea: ??
[11/26/14, 9:25:12 AM] Wedge: what the hell is that?
[11/26/14, 9:25:13 AM] Wedge: a shoe?
[11/26/14, 9:28:10 AM] andrea: not only that... there is no right or left... it goes on both
[11/26/14, 9:28:25 AM] Wedge: for people that get confused easily?
[11/26/14, 9:29:03 AM] andrea: but they did it for people that want to match different ones... but you get it right?
[11/26/14, 9:29:26 AM] Wedge: i don't get it.
[11/26/14, 9:29:28 AM] Wedge: the shoes are hideous.
[11/26/14, 9:29:40 AM] Wedge: it's a frickin bedroom slipper.
[11/26/14, 9:30:45 AM] andrea: no!! they are amazing
[11/26/14, 9:30:56 AM] andrea: plastic/fashion
[11/26/14, 9:31:24 AM] Wedge: please don't wear those in public

The interesting prologue to this conversation is that Andrea had come up with this exact 'one shoe' idea almost a year prior. I, of course, had completely forgotten.

Sofa Redux

November 28, 2014

Another important thing to know about my wife is that she is extremely social. It's very hard to keep up with social engagements, partly because I tend to find out about them only hours (or minutes) in advance. On an average week, we have at most a couple of nights where we do not have dinner guests, or are not going out to dinner to meet up with someone.

What a lot of people don't know about me, is that by nature, I am an introvert. I socialize plenty (mostly for lack of an option) and do enjoy it for the most part, but I eventually need to recharge my batteries and spend some good quality time alone and in perfect silence, or listening to the calming rhythm of myself banging my head on solid objects. However, those blissful moments are very rare and are typically broken by conversations such as this one:

[11/14/14, 4:09:23 PM] andrea: don't forget dinner at 7:00
[11/14/14, 4:12:31 PM] Wedge: of course! how could i forget .. the .. dinner at .. 7pm.. with um....
[11/14/14, 4:12:50 PM] andrea: friends
[11/14/14, 4:12:50 PM] Wedge: the security guy!
[11/14/14, 4:12:53 PM] andrea: no
[11/14/14, 4:12:55 PM] Wedge: i mean friends!
[11/14/14, 4:13:02 PM] Wedge: i don't know why i said security guy. i meant friends.
[11/14/14, 4:13:08 PM] Wedge: because you know.. we planned .. that dinner with .. uh.. friends
[11/14/14, 4:13:09 PM] Wedge: at 7
[11/14/14, 4:13:16 PM] andrea: jesse our partner in the south
[11/14/14, 4:13:24 PM] Wedge: totally
[11/14/14, 4:13:27 PM] Wedge: good ol' jesse.
[11/14/14, 4:13:36 PM] Wedge: really looking forward to catching up with him or her.