If you’ve been working with PHP in a professional capacity on Today’s Internet, you’ve probably run across one or two instances where you’ve had to use cURL. And, if you didn’t use cURL, at least seen where it could be useful. For instance, if allow_url_fopen is turned off. In fact, most of us have reached the point where cURL is regarded as little more than another mundane step in whatever project you’re working on. Or maybe you’ve abstracted it to a new class, as we just did at MECLabs, and you don’t even look at the calls to the client URL library anymore.

Enter the nightmare project.  I was handed a set of specs for one of our clients that uses SOAP for communication with their Web Service.  The manual was 150 pages long in Word; 150 pages, that is, for just one method.  There were the usual starting pages, introduction, credits, contact, credentials, table of contents, etc.  Then there was the 60-or-so page XML Request to be sent.  Immediately following it was the 75+ page XML Response.  I sank into my chair as I opened up the next document to review the second method in their API; even longer.  To top it off, they use NTLM Authentication.  Did I mention all of this is sent over SSL on a custom port?  It’s going to be a long partnership.

Of course, that was only the beginning.  In order to facilitate this process I wrote the class that abstracted the calls to the cURL library into a much easier to use format, tested it with various other APIs until I was satisfied it would work, and then whipped up a simple GET request to grab the main page of the WSDL, which is where I ran into the first, albeit simplest, problem.  I won’t go over this one in detail since it’s not the bigger issue, or really specific to cURL, but just to NTLM Authentication: make sure to send keep-alive headers, and not “Connection: close” as many APIs request you do — this will prevent the NTLM Authentication from completing.

Satisfied that I could now communicate with their secured, authenticated server, I fired up the first API call (sitting quietly in an external file, as it was about ½KLOC, and I didn’t want it cluttering up test.php), only to watch it crash and burn.  Without getting into too much detail of the solutions I tried, and they were legion, let me just describe the issue:

I could successfully perform HEAD and GET requests, and even POST with a zero-length body would get something back (notably, “400 Bad Request” headers), but if I tried to send even 1 byte through POST, the system would fail.  Since the requests that didn’t fail were authorizing successfully, I gave up on it being a handshake issue, which I shouldn’t have done.

As a “Good Web Citizen,” I try, as I hope most developers do, to send the most correct, most standards compliant content that I can, even when making simple HTTP Requests.  That was my downfall.  If you’re not familiar with NTLM Authorization, It’s a multistep authorization process that requires multiple exchanges of headers.  Since cURL will dutifully send any custom headers you specify with every request it makes, you can end up with additional unnecessary, or downright incorrect, information.  In my case, this is how it went:

1. cURL opens the connection by trying to POST the data to the web service
2. The web service sends “401 Unauthorized” headers and identifies itself as an NTLM Authenticating server.
3. cURL sends the first part of the NTLM handshake as well as the “Content-Length” header from the POST request.
4. The web service waits, for about 930 seconds if I don’t force a timeout, for the data to be sent
5. The connection gets reset.

Sadly, the solution was, although simple, to stop being what I consider to be a “Good Web Citizen” and stop sending Content-Length headers to cURL.  While cURL is generally smart enough to send these headers for me when they need to be sent, I’d prefer not to rely on it exclusively.  Unfortunately, cURL is not smart enough to not send these headers for me when it shouldn’t, so I’m painted into a corner.  And, well, at the end of the day, it’s really just not that important; I won’t be losing any sleep over it.

Hopefully, this saves you, the PHP developer using cURL, the pain of dealing with someone who insists that you connect to their SSL-enabled NTLM Authenticating web service and POST data, and the horror of resorting to writing a class using pfsockopen and manually mimicking the NTLM handshake like this guy, when cURL really will, I promise, do this just fine.

July 25th, 2009 by Dereleased | Comments Off on POST via cURL under NTLM Auth: Learn From My Pain

One of the great gems of the information age is the ability of just about anyone, anywhere, to post anything, anytime, about whatever they want.  Of course, this comes with the drawback of just about anyone, anywhere, being able to post anything, anytime, about whatever they want.  Of course, if you’ve embraced the information age as many of us have, you consider those opinions to be vital; since you are, in fact, reading this, I’m going to assume that you are one of us.

If you’re one of us, you probably have a tendency to do a lot of online window-shopping.  That is, irrespective of where you actually buy a product — Apple Store, Ebay, Amazon, Wal*Mart, The Mall — you probably researched it online first.  And researched it some more.  And then, just to be safe, a little more.  You probably read review after review, painstakingly looked over specs, looked at comparable products with different prices, and so on.  Or, you typed name of of the product into google, clicked on the first link, and if it didn’t confirm your preconceived notion, deeply considered clicking on a second.

It is the nature of our information age to be the information saturation age — hence, the subtitle of this blog — and the problem with that arises when the wealth, cornucopia, veritable bevvy, or “Crap-ton” if you prefer, of information just overloads the senses to the point where your eyes glaze over and you don’t really care if $225.50 is $3.23 to much because someone’s brother’s friend got a deal on a slightly dented and non-functional unit.

Alternatively, you might take every word you read just a little bit too seriously.  While there is certainly no problem with reading everything everyone has to say about a particular item, barring time constraints of course, it seems that there is very rarely any continuity or overall opinion that can be gleaned about anything in particular.  Since not being able to establish any coherent ideas can be inherently discouraging and damaging to the process of seeking information about something you intend to buy online, I’d like to share a few tips and tricks I’ve learned about how to parse the plethora of information out there in review form.

• Ask yourself, “If someone is satisfied with this product, how likely are they to rate it?” Going back to my earlier topic of anyone posting anything, the internet has a tendency to be a last-ditch medium for someone who feels sleighted to “confront” their agressor.  Alternatively, someone who has not been mistreated, in reality or imagination, likely has no reason to speak up; it’s not that they wouldn’t, or couldn’t, it’s just that it simply doesn’t occur to them, or worse, they decline to comment either because doing so seems like too much of a hassle, or they don’t want to be attacked with all the venom and vitriol that all the negative posters have spewed thus far.Take, for example, the apartment ratings website. Most people who are happy with where they are living simple aren’t going to go check out www.hey-you-rate-these-here-apartments.com because, and this is key to remember, what is the point for them?  At the same time, someone who has been evicted, turned down, or had anything go wrong, has every reason to post that they are the most vile baby-eating demon-summoning jerks on the face of the earth.  It’s not that they are trying to misuse the system, it’s just that they’re the only ones using the system.  So, instead of gathering a true smattering of opinions from all sorts of residents, they instead gather the foulest words uttered by most displeased of all: those who took it to the internet.

• If they aren’t professional critics, forget the numbers, read the words. If you look at sites like newegg.com, amazon.com, or most any site that allows users to openly comment — and rate — whatever products they’re looking at, forget about the numbers and read the words.  Sure, the numerical ratings are very nice, and oftentimes can be very useful at getting an at-a-glance idea of what you’re buying, they can also be misleading — to a point.  There are four types of reviewers that I’ve noticed in the civilian reviewing world:
  1. The all-stormy reviewer – One of the more annoying to read the posts from is this user, who generally rates everything in about the 10-20^th percentile of the available ratings; No matter what happened, or whose fault it is, this product sucks.  Typical posts might include how the package arrived late due to a shipping error with UPS, or how the color of the part wasn’t as vibrant as they had hoped and dreamed for.  They are usually easy to detect, coming off as unrealistic with a side of self-entitlement the size of the international space station.
  2. The so-so reviewer – While not inherently dangerous, they can have the effect of lessening your excitement over, and perhaps your willingness to buy, a genuinely exciting product.  These users typically seem to believe that no matter how good something was, something could have been done better.  And sure, while it’s important to not immediately jump up and down and scream “10!  10!  10!” every time you get something you don’t hate, it’s equally important to realize that if you’re thrilled to death with you’re new iPod, but cannot believe that you would have to pay 99 cents for a certain app, that doesn’t make it a “6/10.”
  3. The all-sunny reviewer – One of the more dangerous types of reviewers, the person who is not nearly bitter enough about what went wrong.  In general, these “5-star sheriffs” are less common than most of the other types of posters, opting to post on their perceptions of what was going to happen, not their perceptions (or, ideally, the reality) of what actually happened.  As posters they are generally easy to spot: “I was at ‘Nails So Happy’ getting a manicure and my nails are like so pretty now!  lol!  Only one of them cracked and the weave I had to put in after one of the stress-release candles set my hair on fire (lol!  hair burns fast and smells bad!) actually doesn’t look too bad!  ROFL!”  Note the copious use of laughter where it makes no sense for laughter to occur.  Note the insane inability to not be angry over downright infuriating behavior.  Remember, words-not-numbers.
  4. Actual moderate reviewers – One of the more rare species of poster on the internet, all reviewers are elsewise until proven sane.  Remember, it could be anybody writing those words out there.
• Ask yourself, “How likely is it that this will happen to me?” Sometimes, the problem someone has with a particular product or service doesn’t apply to you.  For example, if someone posts that “Jim’s Auto Parts” sold him bad brake pads because he couldn’t remember if he needed drum or disc breaks, surely you, as someone smart enough to be reading this, will not succumb to similar error.  Conversely, if you are reading about the latest video game and people keep saying how hard it is to install, or debug, or how much it crashes, or how obscure it is to fix it, and you know you aren’t a very technical person, maybe just sit this one out, or get a professional before you paint yourself into a corner.
• Don’t put all your faith in professional critics. Remember, professional critics, due to their cheerful, fun-loving and cuddly nature, are more likely to not only take disdain in something you may genuinely enjoy, but to take joy in the “art” of something (like a book or movie) that you won’t care the slightest bit about.  People who love cars may weigh certain features unimportant to the average driver above things you will use and enjoy every day, and people who get paid to write about cars for those people are likewise going to have standards that, at times, will seem rather inane to you.  It is important to remember who the critic’s audience is, where you stand in relation to that audience, and more importantly, what the critic’s outlook is.
• Finally, Filter what you see down to the level that it actually affects you. To retouch on the “apartment ratings” scenario, while you might not be able to trust the numerical ratings at all, and most of the comments about this particular neighbor or that particular person in the leasing office are probably bogus and irrelevant, you should keep an eye out for patterns — if everyone complains that maintenance takes too long, or that the roof is leaky, or that the pest control was lacking, this is probably cause for concern and should be weighed appropriately.  By actively attempting to filter the information you get to the point where it actually affects you, you are successfully using the information around you.

At the end of the day, the biggest trick is to just use common sense.  While some or most of these probably don’t seem all that complicated or mystical, it is the unbiased approach of applying them every time that must be employed.  When you want to buy something, stop thinking of it as “the gadget that you absolutely have to have” and start looking at it like something that, while you personally couldn’t care less about, your friend is just dying to own.  Don’t you want your friend to get the best deal?  It sure would help you –I mean, your friend– out a lot to save themselves the money and heartache on a dude of a product or a bum-deal on a service plan, don’t you think?

July 24th, 2009 by Dereleased | 1 Comment »

A year ago this wouldn’t have even been an issue to me.  In fact, a year ago, I’d probably have started with a table.  Of course, like everyone who intends to stay in this business without becoming a dinosaur of a maintenance programmer, something I just can’t afford to do at my age, I’ve moved up with the times.  And, it seems, I’ve moved out, too.  Smaller HTML, but bigger attached stylesheets; longer development times, but (usually) more pleasing form-factors, and, let’s face it, some of the things you can do with CSS are just downright fun.

Unfortunately, there’s a very ugly side to this debate.  If you’ve actively been on either side of it, then you might hold the opinion that there is no pretty side to the debate, but in a relative sense, there is.  The pretty side is about vertical alignment issues, low resolution displays, and the trauma of floats gone wrong.  The ugly side is, without question, the unholiest member of the family that is CSS-supporting (however loosely the term need be applied) browsers: IE 6.

When I got back a QA document showing how my beautiful layout had become a mangled mess in IE 6, I was stunned.  I’m used to typical IE 6 problems, such as PNG transparency, weird box model rendering, the occasional extra broken line, or borders rendering in the wrong place.  What I was simply not prepared for, though, was the carnage that sat before me.  Entire paragraphs were somehow invisible until highlighted, even though they were a contrasting color.  Form elements drooped off of the carefully sculpted round box.  Text was interrupted in the name of a submit button.  And to top it all off, the PNGs weren’t even transparent.

I immediately did what any sane developer would do: I called over the guy who gave me the specs and said, “Who cares if it doesn’t work in IE 6?”

“The client does,” came the reply; “They still have 20% of the browser market share.”

20% is a pretty big number, considering we support chrome at 1/10th of that usage.  Even if the data from w3schools is accurate, even if IE 6 is down to a paltry 14.9%, they’re still not falling out of the realm of active use.  When you look at the numbers, it’s not really much of a mystery why: According to Appendix Table A (xls) from the U.S. Census Bureau report Computer and Internet Use in the United States: October 2007, in-home saturation of computers with internet access is only about 11% more now than there were in 2001, when IE 6 was released.  With the amount of people still using ancient beige-box monstrosities, it’s little wonder how that browser continues to tread water 8 years later.  Unfortunately, it’s not the IE 6 users who pay the price for their lack of knowledge.

And this leaves us again in the same old place: The War.  All of the arguments over whether tables are bad design or CSS is too complicated are practically rendered moot when you consider that at least 1-in-7 people are using an antiquated piece of technology that will choke, cough, sputter, and finally leave users entirely displeased and demotivated.  And since there’s no real stopping it, short of using extreme guerilla tactics, let me give my final opinion on the matter: Just use whichever one works.  Tables and Divs can be equally messy, so it’s more important that you just keep your code clean, rather than trying to favor or scorn a particular tag.  And, if you’re one of those designers who has become so entrenched in the anti-table battlefield that you refused to use tables to display tabular data, I have 5 words for you: Give Up And Use Tables.

July 23rd, 2009 by Dereleased | 4 Comments »