If you’ve been working with PHP in a professional capacity on Today’s Internet, you’ve probably run across one or two instances where you’ve had to use cURL. And, if you didn’t use cURL, at least seen where it could be useful. For instance, if allow_url_fopen is turned off. In fact, most of us have reached the point where cURL is regarded as little more than another mundane step in whatever project you’re working on. Or maybe you’ve abstracted it to a new class, as we just did at MECLabs, and you don’t even look at the calls to the client URL library anymore.

Enter the nightmare project.  I was handed a set of specs for one of our clients that uses SOAP for communication with their Web Service.  The manual was 150 pages long in Word; 150 pages, that is, for just one method.  There were the usual starting pages, introduction, credits, contact, credentials, table of contents, etc.  Then there was the 60-or-so page XML Request to be sent.  Immediately following it was the 75+ page XML Response.  I sank into my chair as I opened up the next document to review the second method in their API; even longer.  To top it off, they use NTLM Authentication.  Did I mention all of this is sent over SSL on a custom port?  It’s going to be a long partnership.

Of course, that was only the beginning.  In order to facilitate this process I wrote the class that abstracted the calls to the cURL library into a much easier to use format, tested it with various other APIs until I was satisfied it would work, and then whipped up a simple GET request to grab the main page of the WSDL, which is where I ran into the first, albeit simplest, problem.  I won’t go over this one in detail since it’s not the bigger issue, or really specific to cURL, but just to NTLM Authentication: make sure to send keep-alive headers, and not “Connection: close” as many APIs request you do — this will prevent the NTLM Authentication from completing.

Satisfied that I could now communicate with their secured, authenticated server, I fired up the first API call (sitting quietly in an external file, as it was about ½KLOC, and I didn’t want it cluttering up test.php), only to watch it crash and burn.  Without getting into too much detail of the solutions I tried, and they were legion, let me just describe the issue:

I could successfully perform HEAD and GET requests, and even POST with a zero-length body would get something back (notably, “400 Bad Request” headers), but if I tried to send even 1 byte through POST, the system would fail.  Since the requests that didn’t fail were authorizing successfully, I gave up on it being a handshake issue, which I shouldn’t have done.

As a “Good Web Citizen,” I try, as I hope most developers do, to send the most correct, most standards compliant content that I can, even when making simple HTTP Requests.  That was my downfall.  If you’re not familiar with NTLM Authorization, It’s a multistep authorization process that requires multiple exchanges of headers.  Since cURL will dutifully send any custom headers you specify with every request it makes, you can end up with additional unnecessary, or downright incorrect, information.  In my case, this is how it went:

1. cURL opens the connection by trying to POST the data to the web service
2. The web service sends “401 Unauthorized” headers and identifies itself as an NTLM Authenticating server.
3. cURL sends the first part of the NTLM handshake as well as the “Content-Length” header from the POST request.
4. The web service waits, for about 930 seconds if I don’t force a timeout, for the data to be sent
5. The connection gets reset.

Sadly, the solution was, although simple, to stop being what I consider to be a “Good Web Citizen” and stop sending Content-Length headers to cURL.  While cURL is generally smart enough to send these headers for me when they need to be sent, I’d prefer not to rely on it exclusively.  Unfortunately, cURL is not smart enough to not send these headers for me when it shouldn’t, so I’m painted into a corner.  And, well, at the end of the day, it’s really just not that important; I won’t be losing any sleep over it.

Hopefully, this saves you, the PHP developer using cURL, the pain of dealing with someone who insists that you connect to their SSL-enabled NTLM Authenticating web service and POST data, and the horror of resorting to writing a class using pfsockopen and manually mimicking the NTLM handshake like this guy, when cURL really will, I promise, do this just fine.

Recent Entries

• Apache Taught You The Wrong Way To Think About Web Applications
• Adventures in Parsing: PHP’s implicit semicolon (‘;’) before every close tag
• The importance of ZVals and Circular References
• PHP Quirks – String manipulation by offset
• Let’s talk about your password model
• Pour Some Syntactic Sugar On Me: ‘Unless’ Keyword
• Arrays of Objects and __get: Friends Forever
• Did You Know? Class Visibility in PHP
• On Net Neutrality – A Plea
• Blankets – Nature’s Simple Truths

This entry was posted on Saturday, July 25th, 2009 at 14:15 and is filed under PHP Development, Web Development. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.