summaryrefslogtreecommitdiff
path: root/indra/llcorehttp/_httplibcurl.cpp
AgeCommit message (Collapse)Author
2017-09-30MAINT-7081: Eliminate unused variable errors after new refactoring.Nat Goodspeed
The new helper functions check_curl_easy_setopt() and check_curl_multi_setopt() encapsulate the pervasive idiom: code = curl_{easy,multi}_setopt(handle, option, arg); check_curl_{easy,multi}_code(code, option); But since each of these helper functions contains its own local CURL{,M}code variable 'code', having a caller-scope variable reused for every such call is no longer necessary -- in fact is no longer used at all. That produces a fatal warning with MSVC. Get rid of those now-unused variables.
2017-09-29MAINT-7081: Try requesting HTTP/2 when a request wants pipelining.Nat Goodspeed
2016-04-04merge with 4.0.3-releaseOz Linden
2016-02-19MAINT-6137: Re enable pipelining by default, use new version of CURL (7.47) ↵Rider Linden
with corrections for timed out connections in pipelining. Minor fix for safer op retrieval.
2016-01-21MAINT-5915: Put in some extra null checks around completed HTTP checks to ↵Rider Linden
try and eliminate a crash in the wild.
2015-11-10remove execute permission from many files that should not have itOz Linden
2015-10-16MAINT-5271: Converted internal pointers to internal operation to managed ↵Rider Linden
shared pointers. Removed direct cast and dereference of handles.
2015-07-08Convert LLCore::HttpHeaders to use shared_ptr<> rather than an ↵Rider Linden
intrusive_ptr<> for refrence counting.
2014-09-19Introduce libcurl handle cache. Create a private cacheMonty Brandenberg
of used handles and a fast handle factory that's thread- correct.
2014-09-04Pipelining work. Extend transfer timeout by the pipeline depthMonty Brandenberg
as transfers can appear delayed with deep pipelining and more requests in the pool. Added bad HTTP status error (typically getting a 0 back as HTTP status from libcurl) to the list of retryable errors. There's a response stream problem with libcurl and pipelining that induces this problem. Retrying helps but may not be entirely safe. Watch bug 1420 on the libcurl sourceforge bug tracker. Extend options of test/example program to include un-ranged requests. Document the excessive data transfer induced when ranged requests are disabled. This is an abnormal mode for very rare users so we'll just eat that for now.
2014-08-12Better support for dynamic option changes in llcorehttp. Libcurl hasMonty Brandenberg
some problems disabling pipelining on a multi handle with outstanding requests so build a more conservative system that allows requests to drain before setting curl multi options. Would rather not have this but it is significantly safer. "HttpPipelining" debug setting is now fully dynamic. Connection limits can also be made dynamic in the near future. Upped the default connection count back to 8 for now but will revisit this in the tuning phase. It might be time to combine mesh and textures into a single asset class. For normal server operations that would be a clear path, but for server under load, the current scheme may be better. Minor cleanup in logging to elminate some redundant strings. Might add some more tracing to the stall logic 'just in case'.
2014-06-23First HTTP pipelining viewer. Enable pipelining forMonty Brandenberg
GetTexture and GetMesh2 at a pipeline depth of 5. Create global debug option, HttpPipelining, to enable and disable HTTP pipelining (defaults to true). Tweak texture and mesh low- and high-water request levels based on pipelining status and depth. Fixup texture console which was damaged in a recent release. Split logging of the no-request HTTP error case into two cases: one for missing URL in HTTP request, one for HTTP request not created. A refactor in llcorehttp is coming: I will be moving all libcurl- using code into libcurl-specific modules.
2014-02-25merge viewer-release to sunshine-externalBrad Payne (Vir Linden)
2013-09-11SH-4490 More 'humane' error code presentation from llcorehttp callersMonty Brandenberg
Added toTerseString() conversion on HttpStatus to generate a string that's more descriptive than the hex value of the HttpStatus value but still forms a short, searchable token (e.g. "Http_503" or "Core_7"). Using this throughout the viewer now, no live cases of toHex(), I believe.
2013-08-16SH-4407 Tuning to get new code working as well.Monty Brandenberg
Do some runtime code avoidance and skip unnecessary libcurl and syscall invocations.
2013-07-29SH-4371 Reduce 22mS inter-conenction latency on HTTP operations.Monty Brandenberg
Simple change dropped this value by 7-10mS or so. Any time we complete an operation on a transport pass, arrange to skip the run-loop sleep so that we fill a possible empty slot as quickly as possible. With pipelining, this kind of thing should become unnecessary but for now, we'll do this to increase throughput.
2013-05-21mergeBrad Payne (Vir Linden)
2013-04-15SH-4106 Significantly upgrade the HttpHeaders interface for SSB.Monty Brandenberg
Header container moves from a vector of raw lines to a vector of string pairs representing name/value pairs in headers. For incoming headers, we normalize the name to lowercase and trim it. Values are only left-trimmed. Outgoing headers are left as-is. Simple find() method for the common case, forward and reverse iterators for those few who need to do it themselves. The HTTP status line (e.g. 'HTTP/1.1 200 Ok') is no longer treated as a header to be returned to caller. Unit tests, as usual, were a bear but they absolutely ensured outgoing HTTP header conformance after the change. Grunt work paid off. LLTextureFetch was also given a second options structure for texture fetches. Same as the original but with header return to caller requested. Baked textures should use this, the other 20,000 texture fetch requests should continue to use the original.
2013-04-15SH-4106 Significantly upgrade the HttpHeaders interface for SSB.Monty Brandenberg
Header container moves from a vector of raw lines to a vector of string pairs representing name/value pairs in headers. For incoming headers, we normalize the name to lowercase and trim it. Values are only left-trimmed. Outgoing headers are left as-is. Simple find() method for the common case, forward and reverse iterators for those few who need to do it themselves. The HTTP status line (e.g. 'HTTP/1.1 200 Ok') is no longer treated as a header to be returned to caller. Unit tests, as usual, were a bear but they absolutely ensured outgoing HTTP header conformance after the change. Grunt work paid off. LLTextureFetch was also given a second options structure for texture fetches. Same as the original but with header return to caller requested. Baked textures should use this, the other 20,000 texture fetch requests should continue to use the original.
2013-03-29Update Mac and Windows breakpad builds to latestGraham Madarasz
2013-03-13Large changes to the LLCurl::Responder API, as well as pulling in some ↵Don Kjer
changes to common libraries from the server codebase: * Additional error checking in http handlers. * Uniform log spam for http errors. * Switch to using constants for http heads and status codes. * Fixed bugs in incorrectly checking if parsing LLSD xml resulted in an error. * Reduced spam regarding LLSD parsing errors in the default completedRaw http handler. It should not longer be necessary to short-circuit completedRaw to avoid spam. * Ported over a few bug fixes from the server code. * Switch mode http status codes to use S32 instead of U32. * Ported LLSD::asStringRef from server code; avoids copying strings all over the place. * Ported server change to LLSD::asBinary; this always returns a reference now instead of copying the entire binary blob. * Ported server pretty notation format (and pretty binary format) to llsd serialization. * The new LLCurl::Responder API no longer has two error handlers to choose from. Overriding the following methods have been deprecated: ** error - use httpFailure ** errorWithContent - use httpFailure ** result - use httpSuccess ** completed - use httpCompleted ** completedHeader - no longer necessary; call getResponseHeaders() from a completion method to obtain these headers. * In order to 'catch' a completed http request, override one of these methods: ** httpSuccess - Called for any 2xx status code. ** httpFailure - Called for any non-2xx status code. ** httpComplete - Called for all status codes. Default implementation is to call either httpSuccess or httpFailure. * It is recommended to keep these methods protected/private in order to avoid triggering of these methods without using a 'push' method (see below). * Uniform error handling should followed whenever possible by calling a variant of this during httpFailure: ** llwarns << dumpResponse() << llendl; * Be sure to include LOG_CLASS(your_class_name) in your class in order for the log entry to give more context. * In order to 'push' a result into the responder, you should no longer call error, errorWithContent, result, or completed. * Nor should you directly call httpSuccess/Failure/Completed (unless passing a message up to a parent class). * Instead, you can set the internal content of a responder and trigger a corresponding method using the following methods: ** successResult - Sets results and calls httpSuccess ** failureResult - Sets results and calls httpFailure ** completedResult - Sets results and calls httpCompleted * To obtain information about a the response from a reponder method, use the following getters: ** getStatus - HTTP status code ** getReason - Reason string ** getContent - Content (Parsed body LLSD) ** getResponseHeaders - Response Headers (LLSD map) ** getHTTPMethod - HTTP method of the request ** getURL - URL of the request * It is still possible to override completeRaw if you want to manipulate data directly out of LLPumpIO. * See indra/llmessage/llcurl.h for more information.
2012-08-01SH-3308 Beef up retry messaging.Monty Brandenberg
Reformatted messages around request retry. Successfully retried requests also message so you can see the cycle closed. Added additional retryable error codes (timeout, other libcurl failures). Commenting and removed some unnecessary std::min logic.
2012-07-23Big comment and naming cleanup. Ready for prime-time.Monty Brandenberg
Add to-do list to _httpinternal.h to guide anyone who wants to pitch in and help.
2012-07-16SH-3189 Remove/improve naive data structuresMonty Brandenberg
When releasing HTTP waiters, avoid unnecessary sort activity. For Content-Type in responses, let libcurl do the work and removed my parsing of headers. Drop Content-Encoding as libcurl will deal with that. If anyone is interested, they can parse.
2012-07-03More integration work for texture fetch timeouts.Monty Brandenberg
The fetch state machine received a new timeout during the WAIT_HTTP_REQ state. For the integration, rather than jump the state to done, we issue a request cancel and let the notification plumbing do the rest without any race conditions or special-case logic.
2012-06-26SH-3184/SH-3221 More work on cleanup with better unit test work and more ↵Monty Brandenberg
aggressive shutdown of a thread. Some additional work let me enable a memory check for the clean shutdown case and generally do a better job on other interfaces. Request queue waiters now awake on shutdown and don't sleep once the queue is turned off. Much better semantically for how this will be used.
2012-06-23SH-3184/SH-3221 Improve cleanup, destructor, thread termination, etc. logic ↵Monty Brandenberg
in library. With this commit, the cleanup paths should be production quality. Unit tests have been expanded to include cases requiring thread termination and cleanup by the worker thread. Special operation/request added to support the unit tests. Thread interface expanded to include a very aggressive cancel() method that does not do cleanup but prevents the thread from accessing objects that will be destroyed.
2012-06-22This sets down the groundwork for dynamic policy classes.Monty Brandenberg
Groundwork is used for the default class which currently represents texture fetching. Class options implemented from API user into HttpLibcurl. Policy layer is going to start doing some traffic shaping like work to solve problems with consumer-grade gear. Need to have dynamic aspects to policies and that starts now...
2012-06-19Cleanup: move magic nubmers to new _httpinternal.h header file.Monty Brandenberg
2012-06-14LLMutex recursive lock, global & per-request tracing, simple GET request, ↵Monty Brandenberg
LLProxy support, HttpOptions starting to work, HTTP resource waiting fixed. Non-LLThread-based threads need to do some registration or LLMutex locks taken out in these threads will not work as expected (SH-3154). We'll get a better solution later, this fixes some things for now. Tracing of operations now supported. Global and per-request (via HttpOptions) tracing levels of [0..3]. The 2 and 3 levels use libcurl's VERBOSE mode combined with CURLOPT_DEBUGFUNCTION to stream high levels of detail into the log. *Very* laggy but useful. Simple GET request supported (no Range: header). Really just a degenrate case of a ranged get but supplied an API anyway. Global option to use the LLProxy interface to setup CURL handles for either socks5 or http proxy usage. This isn't really the most encapsulated way to do this but a better solution will have to come later. The wantHeaders and tracing options are now supported in HttpOptions giving per-request controls. Big refactoring of the HTTP resource waiter in lltexturefetch. What I was doing before wasn't correct. Instead, I'm implementing the resource wait after the Semaphore model (though not using system semaphores). So instead of having a sequence like: SEND_HTTP_REQ -> WAIT_HTTP_RESOURCE -> SEND_HTTP_REQ, we now do WAIT_HTTP_RESOURCE -> WAIT_HTTP_RESOURCE2 (actual wait) -> SEND_HTTP_REQ. Works well but the prioritized filling of the corehttp library needs some performance work later.
2012-06-11Refactoring of the request completion thread and removal of ↵Monty Brandenberg
206/content-range hack in xport. Retry/response handling is decided in policy so moved that there. Removed special case 206-without-content-range response in transport. Have this sitation recognizable in the API and let callers deal with it as needed.
2012-06-11Unit test fixups. Mac/Linux detect memory leak due to llwarns, try/catch ↵Monty Brandenberg
cleanup. Our logging holds on to a changing bit of memory between operations and the memory leak detection I'm using senses this and complains. So, for now, disable the final memory check on Mac & Linux, leave it active on Windows. Solve this for real some other day. Add try/catch blocks to do cleanup in unit tests that go wrong so that we don't get a cascade of assertion failures when subsequent tests find singletons still alive.
2012-06-08Implemented HTTP retry for requests. Went in rather easily whichMonty Brandenberg
surprised me. Added a retry queue similar to ready queue to the policy object which is sorted by retry time. Currently do five retries (after the initial try) delayed by .25, .5, 1, 2 and 5 seconds. Removed the retry logic from the lltexturefetch module. Upped the waiting time in the unit test for the retries. People won't like this but tough, need tests.
2012-06-06Policy + caching fixes + https support + POST workingMonty Brandenberg
Implemented first global policy definitions to support SSL CA certificate configuration to support https: operations. Fixed HTTP 206 status handling to match what is currently being done by grid services and to lay a foundation for fixes that will be a response to ER-1824. More libcurl CURLOPT options set on easy handles to do peer verification in the traditional way. HTTP POST working and now reporting asset metrics back to grid for the viewer's asset system. This uses LLSD so that is also showing as compatible with the new library.
2012-06-01Major steps towards implementing the policy component.Monty Brandenberg
Identified and reacted to the priority inversion problem we have in texturefetch. Includes the introduction of a priority_queue for the requests that are ready. Start some parameterization in anticipation of having policy_class everywhere. Removed _assert.h which isn't really needed in indra codebase. Implemented async setPriority request (which I hope I can get rid of eventually along with all priorities in this library). Converted to using unsigned int for priority rather than float. Implemented POST and did groundwork for PUT.
2012-05-23Integrate llcorehttp library into lltexturefetch design.Monty Brandenberg
This is the first functional viewer pass with the HTTP work of the texture fetch code performed by the llcorehttp library. Not exactly a 'drop-in' replacement but a work-alike with some changes (e.g. handler notification in consumer thread versus responder notification in worker thread). This also includes some temporary changes in the priority scheme to prevent the kind of priority inversion found in VWR-28996. Scheme used here does provide liveness if not optimal responsiveness or order-of-operation. The llcorehttp library at this point is far from optimally performing. Its worker thread is making relatively poor use of cycles it gets and it doesn't idle or sleep intelligently yet. This early integration step helps shake out the interfaces, implementation niceties will be covered soon.
2012-04-23Okay, imported the core-http library and got it compiling suspiciously easily.Monty Brandenberg
The unit/integration tests don't work yet as I'm still battling cmake/autobuild as usual but first milestone passed.