summaryrefslogtreecommitdiff
path: root/indra/llcorehttp/_httppolicy.cpp
AgeCommit message (Collapse)Author
2022-04-15SL-17219 WIP - Texture pipeline overhaulDave Parks
2017-10-11Automated merge with ssh://bitbucket.org/lindenlab/viewer-releaseNat Goodspeed
2017-08-08MAINT-7634: Logging and instrumentation canges to narrow down viewer crashes.Rider Linden
2017-07-28MAINT-7634: Move StatsAccumulator into llcommon, collect data sent and error ↵Rider Linden
codes from core.
2017-07-24MAINT-7495 Viewer retries too many time apon 504 from login.cgiandreykproductengine
2016-04-04merge with 4.0.3-releaseOz Linden
2016-02-19MAINT-6137: Re enable pipelining by default, use new version of CURL (7.47) ↵Rider Linden
with corrections for timed out connections in pipelining. Minor fix for safer op retrieval.
2015-11-10remove execute permission from many files that should not have itOz Linden
2015-10-16MAINT-5271: Converted internal pointers to internal operation to managed ↵Rider Linden
shared pointers. Removed direct cast and dereference of handles.
2014-10-10MAINT-4564 HTTP Pipelining is not happening in Drano HTTP Phase 4Monty Brandenberg
Incorporate the new libcurl 7.38.0 build with curl bug 1420 workaround. Add developer-centric testing code to evaluate the workaround or a future fix for 1420.
2014-08-12Better support for dynamic option changes in llcorehttp. Libcurl hasMonty Brandenberg
some problems disabling pipelining on a multi handle with outstanding requests so build a more conservative system that allows requests to drain before setting curl multi options. Would rather not have this but it is significantly safer. "HttpPipelining" debug setting is now fully dynamic. Connection limits can also be made dynamic in the near future. Upped the default connection count back to 8 for now but will revisit this in the tuning phase. It might be time to combine mesh and textures into a single asset class. For normal server operations that would be a clear path, but for server under load, the current scheme may be better. Minor cleanup in logging to elminate some redundant strings. Might add some more tracing to the stall logic 'just in case'.
2014-06-23First HTTP pipelining viewer. Enable pipelining forMonty Brandenberg
GetTexture and GetMesh2 at a pipeline depth of 5. Create global debug option, HttpPipelining, to enable and disable HTTP pipelining (defaults to true). Tweak texture and mesh low- and high-water request levels based on pipelining status and depth. Fixup texture console which was damaged in a recent release. Split logging of the no-request HTTP error case into two cases: one for missing URL in HTTP request, one for HTTP request not created. A refactor in llcorehttp is coming: I will be moving all libcurl- using code into libcurl-specific modules.
2013-10-04Merge. Pull in viewer-release after 3.6.7 release.Monty Brandenberg
2013-09-11SH-4490 More 'humane' error code presentation from llcorehttp callersMonty Brandenberg
Added toTerseString() conversion on HttpStatus to generate a string that's more descriptive than the hex value of the HttpStatus value but still forms a short, searchable token (e.g. "Http_503" or "Core_7"). Using this throughout the viewer now, no live cases of toHex(), I believe.
2013-08-19Add 'internal'/'external' token to DEBUG retry message so that dev/QAMonty Brandenberg
can know exactly where a retry value was sourced.
2013-08-16SH-4407 Tuning to get new code working as well.Monty Brandenberg
Do some runtime code avoidance and skip unnecessary libcurl and syscall invocations.
2013-07-30SH-4371 Reduce 22mS inter-connection latency.Monty Brandenberg
This really extended into the client-side request throttling. Moved this from llmeshrepository (which doesn't really want to do connection management) into llcorehttp. It's now a class option with configurable rate. This still isn't the right thing to do as it creates coupling between viewer and services. When we get to pipelining, this notion becomes invalid.
2013-07-29Reduce HTTP request retry log spam. Thought I'd done this in anMonty Brandenberg
earlier maintenance branch but the code never showed up. I'll do it again. Spam is still available by bumping 'CoreHttp' tag up to DEBUGS level logging. Needed for QA. Can also get this data from tracing.
2013-07-12SH-4312 Configuration data between viewer and llcorehttp is clumsy.Monty Brandenberg
Much improved. Unified the global and class options into a single option list. Implemented static and dynamic setting paths as much as possible. Dynamic path does require packet/RPC but otherwise there's near unification. Dynamic modes can't get values back yet due to the response/notifier scheme but this doesn't bother me. Flatten global and class options into simpler struct-like entities. Setter/getter available on these when needed (external APIs) but code can otherwise fiddle directly when it knows what to do. Much duplicated options/state removed from HttpPolicy. Comments cleaned up. Threads better described and consistently mentioned in API docs. Integration test extended for 503 responses with Reply-After headers.
2013-06-20SH-4257 Preparation for a new cap grant: GetMesh2Monty Brandenberg
Mesh repo is using three policy classes now: one for large objects, one for GetMesh2 regions, one for GetMesh regions. It's also detecting the presence of the cap and using the correct class. Class initialization cleaned up significantly in llappcorehttp using data-directed code. Pulled in the changes to HttpHeader done for sunshine-internal then did a refactoring pass on the header callback which now uses a unified approach to clean up and deliver header information to all interested parties. Added support for using Retry-After header information on 503 retries.
2013-05-09Merge downstream code and viewer-betasimon
2013-05-06BUG-2295/MAINT-2624 unexpected crash around Content-Range: header processingMonty Brandenberg
Not certain what the source of the short data is with one resident but I'm going to make these problems retryable as they are transport-related. Lift the retry detection into a method that should be reusable by others interested in determining what is retryable. Trace output handling on the libcurl debug callback was attrocious. Some unsafe length handling on my part was protected by a second layer of defense. Made that correct and more useful by logging actual data sizes during trace.
2013-06-19SH-4252 Add second policy class for large mesh asset downloadsMonty Brandenberg
Added second mesh class as well as an asset upload class. Refactored initialization to use less code and more data to cleanly get http started. Modified mesh to use the new http class for large requests (>2MB for now). Added additional timeout setting to llcorehttp to distinguish connection timeout from transport timeout and are now using transport timeout values for large asset downloads that may need more time.
2013-03-29Update Mac and Windows breakpad builds to latestGraham Madarasz
2012-08-01SH-3308 Beef up retry messaging.Monty Brandenberg
Reformatted messages around request retry. Successfully retried requests also message so you can see the cycle closed. Added additional retryable error codes (timeout, other libcurl failures). Commenting and removed some unnecessary std::min logic.
2012-07-23Big comment and naming cleanup. Ready for prime-time.Monty Brandenberg
Add to-do list to _httpinternal.h to guide anyone who wants to pitch in and help.
2012-07-03Add CURLE_SEND_ERROR and CURLE_RECV_ERROR to the set of retryable errors.Monty Brandenberg
Data problems after connections are established should be retried as well. Extend to appropriate libcurl codes. Also allow our connectivity to drop to as low as a single connection when trying to recover.
2012-07-03More integration work for texture fetch timeouts.Monty Brandenberg
The fetch state machine received a new timeout during the WAIT_HTTP_REQ state. For the integration, rather than jump the state to done, we issue a request cancel and let the notification plumbing do the rest without any race conditions or special-case logic.
2012-06-26SH-3184/SH-3221 More work on cleanup with better unit test work and more ↵Monty Brandenberg
aggressive shutdown of a thread. Some additional work let me enable a memory check for the clean shutdown case and generally do a better job on other interfaces. Request queue waiters now awake on shutdown and don't sleep once the queue is turned off. Much better semantically for how this will be used.
2012-06-23SH-3184/SH-3221 Improve cleanup, destructor, thread termination, etc. logic ↵Monty Brandenberg
in library. With this commit, the cleanup paths should be production quality. Unit tests have been expanded to include cases requiring thread termination and cleanup by the worker thread. Special operation/request added to support the unit tests. Thread interface expanded to include a very aggressive cancel() method that does not do cleanup but prevents the thread from accessing objects that will be destroyed.
2012-06-22This sets down the groundwork for dynamic policy classes.Monty Brandenberg
Groundwork is used for the default class which currently represents texture fetching. Class options implemented from API user into HttpLibcurl. Policy layer is going to start doing some traffic shaping like work to solve problems with consumer-grade gear. Need to have dynamic aspects to policies and that starts now...
2012-06-19Cleanup: move magic nubmers to new _httpinternal.h header file.Monty Brandenberg
2012-06-14LLMutex recursive lock, global & per-request tracing, simple GET request, ↵Monty Brandenberg
LLProxy support, HttpOptions starting to work, HTTP resource waiting fixed. Non-LLThread-based threads need to do some registration or LLMutex locks taken out in these threads will not work as expected (SH-3154). We'll get a better solution later, this fixes some things for now. Tracing of operations now supported. Global and per-request (via HttpOptions) tracing levels of [0..3]. The 2 and 3 levels use libcurl's VERBOSE mode combined with CURLOPT_DEBUGFUNCTION to stream high levels of detail into the log. *Very* laggy but useful. Simple GET request supported (no Range: header). Really just a degenrate case of a ranged get but supplied an API anyway. Global option to use the LLProxy interface to setup CURL handles for either socks5 or http proxy usage. This isn't really the most encapsulated way to do this but a better solution will have to come later. The wantHeaders and tracing options are now supported in HttpOptions giving per-request controls. Big refactoring of the HTTP resource waiter in lltexturefetch. What I was doing before wasn't correct. Instead, I'm implementing the resource wait after the Semaphore model (though not using system semaphores). So instead of having a sequence like: SEND_HTTP_REQ -> WAIT_HTTP_RESOURCE -> SEND_HTTP_REQ, we now do WAIT_HTTP_RESOURCE -> WAIT_HTTP_RESOURCE2 (actual wait) -> SEND_HTTP_REQ. Works well but the prioritized filling of the corehttp library needs some performance work later.
2012-06-12HTTP Proxy, PUT & POST, unit tests and refactoring.Monty Brandenberg
Implemented/modified PUT & POST to not used chunked encoding for the request. Made the unit test much happier and probably a better thing for the pipeline. Have a cheesy static & dynamic proxy capability using both local options and a way to wire into LLProxy in llmessages. Not a clean thing but it will get the proxy path working with both socks5 & http proxies. Refactoring to get rid of unneeded library handler and unified an HttpStatus return for all requests. Big batch of code removed as a result of that and more is possible as well as some syscall avoidance with a bit more work. Boosted the unit tests for simple PUT & POST test which revealed the test harness does *not* like chunked encoding so we'll avoid it for now (and don't really need it in any of our schemes).
2012-06-11Refactoring of the request completion thread and removal of ↵Monty Brandenberg
206/content-range hack in xport. Retry/response handling is decided in policy so moved that there. Removed special case 206-without-content-range response in transport. Have this sitation recognizable in the API and let callers deal with it as needed.
2012-06-08Implemented HTTP retry for requests. Went in rather easily whichMonty Brandenberg
surprised me. Added a retry queue similar to ready queue to the policy object which is sorted by retry time. Currently do five retries (after the initial try) delayed by .25, .5, 1, 2 and 5 seconds. Removed the retry logic from the lltexturefetch module. Upped the waiting time in the unit test for the retries. People won't like this but tough, need tests.
2012-06-06Policy + caching fixes + https support + POST workingMonty Brandenberg
Implemented first global policy definitions to support SSL CA certificate configuration to support https: operations. Fixed HTTP 206 status handling to match what is currently being done by grid services and to lay a foundation for fixes that will be a response to ER-1824. More libcurl CURLOPT options set on easy handles to do peer verification in the traditional way. HTTP POST working and now reporting asset metrics back to grid for the viewer's asset system. This uses LLSD so that is also showing as compatible with the new library.
2012-06-01Missed two instances of priority typed as 'float'. Became anMonty Brandenberg
excuse to go through an use a typedef for priority and policy class id.
2012-06-01Major steps towards implementing the policy component.Monty Brandenberg
Identified and reacted to the priority inversion problem we have in texturefetch. Includes the introduction of a priority_queue for the requests that are ready. Start some parameterization in anticipation of having policy_class everywhere. Removed _assert.h which isn't really needed in indra codebase. Implemented async setPriority request (which I hope I can get rid of eventually along with all priorities in this library). Converted to using unsigned int for priority rather than float. Implemented POST and did groundwork for PUT.
2012-04-23Okay, imported the core-http library and got it compiling suspiciously easily.Monty Brandenberg
The unit/integration tests don't work yet as I'm still battling cmake/autobuild as usual but first milestone passed.