summaryrefslogtreecommitdiff
path: root/indra/llcorehttp
AgeCommit message (Collapse)Author
2012-09-07DRTVWR-209 Merge of viewer-development with SH-3316 drano-http code.Monty Brandenberg
Cmake files not merged correctly and had to be done by hand. New memory allocation made some memory usage tests in the llcorehttp integration tests no longer valid. Would like to work on LLLog sometime and get it to be consistent. Special flags needed for windows build of example program.
2012-08-01SH-3308 Beef up retry messaging.Monty Brandenberg
Reformatted messages around request retry. Successfully retried requests also message so you can see the cycle closed. Added additional retryable error codes (timeout, other libcurl failures). Commenting and removed some unnecessary std::min logic.
2012-07-24Trivial change to tickle build.Monty Brandenberg
2012-07-23Big comment and naming cleanup. Ready for prime-time.Monty Brandenberg
Add to-do list to _httpinternal.h to guide anyone who wants to pitch in and help.
2012-07-23Cleaning up comments, names, miscellany.Monty Brandenberg
2012-07-16SH-3241 Validate header correctness, SH-3243 more unit testsMonty Brandenberg
Define expectations for headers for GET, POST, PUT requests. Document those in the interface, test those with integration tests. Verify that header overrides work as expected.
2012-07-16SH-3189 Remove/improve naive data structuresMonty Brandenberg
When releasing HTTP waiters, avoid unnecessary sort activity. For Content-Type in responses, let libcurl do the work and removed my parsing of headers. Drop Content-Encoding as libcurl will deal with that. If anyone is interested, they can parse.
2012-07-13SH-3241 validate that request headers are correctMonty Brandenberg
First round of integration tests. Added a request header 'reflector' to the web server to sent the client's headers back with a 'X-Reflect-' prefix. Use boost::regex to check various headers. Run a test on a simple GET and a byte-ranged GET a la texture fetch.
2012-07-12SH-3183 Use valgrind on the library.Monty Brandenberg
Using http_texture_load as the test subject, library looks clean. Did some better shutdown in the program itself and it looks better. Libcurl itself is making a lot of noise. Adapted testrunner to run valgrind as well but the memory allocation tester in the tools themselves grossly interferes with Valgrind operations.
2012-07-11SH-3240 Capture Content-Type and Content-Encoding headers.Monty Brandenberg
HttpResponse object now has two strings for these content headers. Either or both may be empty. Tidied up the cross-platform string code and got more defensive about the length of a header line. Integration test for the new response object.
2012-07-10SH-3244 Syscall avoidance in HttpRequest::update() methodMonty Brandenberg
Well, achieved that by doing work in bulk when needed. But turned into some additional things. Change timebase from mS to uS as, well, things are headed that way. Implement an HttpReplyQueue::fetchAll method (advertised one, hadn't implemented it).
2012-07-09Rework the 'sleep' logic in the test HTTP server so that theMonty Brandenberg
30-second hang doesn't break subsequent tests. Did this by introducing threads into the HTTP server as I can't find the magic to detect that my client has gone away.
2012-07-09SH-3187 Issue smarter 'Range' requests for textures.Monty Brandenberg
First, try to issue ranged GETs that are always at least partially satisfiable. This will keep Varnish-type caches from simply sending back 200/full asset responses to unsatisfiable requests. Implement awareness of Content-Range headers as well. Currently they're not coming back but they will be someday.
2012-07-07SH-3185 Fill in some FIXME/TODO casesMonty Brandenberg
Also added some comments and changed the callback userdata argument to be an HttpOpRequest rather than a libcurl handle. Less code, less clutter.
2012-07-06Odd that this was accepted by VS2010. It clearly wasn't right.Monty Brandenberg
2012-07-06Experiment with ignoring priority in the library. Let upper layersMonty Brandenberg
sort things out or use policy classes (eventually) to arrange low and high priority traffic. Subjectively, I think this works better in practice (as I haven't implemented a dynamic priority setter yet).
2012-07-06SH-3222 Slow loading textures on Lag Me 1Monty Brandenberg
Think I have found the major factor that causes the Linksys WRT54G V5 to fall over in testing scenarios: DNS. For some historical reason, we're trying to use libcurl without any DNS caching. My implementation echoed that and implemented it correctly and I was seeing a DNS request per request on the wire. The existing implementation tries to do that and has bugs because it is clearing caching DNS data querying only once every few seconds. Once I started emulating the bug, comms through the WRT became much, much more reliable.
2012-07-04Example program needs to set Accept: header to talk to Caps router.Monty Brandenberg
2012-07-03Add CURLE_SEND_ERROR and CURLE_RECV_ERROR to the set of retryable errors.Monty Brandenberg
Data problems after connections are established should be retried as well. Extend to appropriate libcurl codes. Also allow our connectivity to drop to as low as a single connection when trying to recover.
2012-07-03More integration work for texture fetch timeouts.Monty Brandenberg
The fetch state machine received a new timeout during the WAIT_HTTP_REQ state. For the integration, rather than jump the state to done, we issue a request cancel and let the notification plumbing do the rest without any race conditions or special-case logic.
2012-07-02Merge 3.3.3 release with Drano HTTP library at 3.3.0Monty Brandenberg
Big delta was converting the new texture debugger support code to the new library. Viewer manifest should probably get an eyeball before release.
2012-06-26SH-3184/SH-3221 More work on cleanup with better unit test work and more ↵Monty Brandenberg
aggressive shutdown of a thread. Some additional work let me enable a memory check for the clean shutdown case and generally do a better job on other interfaces. Request queue waiters now awake on shutdown and don't sleep once the queue is turned off. Much better semantically for how this will be used.
2012-06-23SH-3184/SH-3221 Improve cleanup, destructor, thread termination, etc. logic ↵Monty Brandenberg
in library. With this commit, the cleanup paths should be production quality. Unit tests have been expanded to include cases requiring thread termination and cleanup by the worker thread. Special operation/request added to support the unit tests. Thread interface expanded to include a very aggressive cancel() method that does not do cleanup but prevents the thread from accessing objects that will be destroyed.
2012-06-22This sets down the groundwork for dynamic policy classes.Monty Brandenberg
Groundwork is used for the default class which currently represents texture fetching. Class options implemented from API user into HttpLibcurl. Policy layer is going to start doing some traffic shaping like work to solve problems with consumer-grade gear. Need to have dynamic aspects to policies and that starts now...
2012-06-22SH-3177, SH-3180 std::iostream and LLSD serialization for BufferArray objects.Monty Brandenberg
Seems to be working correctly. Not certain this is the fastest possible way to provide a std::streambuf interface but it's visually acceptable.
2012-06-22Always another platform build fix. Mac this time.Monty Brandenberg
2012-06-21Preparing for better shutdown/cleanup logic.Monty Brandenberg
2012-06-21Compiler warning fix on linux.Monty Brandenberg
2012-06-21SH-3177 Add streambuf/iostream adapters to BufferArray object.Monty Brandenberg
Initial version that should have enough of the plumbing to produce a working adapter. Memory test is showing 8 bytes held after one of the tests so I'm going to revisit that later. But basic functionality is there going by the unit tests.
2012-06-20SH-3181 More interface options for API. Also includes returned headers.Monty Brandenberg
Only thing interesting in this changeset is the discovery that a sleep in the fake HTTP server ties up tests. Need to thread that or fail on client disconnect or something to speed that up and make it usable for bigger test scenarios. But good enough for now...
2012-06-19When a Content-Range header is received, make available the full triplet of ↵Monty Brandenberg
<offset, length, fulllength>.
2012-06-19Cleanup: move magic nubmers to new _httpinternal.h header file.Monty Brandenberg
2012-06-19Implement timeout and retry count options for requests.Monty Brandenberg
Pretty straightforward. Still don't like how I'm managing the options block. Struct? Accessors? Can't decide. But the options now speed up the unit test runs even as I add tests.
2012-06-18Move dtors for refcounted objects to protected access.Monty Brandenberg
2012-06-16Add metrics gathering utils for Mac OS X. All platforms have useful numbers ↵Monty Brandenberg
now.
2012-06-16Implement metrics collection for Linux. Next: Mac OS X.Monty Brandenberg
2012-06-16First round of basic tuning work (shorter sleeps, larger BufferArray blocks).Monty Brandenberg
Beefed up the metrics gathering in http_texture_load to get memory sizes and cpu consumption on windows (still need to implement that on Mac & linux). Ran runs with various idle loops with sleeps from 20 ms down to pure spinning, varied Block allocation size from 1504 to 2^20 bytes. 2ms/2ms/65540 appears to be a good spot under the test conditions (Win7, danu grid, client in Boston).
2012-06-15Fix for linux/mac builds.Monty Brandenberg
2012-06-15Ported example (freestanding) program to drive API & generate performance ↵Monty Brandenberg
numbers. This is a command-line utility to pull content down from a service through the llcorehttp library to produce timings and resource footprints.
2012-06-14LLMutex recursive lock, global & per-request tracing, simple GET request, ↵Monty Brandenberg
LLProxy support, HttpOptions starting to work, HTTP resource waiting fixed. Non-LLThread-based threads need to do some registration or LLMutex locks taken out in these threads will not work as expected (SH-3154). We'll get a better solution later, this fixes some things for now. Tracing of operations now supported. Global and per-request (via HttpOptions) tracing levels of [0..3]. The 2 and 3 levels use libcurl's VERBOSE mode combined with CURLOPT_DEBUGFUNCTION to stream high levels of detail into the log. *Very* laggy but useful. Simple GET request supported (no Range: header). Really just a degenrate case of a ranged get but supplied an API anyway. Global option to use the LLProxy interface to setup CURL handles for either socks5 or http proxy usage. This isn't really the most encapsulated way to do this but a better solution will have to come later. The wantHeaders and tracing options are now supported in HttpOptions giving per-request controls. Big refactoring of the HTTP resource waiter in lltexturefetch. What I was doing before wasn't correct. Instead, I'm implementing the resource wait after the Semaphore model (though not using system semaphores). So instead of having a sequence like: SEND_HTTP_REQ -> WAIT_HTTP_RESOURCE -> SEND_HTTP_REQ, we now do WAIT_HTTP_RESOURCE -> WAIT_HTTP_RESOURCE2 (actual wait) -> SEND_HTTP_REQ. Works well but the prioritized filling of the corehttp library needs some performance work later.
2012-06-12Really need to figure out the 'static const' problem on Windows.Monty Brandenberg
For now, workaround...
2012-06-12HTTP Proxy, PUT & POST, unit tests and refactoring.Monty Brandenberg
Implemented/modified PUT & POST to not used chunked encoding for the request. Made the unit test much happier and probably a better thing for the pipeline. Have a cheesy static & dynamic proxy capability using both local options and a way to wire into LLProxy in llmessages. Not a clean thing but it will get the proxy path working with both socks5 & http proxies. Refactoring to get rid of unneeded library handler and unified an HttpStatus return for all requests. Big batch of code removed as a result of that and more is possible as well as some syscall avoidance with a bit more work. Boosted the unit tests for simple PUT & POST test which revealed the test harness does *not* like chunked encoding so we'll avoid it for now (and don't really need it in any of our schemes).
2012-06-12Convert _refcounted.h over to using LLAtomic32<>.Monty Brandenberg
Beware of bad documentation. operator--(int) does not return what the header claimed it did.
2012-06-11Bring in the testrunner/http server scaffold for better integration testing.Monty Brandenberg
This brings in a copy of llmessage's llsdmessage testing server. We run a mocked HTTP service to handle requests and the integration tests run against it by picking up the LL_TEST_PORT environment variable when running. Add some checks and output to produce useful info when run in the wrong environment and when bad status is received. Later will add a dead port as well so we can test that rather than use 'localhost:2'.
2012-06-11Convert BufferArray interfaces to void * (not char *). ↵Monty Brandenberg
HttpRequest::update() honor time limit. Generally, opaque data operations are expected to be over 'void *' and have now converted interfaces to do that. Update() method honors millisecond limit to dwell time. Might want to homologate the millis/uSecs mix later....
2012-06-11Refactoring of the request completion thread and removal of ↵Monty Brandenberg
206/content-range hack in xport. Retry/response handling is decided in policy so moved that there. Removed special case 206-without-content-range response in transport. Have this sitation recognizable in the API and let callers deal with it as needed.
2012-06-11Unit test fixups. Mac/Linux detect memory leak due to llwarns, try/catch ↵Monty Brandenberg
cleanup. Our logging holds on to a changing bit of memory between operations and the memory leak detection I'm using senses this and complains. So, for now, disable the final memory check on Mac & Linux, leave it active on Windows. Solve this for real some other day. Add try/catch blocks to do cleanup in unit tests that go wrong so that we don't get a cascade of assertion failures when subsequent tests find singletons still alive.
2012-06-08Implemented HTTP retry for requests. Went in rather easily whichMonty Brandenberg
surprised me. Added a retry queue similar to ready queue to the policy object which is sorted by retry time. Currently do five retries (after the initial try) delayed by .25, .5, 1, 2 and 5 seconds. Removed the retry logic from the lltexturefetch module. Upped the waiting time in the unit test for the retries. People won't like this but tough, need tests.
2012-06-06I am getting really bad about adding files. For preceding commit.Monty Brandenberg
2012-06-06Policy + caching fixes + https support + POST workingMonty Brandenberg
Implemented first global policy definitions to support SSL CA certificate configuration to support https: operations. Fixed HTTP 206 status handling to match what is currently being done by grid services and to lay a foundation for fixes that will be a response to ER-1824. More libcurl CURLOPT options set on easy handles to do peer verification in the traditional way. HTTP POST working and now reporting asset metrics back to grid for the viewer's asset system. This uses LLSD so that is also showing as compatible with the new library.