STORM-472 Fixed crash that happened while decoding jpg image.

In case of files provided in ticket problem occured in libjpeg's jpeg_start_decompress(to be more specofic- inside it's _jinit_color_deconverter()) because if cinfo's out_color_space is RGB, then jpeg_color_space can't be JCS_YCCK (JCS_YCCK jpeg_color_space is possible for JCS_CMYK out_color_space). So when the combination of RGB and JCS_YCCK was encountered, jpeglib called ERREXIT() inside which exit() was called and viewer crashed.

- Checking for this combination before calling jpeg_start_decompress() would solve this problem in this specific case, but there are a lot of possible error combinations which cause libjpeg to exit and thus crash viewer, so copypasting checks from it into viewer code would be cumbersome, uneffective and ugly. So another approach was used instead- by default libjpeg calls exit() after encountering an error, but user can provide his own error handling function instead. on_jpeg_error() function was added in fix for this. It sets true a boolean flag that is used to determine whether there were errors in getImageDimensionsJpeg(), and this function's return value depends on it.

1 files changed, 15 insertions, 1 deletions

diff --git a/indra/llimage/llimagedimensionsinfo.cpp b/indra/llimage/llimagedimensionsinfo.cpp
index 5ea4a236b5..835664c60f 100644
--- a/indra/llimage/llimagedimensionsinfo.cpp
+++ b/indra/llimage/llimagedimensionsinfo.cpp
@@ -30,6 +30,9 @@
 
 #include "llimagedimensionsinfo.h"
 
+// Value is true if one of Libjpeg's functions has encountered an error while working.
+static bool sJpegErrorEncountered = false;
+
 bool LLImageDimensionsInfo::load(const std::string& src_filename,U32 codec)
 {
 	clean();
@@ -101,9 +104,17 @@ bool LLImageDimensionsInfo::getImageDimensionsPng()
 	return true;
 }
 
+// Called instead of exit() if Libjpeg encounters an error.
+void on_jpeg_error(j_common_ptr cinfo)
+{
+	(void) cinfo;
+	sJpegErrorEncountered = true;
+	llwarns << "Libjpeg has encountered an error!" << llendl;
+}
 
 bool LLImageDimensionsInfo::getImageDimensionsJpeg()
 {
+	sJpegErrorEncountered = false;
 	clean();
 	FILE *fp = fopen (mSrcFilename.c_str(), "rb");
 	if (fp == NULL) 
@@ -115,6 +126,9 @@ bool LLImageDimensionsInfo::getImageDimensionsJpeg()
 	jpeg_error_mgr jerr;
 	jpeg_decompress_struct cinfo;
 	cinfo.err = jpeg_std_error(&jerr);
+	// Call our function instead of exit() if Libjpeg encounters an error.
+	// This is done to avoid crash in this case (STORM-472).
+	cinfo.err->error_exit = on_jpeg_error;
 
 	jpeg_create_decompress	(&cinfo);
 	jpeg_stdio_src		(&cinfo, fp);
@@ -128,6 +142,6 @@ bool LLImageDimensionsInfo::getImageDimensionsJpeg()
 	jpeg_destroy_decompress(&cinfo);
 	fclose(fp);
 
-	return true;
+	return !sJpegErrorEncountered;
 }