How Adobe DNG Converter works
What is DNG file format
First, what are DNG files? When you take a picture with your camera, it saves a file to the memory card. The file contains data from the image sensor: for each pixel a single value that is proportional to the amount of light that hits the pixel.
The file also contains lots of other data which may be relevant for processing pixel values into a high quality image, or which are useful for other things, like organizing files in a database. Examples for such non-image data are date and time, GPS coordinates, shutter speed and other camera settings, camera model and serial number, information on which lens was used, focus information, etc. This is usually called metadata, and a file from a modern camera can easily contain hundreds of metadata fields.
In addition, a camera usually generates some low-resolution processed versions of the image data which are added to the file as a thumbnail or image preview. All this data must be organized so that it can be easily retrieved.
The file also contains lots of other data which may be relevant for processing pixel values into a high quality image, or which are useful for other things, like organizing files in a database. Examples for such non-image data are date and time, GPS coordinates, shutter speed and other camera settings, camera model and serial number, information on which lens was used, focus information, etc. This is usually called metadata, and a file from a modern camera can easily contain hundreds of metadata fields.
In addition, a camera usually generates some low-resolution processed versions of the image data which are added to the file as a thumbnail or image preview. All this data must be organized so that it can be easily retrieved.
How DNG compares to proprietary formats like ARW, CR2, NEF and RAF
Most camera manufacturers have come up with their own preferred way of data organization, and this is how we ended up with the various raw formats that we know: ARW, CR2, NEF, RAF, and possibly others.
The thing is that manufacturers rarely release the specifications of the file formats to the public, and accessing the image data, especially the metadata, can be extremely challenging outside the software that the manufacturer provides. The fact that new camera models often silently introduce changes or new features to the specification makes the task even harder.
The lack of open specifications for these proprietary formats entails a number of issues. Among other things, it is extremely hard to develop software for image processing, and long-term archival made difficult, because one needs to preserve not only the image files themselves, but also the necessary software (and compatible hardware) for reading the files. These are, supposedly, some of the reasons why Adobe introduced the Digital Negative (DNG) format.
Like the proprietary formats, DNG is a way to organize image and metadata in a file. Contrary to the propriety formats, the DNG specification is open and free of charge for most practical purposes.
The thing is that manufacturers rarely release the specifications of the file formats to the public, and accessing the image data, especially the metadata, can be extremely challenging outside the software that the manufacturer provides. The fact that new camera models often silently introduce changes or new features to the specification makes the task even harder.
The lack of open specifications for these proprietary formats entails a number of issues. Among other things, it is extremely hard to develop software for image processing, and long-term archival made difficult, because one needs to preserve not only the image files themselves, but also the necessary software (and compatible hardware) for reading the files. These are, supposedly, some of the reasons why Adobe introduced the Digital Negative (DNG) format.
Like the proprietary formats, DNG is a way to organize image and metadata in a file. Contrary to the propriety formats, the DNG specification is open and free of charge for most practical purposes.
Metadata in DNG Converter
Adobe DNG Converter is a software that allows to convert files from various proprietary formats to DNG. Considering the number of metadata fields in a typical file, and that each manufacturer has a different way for specifying the metadata, this is a highly non-trivial task.
By default, the conversion (not compression!) of image data is lossless (see also below), meaning the image data is preserved bit by bit in the DNG file. However, the metadata cannot always be preserved 100%, because of deviations between the proprietary and DNG specifications. For these purposes, a copy of the original metadata can be embedded in the DNG file, but this brings us back to the issue of the proprietary format.
Even though the image data is perfectly preserved in the conversion to DNG, the images may appear slightly different when viewed, depending on the software used. This can be caused by differences in the metadata, or by the software using different processing pipelines for different file formats.
By default, the conversion (not compression!) of image data is lossless (see also below), meaning the image data is preserved bit by bit in the DNG file. However, the metadata cannot always be preserved 100%, because of deviations between the proprietary and DNG specifications. For these purposes, a copy of the original metadata can be embedded in the DNG file, but this brings us back to the issue of the proprietary format.
Even though the image data is perfectly preserved in the conversion to DNG, the images may appear slightly different when viewed, depending on the software used. This can be caused by differences in the metadata, or by the software using different processing pipelines for different file formats.
Compression in DNG Converter
The DNG specification allows to store data uncompressed, or with lossless or lossy compression. Let us go through an example for an image sensor with 46 megapixels (MP) and a bit depth of 14 bits per pixel (bpp), similar to that of the Nikon D850.
The original uncompressed image data directly from the sensor requires 46 MP x 14 bpp = 644 Mbit = 80.5 MB of storage space. However, DNG respects the preference of computers to arrange data in chunks of 8 bits (1 byte), so it would actually use 46 MP x 16 bpp = 736 Mbit = 92 MB of storage space, disregarding additional space for metadata and preview images.
The original uncompressed image data directly from the sensor requires 46 MP x 14 bpp = 644 Mbit = 80.5 MB of storage space. However, DNG respects the preference of computers to arrange data in chunks of 8 bits (1 byte), so it would actually use 46 MP x 16 bpp = 736 Mbit = 92 MB of storage space, disregarding additional space for metadata and preview images.
Lossless compression in DNG Converter
The amount of storage space needed can be reduced without any worries whatsoever by activating lossless compression in the DNG Converter, and most proprietary formats have a corresponding lossless mode. In the DNG specification, lossless compression is achieved by using Lossless JPEG, which despite its name has nothing in common with the better known lossy JPEG.
In Lossless compression, the data is processed using reversible mathematical operations and an encoding with a variable number of bits per pixel to reduce file size.
It may, for example, be possible that a large part of the image is a smooth blue sky, where differences between neighbouring pixel values are close to 0. These differences can then be encoded in, say, 2 bits instead of the original 16, saving a factor of 8 for this part of the image. Usually, there will then be other parts of the image where pixels take up more than 16 bits, but such that there is still a net reduction in file size. The reduction in file size comes at the cost of processing time. The software that writes or reads the image data has to figure out how many bits each pixel occupies and to the calculation necessary to retrieve the original pixel value. This is usually a compromise: the more complex mathematical operations are available, the more the file size can be reduced. However, typical raw images can only be compressed by a factor of about 2, because about half of the image data is occupied by noise, and noise cannot be "simplified" with any reversible mathematical operations.
So in a DNG with lossless compression, our sample image file will typically take up between 40 and 50 MB, instead of the original 92 MB.
In Lossless compression, the data is processed using reversible mathematical operations and an encoding with a variable number of bits per pixel to reduce file size.
It may, for example, be possible that a large part of the image is a smooth blue sky, where differences between neighbouring pixel values are close to 0. These differences can then be encoded in, say, 2 bits instead of the original 16, saving a factor of 8 for this part of the image. Usually, there will then be other parts of the image where pixels take up more than 16 bits, but such that there is still a net reduction in file size. The reduction in file size comes at the cost of processing time. The software that writes or reads the image data has to figure out how many bits each pixel occupies and to the calculation necessary to retrieve the original pixel value. This is usually a compromise: the more complex mathematical operations are available, the more the file size can be reduced. However, typical raw images can only be compressed by a factor of about 2, because about half of the image data is occupied by noise, and noise cannot be "simplified" with any reversible mathematical operations.
So in a DNG with lossless compression, our sample image file will typically take up between 40 and 50 MB, instead of the original 92 MB.
Lossy compression in DNG Converter
Finally, there is lossy compression. Lossy compression also involves processing of the image data such that the output file has a variable number of bits per pixel. However, here the mathematical operations are not required to be reversible. On one hand, this does not allow to overcome the limit of a factor of 2, because the noise now can be simplified. On the other hand, this simplification may visibly affect the image data, resulting in compression artefacts. In particular, lossy DNG uses a variation of lossy JPEG compression with all its advantages and disadvantages.
A DNG file with lossy compression of the above mentioned sample image of 92MB would typically result in about 25 to 35 MB in size, with some artefacts present, often noticeable by human eye.
A DNG file with lossy compression of the above mentioned sample image of 92MB would typically result in about 25 to 35 MB in size, with some artefacts present, often noticeable by human eye.
Difference between DNG Converter and Rawsie compression
Because it is practically impossible to read and write proprietary files, Rawsie relies on the Adobe DNG Converter. Once the files have been converted, Rawsie can run its algorithms to optimize the image data inside the DNG.
Rawsie does not use compression part of DNG converter. And this is exactly the difference between compressing with DNG Converter and Rawsie.
Rawsie does not use compression part of DNG converter. And this is exactly the difference between compressing with DNG Converter and Rawsie.
Get more resources like this once a month:
Proudly supported by
Powered by Dotphoton
2020 @ Dotphoton SA
2020 @ Dotphoton SA