I've finished my current "part-time" project and am looking for something new.
I've decided to have a crack at writing my own compression / decompression utility. I'm not planning on selling it or anything - it's more for my own interest.
Preferably, it will be be in C# but Java or C is OK.
Can anyone suggest some good sources on compression / decompression techniques that I can study. Hopefully, some that don't involve maths at a doctorate level!
A good book on this topic is The Data Compression Book. It starts off with the basics and ends up with JPEG and fractal image compression algorithms.
I worked through this whole book years ago (the first edition, I think) and I still remember it as a really rewarding experience.
I find that Wikipedia is a pretty good source in this respect. See, for example, Huffman coding, LZW and Deflate for some common lossless general-purpose data compression algorithms.
If you want to do something more specific, the article on JPEG is also really good.
This competition gives a good idea of the current state of the art for text compression, and something to aim for if you come up with something good!
I've found this a useful survey of lossless image compression.
If you read only one academic paper on the subject, make it C.E. Shannon's "A Mathematical Theory of Communication". The ideas there are absolutely fundamental to compression.
This is completely not an answer, but....Compression and Decompression, for me, have never really been of high interest. There are compression classes in System.IO.Compression, and those things take care of it for me. What more do I need to know? If the built-in classes are not sufficient, there are third-party class libs.
(Actually, the DeflateStream and GZipStream classes in .NET exhibit anomalous behavior with previously-compressed data : they can actually increase the size of the data they are asked to compress. Whoops! It's been reported as a problem but the shot was deflected off of the Borg's shield: Connect.
Kudos to Microsoft for soliciting public bug reports; Bados on Microsoft for simply Closing bugs they didn't feel like fixing. )
Related
I have prototyped a library with some image-processing algorithms in Python/Numpy/Scipy, and now I want to port the code to C# and WPF.
I have realized that, although the input files are images (photographs), conceptually what matters to my domain problem is that they are bidimensional arrays of floats, and the operations I perform (grayscale conversion, blur, blob detection, skeletonization), and even persistence, are best performed in floating-point "space", rather than in integer space (which means bytes - uint8 -, usually).
So, I took a look at .NET namespaces, and there are a lot of "Drawing" this, "Imaging" that, "Media" something, and I am utterly confused.
So, the question is: Which .NET class is the most obvious and commonly used "image data container" for floating point image processing.
I know about AForge, but since I am learning C# and my image-processing needs are not so heavy at this point, I'd like to give native .NET a chance (but that could be a bad idea anyway, so please let me know if it is).
Based on what you already have, why not looking for the same libraries you used in Python but for C#/.NET? for example, for numeric calculations look at:
Project:
http://numerics.mathdotnet.com/
Examples: https://github.com/mathnet/mathnet-numerics/tree/master/src/Examples
And for examples of image processing, maybe looking at the source code of Paint.NET (its latest open sourced version - openpdn Fork of Paint.NET 3.36.7) may give you an idea of what libraries to use for images:
http://code.google.com/p/openpdn/source/browse/#hg%2Fsrc
Both libraries are in C#.
Greetings each and all!
I'm currently looking into procedural generation of a road network and stumbled upon the L-system algorithm. From what I understand from various scientific papers on the subject, and further papers on the papers on the subject, the algorithm is changed to use "global goals and local constraints", in which the taken path is modified to fit input values such as terrain and population density. Now that part I understand, or atleast the overall concept, but how am I supposed to modify the algorithm?
Right now I have a string which is modified over timesteps according to a set of rules. I then analyze the string and move and turn as I go through the chars, render the result and get beautiful patterns on screen.
Now, to create a network of major roads, should I still use a base axiom with a ruleset and then apply the constraints? And if so, what could a set of good startvalues and rules be?
Or should I rather replace the basic ruleset with the constraints and global goals? And if so, what remains of the original L-system algorithm?
Any help is greatly appreciated, and for the record I'm doing this in C# and XNA, allthough I reccon this is more on a theoretical plane.
Thanks for your time,
Karl
So, I've been googling, reading and understanding more the last week and I've found a solution to the problem which I thought I might share.
I found this brilliant blog post which basically straightened everything out for me:
http://www.newton64.ca/blog/?p=747#7472
That post is based upon another blog post founded here:
http://mollyrocket.com/forums/viewtopic.php?t=730&sid=a9a2628b059a727cbde67309757ed178
Now, as far as the L-system goes, I'm not quite sure whether this approach really is an L-system anymore. Sure, there are similarities regarding the iterative process of building the network. In L-systems the string grows over iterations and branches are created using "[" or "]" (atleast in the cases I've seen), and in the approach I'm taking now a while-loop and a priority queue does pretty much the same thing.
I also would like to point out that I did not fully understand the papers "describing" how to use an L-system for generating a road network, so my reasoning might be a bit off here. But algorithm naming and boundries aside, I found a solution that works for me, which is good for now.
Happy coding!
Karl
I'm the author of the above blog post -- glad you found it useful! I never did get around to finishing Spare Parts -- and if nothing else, I'd have to change the name -- but you've got me thinking about it again.
Certainly, the algorithm I described probably isn't much of an L-system anymore; importantly, though, I think it's pretty much functionally equivalent. I'm a positivist when it comes to programming, so if it works, compile it!
EDIT: I've since taken down my old website, but I've recreated the post here. Hope it's still helpful!
Does anyone know of an open source H.264 encoder in C# (or any other managed language)? I might be able to make do with a python implementation as well.
The libraries that I’ve found (e.g. x264) are written in pretty low level c (procedural with lots of macros) and assembly. Tweaking them is turning out to be far more complex than I'd thought. My project has no concern for performance or compatibility. We just want to test how some ideas will impact the perception of the outputted video.
We’d be willing to pay for or license the code if need be.
Thanks in advance!
Edit - Some important points:
I don't care about performance (e.g. real time encoding) at all. It could take 10 days to encode for all I care.
A wrapper isn't helpful since I want to actually modify the encoder itself.
No one would likely spend the time to develop the codec in those languages because they would be hopelessly slow for actual encoding. However, the reference implementation should be less optimized and more useful to you. It is still in C.
http://iphome.hhi.de/suehring/tml/download/
I don't think there is already such a port you need - you'll find wrappers for any langugae but a pure implementation does not have the critical mass. I'd recommend to port it yourself, document your port well and then start tweakening it.
If you'd want to port some encoder to C#, this should be easier to start with - about 8k LOC: http://sourceforge.net/projects/fevh264/
How about http://www.ffmpeg-csharp.com/?
I want to convert Writablebitmap to Jpeg stream, and it looks like there is no platform support as well as I can see a bunch of opensource Encoder libraries on web, I want to get your opinion on which is the recommended one in terms of performance and reliability.
I made good experience with FJCore.
I also blogged about it a while ago http://kodierer.blogspot.com/2009/11/convert-encode-and-decode-silverlight.html
I've spent quite a bit of time with both FJCore and LibJpeg.Net. FJCore is easier to use, since it was ported over from Java, and has an object model that vaguely resembles what you'd expect to see in C#. However, LibJpeg.NET is by far the more complete library (it's based on the informally canonical libjpeg), and it's significantly faster as well. To give one example, FJCore uses a naive implementation of an inverse discrete cosine transform that involves something like 1024 multiplications and an additional 1024 additions for each 8x8 block. In contrast, LibJpeg.NET uses the high performance AAN algorithm which only takes 144 multiplications and 464 additions (see http://datasheets.chipdb.org/Intel/x86/MMX/MMX/AP528.HTM#AAN Algorithm). In addition, FJCore is fairly inefficient in how it uses memory, constantly recreating objects that could easily be re-used. At the same time, because FJCore has fewer optimizations, it's significantly easier to hack.
For my current project (which involves writing a video codec for Silverlight), I used FJCore as a starting point, fixed a whole bunch of its inefficiencies, replaced its IDCT algorithm with the one from LibJpeg.NET, and ended up with something that gave me about 10x the original performance.
Ken why don't you submit your updated code to the FJCore source?
http://code.google.com/p/fjcore/
I'm building an network application that needs to be able to switch from normal network traffic to a zlib compressed stream, mid stream. My thoughts on the matter involve boolean switch that when on will cause the network code to pass all the data through a class that I can feed IEnumerable<byte> into, and then pull out the decompressed stream, passing that on to the already existing protocol parsing code.
Things I've looked at:
ZLib.NET - It seems a little... Ecclectic, and not quite what I want. Would still make a decent start to build off though. (Jon Skeet's comments here hardly inspire me either.)
SharpZipLib - This doesn't seem to support zlib at all? Can anyone confirm or deny this?
I would very much prefer and all managed solution, but let's have at it... are there any other implementations of this library out there in .NET, that might be better suited to what I want to do, or should I take ZLib.NET and build off that as a start?
PS:
Jon's asked for more detail, so here it is.
I'm trying to implement MCCP 2. This involves a signal being sent in the network stream, and everything after this signal is a zlib compressed data stream. There's links to exactly what they mean by that in the above link. Anyway, to be clear, I'm on the recieving end of this (client, not server), and I have a bunch of data read out of the network stream already, and the toggle will be in the middle of this (in all likelyhood atleast), so any solution needs to be able to have some extra data fed into it, before it takes over the NetworkStream (or I manually feed in the rest of the data).
SharpZipLib does support ZLib. Look in the FAQ.
Additionally, have you checked whether the System.IO.Compression namespace supports what you need?
I wouldn't use an IEnumerable<byte> though - streams are designed to be chained together.
EDIT: Okay... it sounds like you need a stream which supports buffering, but with more control than BufferedStream provides. You'd need to "rewind" the stream if you saw the decompression toggle, and then create a GZipStream on top of it. Your buffer would need to be at least as big as your biggest call to Read() so that you could always have enough buffer to rewind.
Included in DotNetZip there is a ZlibStream, for compressing or decompressing zlib streams of data. You didn't ask, but there is also a GZipStream and a DeflateStream. As well as a ZlibCodec class, if that is your thing. (just inflates or deflates buffers, as opposed to streams).
DotNetZip is a fully-managed library with a liberal license. You don't need to use any of the .zip capability to get at the Zlib stuff. And the zlib stuff is packaged as a separate (smaller) DLL just for this purpose.
I can recommend you Gerry Shaw's zlib wrapper for .NET:
http://www.organicbit.com/zip/
As far as I know the ZLib (gzip) library doesn't support listing the files in the header. Assuming that matters to you, but it seems a big shortcoming. This was when I used the sharp zip library a while ago, so I'm willing to delete this :)
Old question, but System.IO.Compression.DeflateStream is actually the right answer if you need proper zlib support:
Starting with the .NET Framework 4.5, the DeflateStream class uses the
zlib library. As a result, it provides a better compression algorithm
and, in most cases, a smaller compressed file than it provides in
earlier versions of the .NET Framework.
Doesn't get better than that.