We Keep Coding

Raycasting Engine with multiple layers

I will start with my situation right now.
I downloaded the raycast project from: https://github.com/ChrisSerpico/raycasting
This is based on the tutorial from here: https://lodev.org/cgtutor/raycasting.html
After I got the project to work, played around a bit and modified some things, I'm currently stuck with adding multiple layers (based on one map per layer). I read a lot of things around the Internet but had no luck implementing that feature.
In this project: https://github.com/Owlzy/OwlRaycastEngine
there are multiple layers added, but that is done with slices and I can't figure out how to implement this in the Serpico project (took this because the floor/ceiling drawing works a lot better there). Textures are saved like this:
Texture2D canvas; // used to convert the buffer to a single texture to be drawn
Color[] buffer; // screen buffer with raw color data to be drawn
Color[][] rawData; // raw data of the individual external textures
// initialize graphics rendering objects
canvas = new Texture2D(GraphicsDevice, SCREEN_WIDTH, SCREEN_HEIGHT);
buffer = new Color[SCREEN_WIDTH * SCREEN_HEIGHT];
rawData = new Color[NUM_TEXTURES][]; //number of Textures
for (int i = 0; i < NUM_TEXTURES; i++)
{
rawData[i] = new Color[TEXTURE_WIDTH * TEXTURE_HEIGHT];
}
The buffer gets filles this way in the Wallcasting loop:
if (TEXTURE_WIDTH * texY + texX <= rawData[texNum].Length - 1)
{
buffer[SCREEN_WIDTH * y + x] = rawData[texNum][TEXTURE_WIDTH * texY + texX];
}
else //avoid crash when running into walls
{
buffer[SCREEN_WIDTH * y + x] = rawData[texNum][rawData[texNum].Length - 1];
}
and finally drawn this way:
canvas.SetData<Color>(buffer);
b.Draw(canvas, new Rectangle(0, 0, SCREEN_WIDTH, SCREEN_HEIGHT), Color.White);
The code is straight from the lodev tutorial. I tried around with the variables lineHeight , texY and so on but no result. The textures just get stretched, cut off or the screen gets drawn with terrible effects.
Could someone help please? I really dispair...
Thanks a lot!

The problem is in the call in Draw() canvas.SetData<Color>(buffer);
Move this line to Update() and it will "mostly" work. Texture memory is shared between the CPU and GPU. By the time Draw() is called it is expected the textures already exist in GPU memory. Transferring data during draws causes random tearing.
The mostly comes in the nondeterministic delays between Update() and Draw() and the PCIE memory transfer.

Image resize optimization causes memory usage to sky rocket

I made a function that takes an original image and resizes it to 3 different zoom scales -> 16x, 10x, and 4x. For a better understanding, continue reading this paragraph. Let's say the original image is 1000x1000. I declare that at 1x zoom it's dimensions will be 50x50. That means 4x zoom will be 200x200, 10x zoom will be 500x500 and 16x zoom will be 800x800. So my function needs to resize the original 1000x1000 down to 800x800, then down to 500x500, then down to 200x200. Please note I have done this successfully and my question is regarding memory usage.
Below I have two methods of doing so. Both methods work, but one causes a HUGE memory usage bloat using approximately 3x/4x more memory than the other... I like the 2nd method better, because it loads significantly faster than the first method because it's not resizing each of the 3 images from the original image, instead it's resizing them from the previously resized image.
Notes: I'm using Xcode Instruments to measure memory usage. The ImageResizer Class contains a function called "Resize" which resizes the Image.
Method 1.)
public List<UIImage> InitImageList_BFObjects ( UIImage image, SizeF frameSize )
{
List<UIImage> listOfImages = new List<UIImage>();
for ( int i = 0; i < 3; i++ )
{
if ( i == 0 )
zoomScale = 16f;
else if ( i == 1 )
zoomScale = 10f;
else// if ( i == 2 )
zoomScale = 4f;
Resizer = new ImageResizer(image);
Resizer.Resize(frameSize.Width * zoomScale, frameSize.Height * zoomScale);
UIImage resizedImage = Resizer.ModifiedImage;
listOfImages.Insert(0, resizedImage);
}
return listOfImages;
}
Method 1 works and uses very little memory usage. I ran this with a group of about 20 images. My app had about 14mb of memory usage after this loaded (using Xcodes Instruments to examine memory usage)
Method 2.)
public List<UIImage> InitImageList_BFObjects ( UIImage image, SizeF frameSize )
{
List<UIImage> listOfImages = new List<UIImage>();
for ( int i = 0; i < 3; i++ )
{
if ( i == 0 )
zoomScale = 16f;
else if ( i == 1 )
zoomScale = 10f;
else// if ( i == 2 )
zoomScale = 4f;
if ( listOfImages.Count == 0 )
{
Resizer = new ImageResizer(image);
Resizer.Resize(frameSize.Width * zoomScale, frameSize.Height * zoomScale);
UIImage resizedImage = Resizer.ModifiedImage;
listOfImages.Insert(0, resizedImage);
}
else
{
// THIS LINE CONTAINS THE MAIN DIFFERENCE BETWEEN METHOD 1 AND METHOD 2
// Notice how it resizes from the most recent image from listOfImages rather than the original image
Resizer = new ImageResizer(listOfImages[0]);
Resizer.Resize(frameSize.Width * zoomScale, frameSize.Height * zoomScale);
UIImage resizedImage = Resizer.ModifiedImage;
listOfImages.Insert(0, resizedImage);
}
}
return listOfImages;
}
Method 2 works but the memory usage sky rockets! I ran this with the same group of about 20 images. My app had over 60mb of memory usage after this loaded (using Xcodes Instruments to examine memory usage) Why is the memory usage so high? What is it about Method 2 that causes the memory to sky rocket? It's almost as if a variable is not getting cleaned up properly
* Additional Information, ImageResizer Class **
I cut out the non-needed functions from my ImageResizer Class and renamed it "ImageResizer_Abridged". I even switched over to using this class to make sure I didn't accidentally cut out anything needed.
public class ImageResizer_Abridged
{
UIImage originalImage = null;
UIImage modifiedImage = null;
public ImageResizer_Abridged ( UIImage image )
{
this.originalImage = image;
this.modifiedImage = image;
}
/// <summary>
/// strech resize
/// </summary>
public void Resize( float width, float height )
{
UIGraphics.BeginImageContext( new SizeF( width, height ) );
//
modifiedImage.Draw( new RectangleF( 0,0, width, height ) );
modifiedImage = UIGraphics.GetImageFromCurrentImageContext();
//
UIGraphics.EndImageContext();
}
public UIImage OriginalImage
{
get
{
return this.originalImage;
}
}
public UIImage ModifiedImage
{
get
{
return this.modifiedImage;
}
}
}
I created a simplified test project showing this problem *
Here is a dropbox link to the project: https://www.dropbox.com/s/4w7d87nn0aafph9/TestMemory.zip
Here is Method 1's Xcode Instruments screen shot as evidence (9 mb memory usage):
http://i88.photobucket.com/albums/k194/lampshade9909/AllImagesResizedFromOriginalImage_zps585228c6.jpg
Here is Method 2's Xcode Instruments screens hot as evidence (55 mb memory usage):
http://i88.photobucket.com/albums/k194/lampshade9909/SignificantIncreaseInMemoryUsage_zps19034bad.jpg
Below is the code block needed to run the test project
// Initialize My List of Images
ListOfImages = new List<UIImage>();
for ( int i = 0; i < 30; i++ )
{
// Create a UIImage Containing my original Image
UIImage originalImage = UIImage.FromFile ("b2Bomber.png");
float newWidth = 100f;
float newHeight = 40f;
float zoomScale;
float resizedWidth, resizedHeight;
UIImage resizedImage1;
UIImage resizedImage2;
// Basically, I want to take the originalImage Image and resize it twice.
// Method 1.) Resize the originalImage and save it as ResizedImage1. Resize the originalImage and save it as ResizedImage2. We're finished!
// Method 2.) Resize the originalImage and save it as ResizedImage1. Resize ResizedImage1 and save it as ResizedImage2. We're finished!
// The pro to Method 1 is that we get the best possible quaility on all resized images. The con is, this takes a long time if we're doing dozens of very large images
// The pro to Method 2 is that it's faster than Method 1. This is why I want to use Method 2, it's speed. But it has a HUGE con, it's memory usage.
// Please run this project on an iPad connected to XCodes Instruments to monitor memory usage and see what I mean
zoomScale = 10f;
resizedWidth = newWidth*zoomScale;
resizedHeight = newHeight*zoomScale;
UIGraphics.BeginImageContext( new SizeF( resizedWidth, resizedHeight ) );
originalImage.Draw( new RectangleF( 0, 0, resizedWidth, resizedHeight ) );
resizedImage1 = UIGraphics.GetImageFromCurrentImageContext();
UIGraphics.EndImageContext();
zoomScale = 4f;
resizedWidth = newWidth*zoomScale;
resizedHeight = newHeight*zoomScale;
UIGraphics.BeginImageContext( new SizeF( resizedWidth, resizedHeight ) );
// Run this project on an iPad and examine the memory usage in XCode's Instruments.
// The Real Memory Usage will be aroud 9 MB.
// Uncomment this "originalImage.Draw" line to see this happening, make sure to comment out the "resizedImage1.Draw" line
// originalImage.Draw( new RectangleF( 0, 0, resizedWidth, resizedHeight ) );
// Run this project on an iPad and examine the memory usage in XCode's Instruments.
// The Real Memory Usage will be aroud 55 MB!!
// My question is, why does the memory sky rocket when doing this, and how can I prevent the memory from sky rocketing??
// My App requires me to resize around a hundred images and I want to be able to resize an already resized image (like in this example) without the memory usage sky rocketing like this...
// Uncomment this "resizedImage1.Draw" line to see this happening, make sure to comment out the "originalImage.Draw" line
resizedImage1.Draw( new RectangleF( 0, 0, resizedWidth, resizedHeight ) );
resizedImage2 = UIGraphics.GetImageFromCurrentImageContext();
UIGraphics.EndImageContext();
// Add my resized images to the list of Images
ListOfImages.Add (resizedImage1);
ListOfImages.Add (resizedImage2);
}

I'm not sure about your Resize code but I have seen Scale do strange thing. It's not really strange, once you dig into it, but it's definitively not obvious.
Creating an UIImage can be very cheap, memory wise, as long as its backing CGImage is not created. IOW iOS might not immediately allocate a new CGImage backing image that match the new size. That allocation will be differed until the CGImage is needed.
In such case it's possible for some code (like your method 1) to require almost no additional memory when scaling up. However your 2nd method is using a scaled-up image (and will need to allocate the backing CGImage to do so) so it end up requiring the memory earlier.
How can you check for this ?
Compare your resizedImage.Size with the resizedImage.CGImage.Size. If they don't match then you're likely hitting caching.
Notes
I say might because the caching logic is unknown (undocumented). I know that this can differ from running on the simulator and devices - and it also vary between iOS versions;
Caching is a good thing - but it can be surprising :-) I just wish this was documented.

Have you checked whether Resizer implements the Dispose() method? I don't see you disposing it anywhere.
I believe your new code line is implementing the Zoom on the entire image, hence the increased memory usage.
Resizer is zooming the ENTIRE image at the new zoomed in scale, so that an incoming 4MB image is zoomed to 8MB, 16MB and 32MB, consuming your memory.

UIImage implements IDisposable, so something will have to Dispose of it eventually. The Resize method appears to "lose" the reference to modifiedImage, so I'm going to Dispose() it. Hopefully the caller is doing the same to all images in the list returned by InitImageList_BFObjects when it's done with them. Or that class implements IDisposable and it's kicked up the line of who has to deal with it. But rest assured, these images being created need to be Dispose()d somewhere sometime.
public class ImageResizer_Abridged
{
private readonly UIImage originalImage;
private UIImage modifiedImage;
public ImageResizer_Abridged(UIImage image)
{
this.originalImage = image;
this.modifiedImage = image;
}
/// <summary>
/// stretch resize
/// </summary>
public void Resize(float width, float height)
{
UIGraphics.BeginImageContext(new SizeF(width, height));
//
var oldImage = this.modifiedImage;
this.modifiedImage.Draw(new RectangleF(0, 0, width, height));
this.modifiedImage = UIGraphics.GetImageFromCurrentImageContext();
oldImage.Dispose();
//
UIGraphics.EndImageContext();
}
public UIImage OriginalImage
{
get
{
return this.originalImage;
}
}
public UIImage ModifiedImage
{
get
{
return this.modifiedImage;
}
}
}

Finding an Image Inside Another Image

I'm trying to build an application that solves a puzzle (trying to develop a graph algorithm), and I don't want to enter sample input by hand all the time.
Edit: I'm not trying to build a game. I'm trying to build an agent which plays the game "SpellSeeker"
Say I have an image (see attachment) on the screen with numbers in it, and I know the locations of the boxes, and I have the exact images for these numbers. What I want to do is simply tell which image (number) is on the corresponding box.
So I guess I need to implement
bool isImageInsideImage(Bitmap numberImage,Bitmap Portion_Of_ScreenCap) or something like that.
What I've tried is (using AForge libraries)
public static bool Contains(this Bitmap template, Bitmap bmp)
{
const Int32 divisor = 4;
const Int32 epsilon = 10;
ExhaustiveTemplateMatching etm = new ExhaustiveTemplateMatching(0.9f);
TemplateMatch[] tm = etm.ProcessImage(
new ResizeNearestNeighbor(template.Width / divisor, template.Height / divisor).Apply(template),
new ResizeNearestNeighbor(bmp.Width / divisor, bmp.Height / divisor).Apply(bmp)
);
if (tm.Length == 1)
{
Rectangle tempRect = tm[0].Rectangle;
if (Math.Abs(bmp.Width / divisor - tempRect.Width) < epsilon
&&
Math.Abs(bmp.Height / divisor - tempRect.Height) < epsilon)
{
return true;
}
}
return false;
}
But it returns false when searching for a black dot in this image.
How can I implement this?

I'm answering my question since I've found the solution:
this worked out for me:
System.Drawing.Bitmap sourceImage = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\1.jpg");
System.Drawing.Bitmap template = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\2.jpg");
// create template matching algorithm's instance
// (set similarity threshold to 92.5%)
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.921f);
// find all matchings with specified above similarity
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);
// highlight found matchings
BitmapData data = sourceImage.LockBits(
new Rectangle(0, 0, sourceImage.Width, sourceImage.Height),
ImageLockMode.ReadWrite, sourceImage.PixelFormat);
foreach (TemplateMatch m in matchings)
{
Drawing.Rectangle(data, m.Rectangle, Color.White);
MessageBox.Show(m.Rectangle.Location.ToString());
// do something else with matching
}
sourceImage.UnlockBits(data);
The only problem was it was finding all (58) boxes for said game. But changing the value 0.921f to 0.98 made it perfect, i.e. it finds only the specified number's image (template)
Edit: I actually have to enter different similarity thresholds for different pictures. I found the optimized values by trying, in the end I have a function like
float getSimilarityThreshold(int number)

A better approach is to build a custom class which holds all the information you need instead of relying on the image itself.
For example:
public class MyTile
{
public Bitmap TileBitmap;
public Location CurrentPosition;
public int Value;
}
This way you can "move around" the tile class and read the value from the Value field instead of analyzing the image. You just draw whatever image the class hold to the position it's currently holding.
You tiles can be held in an array like:
private list<MyTile> MyTiles = new list<MyTile>();
Extend class as needed (and remember to Dispose those images when they are no longer needed).
if you really want to see if there is an image inside the image, you can check out this extension I wrote for another post (although in VB code):
Vb.Net Check If Image Existing In Another Image

How can I get a convert a Graphics object to a Bitmap (or access it as a Bitmap)

I know the whole
Bitmap bmp = new Bitmap(100,100) ;
Graphics graphics = Graphics.FromImage(bmp) ;
But that is not what I need to do. Essentially what I am trying to do is slightly modify how something is draw on the screen (it is a rectangle with a line in the middle). I need to change the background of the rectangle (which I could do with a pixel comparison or other similar method).
If I could somehow turn it into a bitmap. There is no way (that I can tell to get/set/modify) the graphics object on the pixel level.

Bob Powell has written an excellent article about accessing image bits directly.
https://web.archive.org/web/20141229164101/http://bobpowell.net/lockingbits.aspx

Once you've used the Graphics, just save the Bitmap - the changes made with the graphics are made to the bitmap.

You could do this using a for loop like so:
for(int i = 0; i < bmp.Width; i++)
{
for(int j = 0; j < bmp.Height; j++)
{
var pixel = bmp.GetPixel(i, j);
//do stuff with pixel here :)
}
}
EDIT: After seeing Tilak's comment, if performance is an issue for you then he is right! (Thanks, I did not know about LockBits! +1) Some examples can be found on the MS BitmapData Class page. However, if this seems complicated the above method should do the trick :)

How to implement pan/zoom on gigapixel bitmaps?

In my project, I'm using (uncompressed 16-bit grayscale) gigapixel images which come from a high resolution scanner for measurement purposes. Since these bitmaps can not be loaded in memory (mainly due to memory fragmentation) I'm using tiles (and tiled TIFF on disc). (see StackOverflow topic on this)
I need to implement panning/zooming in a way like Google Maps or DeepZoom. I have to apply image processing on the fly before presenting it on screen, so I can not use a precooked library which directly accesses an image file. For zooming I intend to keep a multi-resolution image in my file (pyramid storage). The most useful steps seem to be +200%, 50% and show all.
My code base is currently C# and .NET 3.5. Currently I assume Forms type, unless WPF gives me great advantage in this area. I have got a method which can return any (processed) part of the underlying image.
Specific issues:
hints or references on how to implement this pan/zoom with on-demand generation of image parts
any code which could be used as a basis (preferably commercial or LGPL/BSD like licenses)
can DeepZoom be used for this (i.e. is there a way that I can provide a function to provide a tile at the right resulution for the current zoom level?) ( I need to have pixel accurate addressing still)

This CodeProject article: Generate...DeepZoom Image Collection might be a useful read since it talks about generating a DeepZoom image source.
This MSDN article has a section Dynamic Deep Zoom: Supplying Image Pixels at Run Time and links to this Mandelbrot Explorer which 'kinda' sounds similar to what you're trying to do (ie. he is generating specific parts of the mandelbrot set on-demand; you want to retrieve specific parts of your gigapixel image on-demand).
I think the answer to "can DeepZoom be used for this?" is probably "Yes", however as it is only available in Silverlight you will have to do some tricks with an embedded web browser control if you need a WinForms/WPF client app.
Sorry I can't provide more specific answers - hope those links help.
p.s. I'm not sure if Silverlight supports TIFF images - that might be an issue unless you convert to another format.

I decided to try something myself. I came up with a straightforward GDI+ code, which uses the tiles I've already got. I just filter out the parts which are relevant for current clipping region. It works like magic! Please find my code below.
(Form settings double buffering for the best results)
protected override void OnPaint(PaintEventArgs e)
{
base.OnPaint(e);
Graphics dc = e.Graphics;
dc.ScaleTransform(1.0F, 1.0F);
Size scrollOffset = new Size(AutoScrollPosition);
int start_x = Math.Min(matrix_x_size,
(e.ClipRectangle.Left - scrollOffset.Width) / 256);
int start_y = Math.Min(matrix_y_size,
(e.ClipRectangle.Top - scrollOffset.Height) / 256);
int end_x = Math.Min(matrix_x_size,
(e.ClipRectangle.Right - scrollOffset.Width + 255) / 256);
int end_y = Math.Min(matrix_y_size,
(e.ClipRectangle.Bottom - scrollOffset.Height + 255) / 256);
// start * contain the first and last tile x/y which are on screen
// and which need to be redrawn.
// now iterate trough all tiles which need an update
for (int y = start_y; y < end_y; y++)
for (int x = start_x; x < end_x; x++)
{ // draw bitmap with gdi+ at calculated position.
dc.DrawImage(BmpMatrix[y, x],
new Point(x * 256 + scrollOffset.Width,
y * 256 + scrollOffset.Height));
}
}
To test it, I've created a matrix of 80x80 of 256 tiles (420 MPixel). Of course I'll have to add some deferred loading in real life. I can leave tiles out (empty) if they are not yet loaded. In fact, I've asked my client to stick 8 GByte in his machine so I don't have to bother about performance too much. Once loaded tiles can stay in memory.
public partial class Form1 : Form
{
bool dragging = false;
float Zoom = 1.0F;
Point lastMouse;
PointF viewPortCenter;
private readonly Brush solidYellowBrush = new SolidBrush(Color.Yellow);
private readonly Brush solidBlueBrush = new SolidBrush(Color.LightBlue);
const int matrix_x_size = 80;
const int matrix_y_size = 80;
private Bitmap[,] BmpMatrix = new Bitmap[matrix_x_size, matrix_y_size];
public Form1()
{
InitializeComponent();
Font font = new Font("Times New Roman", 10, FontStyle.Regular);
StringFormat strFormat = new StringFormat();
strFormat.Alignment = StringAlignment.Center;
strFormat.LineAlignment = StringAlignment.Center;
for (int y = 0; y < matrix_y_size; y++)
for (int x = 0; x < matrix_x_size; x++)
{
BmpMatrix[y, x] = new Bitmap(256, 256, PixelFormat.Format24bppRgb);
// BmpMatrix[y, x].Palette.Entries[0] = (x+y)%1==0?Color.Blue:Color.White;
using (Graphics g = Graphics.FromImage(BmpMatrix[y, x]))
{
g.FillRectangle(((x + y) % 2 == 0) ? solidBlueBrush : solidYellowBrush, new Rectangle(new Point(0, 0), new Size(256, 256)));
g.DrawString("hello world\n[" + x.ToString() + "," + y.ToString() + "]", new Font("Tahoma", 8), Brushes.Black,
new RectangleF(0, 0, 256, 256), strFormat);
g.DrawImage(BmpMatrix[y, x], Point.Empty);
}
}
BackColor = Color.White;
Size = new Size(300, 300);
Text = "Scroll Shapes Correct";
AutoScrollMinSize = new Size(256 * matrix_x_size, 256 * matrix_y_size);
}
Turned out this was the easy part. Getting async multithreaded i/o done in the background was a lot harder to acchieve. Still, I've got it working in the way described here. The issues to resolve were more .NET/Form multithreading related than to this topic.
In pseudo code it works like this:
after onPaint (and on Tick)
check if tiles on display need to be retrieved from disc
if so: post them to an async io queue
if not: check if tiles close to display area are already loaded
if not: post them to an async io/queue
check if bitmaps have arrived from io thread
if so: updat them on screen, and force repaint if visible
Result: I now have my own Custom control which uses roughly 50 MByte for very fast access to arbitrary size (tiled) TIFF files.

I guess you can address this issue following the steps below:
Image generation:
segment your image in multiple subimages (tiles) of a small resolution, for instace, 500x500. These images are depth 0
combine a series of tiles with depth 0 (4x4 or 6x6), resize the combination generating a new tile with 500x500 pixels in depth 1.
continue with this approach until get the entire image using only a few tiles.
Image visualization
Start from the highest depth
When user drags the image, load the tiles dynamically
When the user zoom a region of the image, decrease the depth, loading the tiles for that region in a higher resolution.
The final result is similar to Google Maps.