I currently work on a project in Unity 5. I am trying to apply a shader to one of my cameras using Camera.RenderWithShader, and after that read and save the image. Here is the code:
Texture2D screenshot = new Texture2D(this.screenWidth, this.screenHeight, TextureFormat.RGB24, false);
this.mainCamera.RenderWithShader(this.myShader,"RenderType");
screenshot.ReadPixels(new Rect(0, 0, this.cameraWidth, this.cameraHeight), 0, 0);
The problem is that, after I save the screenshot texture as a Bitmap, the shader is not applied on the entire image.
But if I use Camera.Render() and apply the shader using OnRenderImage(RenderTexture,RenderTexture), it works.
void OnRenderImage(RenderTexture source, RenderTexture destination)
{
Graphics.Blit(source, destination, this.disparityMaterial);
}
So, my question is: What is the difference between these two approaches and how can I make the Camera.RenderWithShader function work properly?
RenderWithShader and OnRenderImage are two completely different things and have nothing to do with each other.Read the linked manual pages for details and a better understand, but long story short:Prior is applying a shader to all (game)objects the camera can see without any image filters applied so basically it's about using a different shader for the same objects/prefabs/materials to alter something the way you want for the viewer (in your case, GOs should also have their tag set to "RenderType", otherwise the shader will not be applied on them),Latter one however is a "post processing" feature, applying filters only on images already rendered. I.e. an image effect feature.So a good use to prior one is e.g. nightvision on/off, or remove cloth from chicks with that special glasses the player can put hands on (mmmmm), etc while the latter one is clearly just image effects, e.g. a secret agent takes photos while one-finger-kills enemies, but when he gets hit, his equipment is more and more damaged, so photos taken as intel are getting more and more blurry, broken up and such - if that makes sense.
Related
The problem:
I've been working on trying to create a 3D position in Worldspace based on a 2D face RGB face detection, similar to this Microsoft example. I am using Unity 2020.3.16f1, MRTK 2.8.2 and C# for the Unity scripts. I have been able to convert the C++ code shown in the link to C# with a lot of success. One final issue is accessing the HoloLens 2 origin SpatialCoordinateSystem to be used in the Transform between the camera's 2D coordinate system and the 3D worldspace system.
The SO question at this link asks a very similar question, and I have tried to use SpatialLocator.GetDefault().CreateStationaryFrameOfReferenceAtCurrentLocation().CoordinateSystem() as the answers suggest. I call this function in Unity's "Awake" method, to ensure it is set as early as possible, as shown below.
private void Awake()
{
worldSpatialCoordinateSystem = SpatialLocator.GetDefault().CreateStationaryFrameOfReferenceAtCurrentLocation().CoordinateSystem;
}
Problem is that, if the user's headset is moving while the application starts, I notice an offset in the 3D locations commensurate with the direction/position of the head when the application was starting. I have narrowed the problem down to the fact that the HL2 and Unity set an origin SpatialCoordinateSystem just before the function in Awake is called, accounting for the offset between what I expect and what I see.
What I've tried:
I have tried using some of the other solutions listed here as well. I cannot use UnityEngine.Windows.WebCam.PhotoCapture becuase of the way I am create still image captures, and (SpatialCoordinateSystem)Marshal.GetObjectForIUnknown(WorldManager.GetNativeISpatialCoordinateSystemPtr()) appears to be deprecated and unusable. Finally, I tried CreateStationaryFrameOfReferenceAtCurrentLocation(Vector3, Quaternion), and used the inverse of the current Camera.main position and rotation, hoping to compensate for the offset, but it did not appear to work (NumericsConversionExtensions is the UnityEngine-to-System.Numerics converver found here). That code is below.
worldSpatialCoordinateSystem = SpatialLocator.GetDefault().CreateStationaryFrameOfReferenceAtCurrentLocation(NumericsConversionExtensions.ToSystem(Camera.main.transform.position*-1),
NumericsConversionExtensions.ToSystem(Quaternion.Inverse(Camera.main.transform.rotation))).CoordinateSystem;
My question:
Is there either another way to access the origin spatial coordinates or possibly to compensate for the offset when the user is moving their head before Awake is called?
I spent 3 days working on the solution, and found one 1 hour after asking SO. For those who come here, use the code below, originally found here.
using Microsoft.MixedReality.OpenXR;
worldSpatialCoordinateSystem = PerceptionInterop.GetSceneCoordinateSystem(Pose.identity) as SpatialCoordinateSystem;
I'm having difficulty of knowing how to approach or how to tackle this problem. I've looked at some tutorials but they are meant for programmers that already know what they're doing. I followed a video on how to perform a form of pixel collision that applies to regular bounding boxes, where if the bounding boxes collide it checks if any non-transparent pixel in both intersecting boxes are overlapping. If they do, then a boolean will return a true value. Where and how could I start to implement the changing of the bounding box's axis in a rotating object to compliment the texture's appreance? I wouldn't prefer being pointed to an external tutorial because most of the ones I've read assumes the programmer knows everything the writer is talking about.
I've also looked at some source code that perfectly demonstrates what I'm looking for, but it seems I need a very in depth explanation to make any use of reading code as well.
First off, I don't really recommend doing this, as it's gonna be either computing- or resource-intensive (or both).
That said, one idea is to still do your aforementioned AABB method of straight-up pixel on pixel. This requires you to maintain your own pixel data in memory to only be used for collision, as opposed to relying strictly on the texture's data.
To be more specific, using this method you will have to generate what is essentially an "image", or 2 dimensional matrix of some kind, one that represents/follows your rotated image's pixels. But you will not be storing color information in it, as you would with a normal image. Instead, each "pixel" or entry in the structure shall be collision data: "block" or "not block". You could easily use a bitmask to represent this, with 1 meaning "block" and 0 meaning "not block", and you'd need one bit per pixel. (NOTE: Usually you don't need more than just a boolean "on" & "off" for this, but it's possible you may want different types of collision per pixel; if so, bitmasks won't work, instead encode whatever you need per pixel, regardless the overall idea remains the same)
Generating a bitmask (or other such structure) for your sprite will enable you to just use the AABB method; all you'd have to do is use the generated bitmask instead of the texture data directly, and everything else is the same as before. But how do we generate this? That's the true difficulty of this method, because you generating your own image is basically replicating the work of your graphics card when you tell it to do rotations.
You would essentially "draw" out the rotated image yourself. This could be done by stepping through your base texture image data pixel by pixel, and applying a rotational transformation matrix to each pixel to get it to the correct destination in your bitmask/buffer. Once you have the correct destination, you then would test the image data for "block" or "not block" (using transparency as you mentioned) and write a 1 or 0 there accordingly.
While you're generating, you should also keep track of local minima and maxima; that is, how far left, right, up, and down your rotated image goes, just to give it an actual true AABB to live inside for quick checks (i.e. "Do I even need to check per-pixel collision?")
To be fully accurate, you will probably need to know which interpolation/rounding algorithm you're using (bilinear, nearest neighbor, etc.), which can get ugly. Graphics systems often do very complicated things, so taking ALL of this into account just for collision is pretty extreme. At the end of the day, even applying this method, it may not truly be "pixel perfect" as far as "perfectly synchronized with the rendered image output", unless you really go far in replicating exactly what XNA / DirectX is doing.
Finally, when does this generation occur? The answer is every time anything rotates! Otherwise you'll be checking stale data. Obviously you could just keep one buffer per sprite and just keep changing that, to not hog so much memory. But this does mean potentially once per frame if you're rotating consistently. Which means multiple times per frame if multiple sprites are all rotating a lot. Might not be the most computationally friendly.
Heey, i've been looking alot on Lidgren, and i've managed to get some simple console client and servers, but i'm having a really hard time with 2D...
Basically what i have so far is only Console based applications but i found an Example of a 2D game using Lidgren. You opened a server, and then two applications. They connected automatically and you could play with the two windows, seeing the character move in the other screen. That project used an array to load the textures and in the Draw() method it simple draws the array, but using a value from:
foreach (var kvp in positions)
{
// use player unique identifier to choose an image
int num = Math.Abs((int)kvp.Key) % textures.Length;`
// draw player
spriteBatch.Draw(textures[num], kvp.Value, Color.White);
}
Could someone explain what that num variable does? And if i wanted to use diffrent classes to do this, would i simply just do the same but in the player classes, and also, animations - how do you send texture update data?? Sorry that i'm asking so much..but i haven't found anything that actually helps :/
Thanks in advance and if you need to know something else, tell me! :)
The num variable simply chooses a unique texture, that will always be the same, from the textures array. Say you have 4 textures, but the Key the modulo operator (%) will get the remainder of 7/4, which would be 3. Basically it would wrap it around the amount of textures, because they would have to repeat (Ex, Id 4 and Id 8 have the same textures)
For now you can just use a static texture, but you can use the same code to choose a texture variation.
What do you mean texture update data? Such as changing textures, or updating positions? If changing textures, is the texture known or not? (Ex, can an ID/name be sent, or does the data have to?)
We're prerendering large sets of textures to RenderTexture2D and this is the issue we're having:
It seems that randomly during the render of a chunk, the textures for each cell (the top and sides) will corrupt and disappear. The weird things is that they come back when the next chunk is rendered though, so it seems to be something that is occurring on a per-frame basis.
Does anyone know why this occurs (and randomly it seems; note the white rectangle is where a side texture corrupts and you can see from there on out the texture contains just transparent)?
EDIT: The sides of the cubes are being saved to Texture2D but they are still disappearing in the middle of a chunk render and then coming back on the next one. So I don't understand why graphics that are in Texture2D are disappearing and coming back, without reinitialization (and that's the weird part).
RenderTexture2D is only a temporary memory construct, and gets flushed quite quickly and regularly. It is because it is reused in an effort to save memory and to a lesser extent to speed things up. As such you should only treat it as a very temporary place to store your texture. You will want to shift it to a proper Texture2D which will be stored for longer. As just doing a simple:
Texture2D YourPic = (RenderTexture2D)SomeRenderedPic;
Will not do it. This just passes the pointer to the memory space of the rendered image. When the graphics card discards it, then it will still just vanish. What you want to do is something more like:
Color[] MyColorArray = new Color[SomeRenderedPic.Width * SomeRenderedPic.Height];
SomeRendeerPic.GetData<Color>(MyColorArray);
Texture2D YourPic = new Texture2D(
GraphicsDevice,
SomeRenderedPic.Width,
SomeRenderedPic.Height);
YourPic.SetData<Color>(MyColorArray);
Now if I have whipped up that code right then it should store the data and not the pointer into the new texture. This makes the new texture its own unique memory space that won't get flushed the same way a Render Target space would.
There is a down side to this method. It cannot be done at the full refresh rate of XNA. (Something like 60 frames a second... I think... maybe 30... I forget.) At any rate, this may not be fast enough if you need a very constant refreshing. However if you are creating a static texture that doesn't really change much if ever, then this may do the trick for you.
Hopefully this made sense as I am writing this on the fly and late at night. If this doesn't work I apologize. Feel free to write me at jareth_gk#hotmail.com if need be. If I am able to answer your questions I will be happy to.
Otherwise good luck, and be inventive. I am sure there is a solution.
x Jeremy M.
I can't say that we ever solved this issue for sure, but it appears to have been something caused by either threading or splitting the task across multiple cycles. It wasn't an issue with the RenderTarget2D since we were already doing that at the time.
I am trying to apply image manipulation effects in Windows 8 app on camera feeds directly.
I have tried a way using canvas and redrawing images after applying effects getting from webcam directly. But this approach works fine for basic effects but for effects like edge detection its creating large lag and flickering while using canvas approach.
Other way is to create MFT(media foundation transform) but it can be implemented in C about which i have no idea.
Can anyone tell me how can i achieve my purpose of applying effects on webcam stream directly in Windows 8 metro style app either by improving canvas approach so that large effects like edge detection don not have any issues or how can i apply MFT in C# since i have worked on C# language or by some other approach?
I have just played quite a bit in this area the last week and even considered writing a blog post about it. I guess this answer can be just as good.
You can go the MFT way, which needs to be done in C++, but the things you would need to write would not be much different between C# and C++. The only thing of note is that I think the MFT works in YUV color space, so your typical convolution filters/effects might behave a bit differently or require conversion to RGB. If you decide to go that route On the C# application side the only thing you would need to do is to call MediaCapture.AddEffectAsync(). Well that and you need to edit your Package.appxmanifest etc., but let's go with first things first.
If you look at the Media capture using webcam sample - it already does what you need. It applies a grayscale effect to your camera feed. It includes a C++ MFT project that is used in an application that is available in C# version. I had to apply the effect to a MediaElement which might not be what you need, but is just as simple - call MediaElement.AddVideoEffect() and your video file playback now applies the grayscale effect. To be able to use the MFT - you need to simply add a reference to the GrayscaleTransform project and add following lines to your appxmanifest:
<Extensions>
<Extension Category="windows.activatableClass.inProcessServer">
<InProcessServer>
<Path>GrayscaleTransform.dll</Path>
<ActivatableClass ActivatableClassId="GrayscaleTransform.GrayscaleEffect" ThreadingModel="both" />
</InProcessServer>
</Extension>
</Extensions>
How the MFT code works:
The following lines create a pixel color transformation matrix
float scale = (float)MFGetAttributeDouble(m_pAttributes, MFT_GRAYSCALE_SATURATION, 0.0f);
float angle = (float)MFGetAttributeDouble(m_pAttributes, MFT_GRAYSCALE_CHROMA_ROTATION, 0.0f);
m_transform = D2D1::Matrix3x2F::Scale(scale, scale) * D2D1::Matrix3x2F::Rotation(angle);
Depending on the pixel format of the video feed - a different transformation method is selected to scan the pixels. Look for these lines:
m_pTransformFn = TransformImage_YUY2;
m_pTransformFn = TransformImage_UYVY;
m_pTransformFn = TransformImage_NV12;
For my sample m4v file - the format is detected as NV12, so it is calling TransformImage_NV12.
For pixels within the specified range (m_rcDest) or within the entire screen if no range was specified - the TransformImage_~ methods call TransformChroma(mat, &u, &v).
For other pixels - the values from original frame are copied.
TransformChroma transforms the pixels using m_transform. If you want to change the effect - you can simply change the m_transform matrix or if you need access to neighboring pixels as in an edge detection filter - modify the TransformImage_ methods to process these pixels.
This is one way to do it. I think it is quite CPU intensive, so personally I prefer to write a pixel shader for such operations. How do you apply a pixel shader to a video stream though? Well, I am not quite there yet, but I believe you can transfer video frames to a DirectX surface fairly easily and call a pixel shader on them later. So far - I was able to transfer the video frames and I am hoping to apply the shaders next week. I might write a blog post about it. I took the meplayer class from the Media engine native C++ playback sample and moved it to a template C++ DirectX project converted to a WinRTComponent library, then used it with a C#/XAML application, associating the swapchain the meplayer class creates with the SwapChainBackgroundPanel that I use in the C# project to display the video. I had to make a few changes in the meplayer class. First - I had to move it to a public namespace that would make it available to other assembly. Then I had to modify the swapchain it creates to a format accepted for use with a SwapChainBackgroundPanel:
DXGI_SWAP_CHAIN_DESC1 swapChainDesc = {0};
swapChainDesc.Width = m_rcTarget.right;
swapChainDesc.Height = m_rcTarget.bottom;
// Most common swapchain format is DXGI_FORMAT_R8G8B8A8-UNORM
swapChainDesc.Format = m_d3dFormat;
swapChainDesc.Stereo = false;
// Don't use Multi-sampling
swapChainDesc.SampleDesc.Count = 1;
swapChainDesc.SampleDesc.Quality = 0;
//swapChainDesc.BufferUsage = DXGI_USAGE_BACK_BUFFER | DXGI_USAGE_RENDER_TARGET_OUTPUT;
swapChainDesc.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT; // Allow it to be used as a render target.
// Use more than 1 buffer to enable Flip effect.
//swapChainDesc.BufferCount = 4;
swapChainDesc.BufferCount = 2;
//swapChainDesc.Scaling = DXGI_SCALING_NONE;
swapChainDesc.Scaling = DXGI_SCALING_STRETCH;
swapChainDesc.SwapEffect = DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL;
swapChainDesc.Flags = 0;
Finally - instead of calling CreateSwapChainForCoreWindow - I am calling CreateSwapChainForComposition and associating the swapchain with my SwapChainBackgroundPanel:
// Create the swap chain and then associate it with the SwapChainBackgroundPanel.
DX::ThrowIfFailed(
spDXGIFactory.Get()->CreateSwapChainForComposition(
spDevice.Get(),
&swapChainDesc,
nullptr, // allow on all displays
&m_spDX11SwapChain)
);
ComPtr<ISwapChainBackgroundPanelNative> dxRootPanelAsSwapChainBackgroundPanel;
// Set the swap chain on the SwapChainBackgroundPanel.
reinterpret_cast<IUnknown*>(m_swapChainPanel)->QueryInterface(
IID_PPV_ARGS(&dxRootPanelAsSwapChainBackgroundPanel)
);
DX::ThrowIfFailed(
dxRootPanelAsSwapChainBackgroundPanel->SetSwapChain(m_spDX11SwapChain.Get())
);
*EDIT follows
Forgot about one more thing. If your goal is to stay in pure C# - if you figure out how to capture frames to a WriteableBitmap (maybe by calling MediaCapture.CapturePhotoToStreamAsync() with a MemoryStream and then calling WriteableBitmap.SetSource() on the stream) - you can use WriteableBitmapEx to process your images. It might not be top performance, but if your resolution is not too high or your frame-rate requirements are not high - it might just be enough. The project on CodePlex does not officially support WinRT yet, but I have a version that should work that you can try here (Dropbox).
As far as I know, MFTs need to be implemented in C++. I believe that there is a media transform SDK sample which shows implementing some straightforward transforms from a metro style application.