We Keep Coding

Applying fragment shaders to images

My goal is to apply any shader written in SKSL to an image. I am aware that this feature is available in Skia but not in SkiaSharp, but I see that there has been some effort recently to port the functionality. I've been trying to replicate a solution for one of the pre-release versions that one of the contributors made here: https://github.com/mono/SkiaSharp/issues/1319#issuecomment-640529824 but I had to make a few changes to my code and shader as they weren't seemingly compatible with the latest pre-release version (2.88.0-preview.232). Here is mine for reference:
// input values
SKBitmap bmp = new SKBitmap(1000, 1000);
SKCanvas canvas = new SKCanvas(bmp);
float threshold = 1.05f;
float exponent = 1.5f;
// shader
var src = #"
in fragmentProcessor color_map;
uniform float scale;
uniform half exp;
uniform float3 in_colors0;
half4 main(float2 p) {
half4 texColor = sample(color_map, p);
if (length(abs(in_colors0 - pow(texColor.rgb, half3(exp)))) < scale)
discard;
return texColor;
}";
using var effect = SKRuntimeEffect.Create(src, out var errorText);
// input values
var inputs = new SKRuntimeEffectUniforms(effect);
inputs["scale"] = threshold;
inputs["exp"] = exponent;
inputs["in_colors0"] = new[] {1f, 1f, 1f};
// shader values
using var blueShirt = SKImage.FromEncodedData(Path.Combine(PathToImages, "blue-shirt.jpg"));
using var textureShader = blueShirt.ToShader();
var children = new SKRuntimeEffectChildren(effect);
children["color_map"] = textureShader;
// create actual shader
using var shader = effect.ToShader(true, inputs, children);
// draw as normal
canvas.Clear(SKColors.Black);
using var paint = new SKPaint { Shader = shader };
canvas.DrawRect(SKRect.Create(400, 400), paint);
However, I get a System.Runtime.InteropServices.SEHException: 'External component has thrown an exception.' on canvas.DrawRect() which I believe is either a problem with my shader or that this feature is not yet fully supported. Besides manually modifying individual pixels with C# which should be a lot slower than a shader operation, is the latter not possible within the current realm of SkiaSharp? If not, I think I'll be needing to load an image into some library with OpenGL bindings (like OpenTK), apply the shader, take a screenshot of the canvas, then load that image into SkiaSharp, which I imagine is not a great solution. I'd like to see possible solutions even outside of Skia.

OpenTK - fragment shader is not working (triangle is always black)

I'm trying to draw a triangle using an OpenGL(OpenTK) fragment shader.
But always displayed black triangle. (even I changed color in fragment shader.)
Maybe fragment shader is not working.
How to fix it?
I attached my code.
P.S. I'm sorry if I do something wrong with this Post. This is my first time on this site.
Render
window.RenderFrame += (FrameEventArgs args) =>
{
GL.UseProgram(shaderProgram.shaderProgramId);
GL.Clear(ClearBufferMask.ColorBufferBit | ClearBufferMask.DepthBufferBit);
float[] verts = { -0.5f, -0.5f, 0.0f, 0.5f, -0.5f, 0.0f, 0.0f, 0.5f, 0.0f };
int vao = GL.GenVertexArray();
int vertices = GL.GenBuffer();
GL.BindVertexArray(vao);
GL.BindBuffer(BufferTarget.ArrayBuffer, vertices);
GL.BufferData(BufferTarget.ArrayBuffer, verts.Length * sizeof(float), verts, BufferUsageHint.StaticCopy);
GL.VertexAttribPointer(0, 3, VertexAttribPointerType.Float, false,3 * sizeof(float),0);
GL.EnableVertexAttribArray(0);
GL.DrawArrays(OpenTK.Graphics.OpenGL4.PrimitiveType.Triangles, 0, 3);
GL.BindBuffer(BufferTarget.ArrayBuffer, 0);
GL.BindVertexArray(0);
GL.DeleteVertexArray(vao);
GL.DeleteBuffer(vertices);
window.SwapBuffers();
};
Shader Load
public static Shader LoadShader(string shaderLocation, ShaderType shaderType)
{
int shaderId = GL.CreateShader(shaderType);
GL.ShaderSource( shaderId, File.ReadAllText( shaderLocation ) );
GL.CompileShader( shaderId );
string infoLog = GL.GetShaderInfoLog(shaderId);
if (!string.IsNullOrEmpty(infoLog))
{
throw new Exception(infoLog);
}
return new Shader() {shaderId = shaderId};
}
Program binding
public static ShaderProgram LoadShaderProgram(string vertexShaderLocation, string fragmentShaderLocation)
{
int shaderProgramId = GL.CreateProgram();
Shader vertexShader = LoadShader(vertexShaderLocation, ShaderType.VertexShader);
Shader fragShader = LoadShader(fragmentShaderLocation, ShaderType.FragmentShader);
GL.AttachShader(shaderProgramId, vertexShader.shaderId);
GL.AttachShader(shaderProgramId, fragShader.shaderId);
GL.LinkProgram(shaderProgramId);
GL.DetachShader(shaderProgramId, vertexShader.shaderId);
GL.DetachShader(shaderProgramId, fragShader.shaderId);
GL.DeleteShader(vertexShader.shaderId);
GL.DeleteShader(fragShader.shaderId);
string infoLog = GL.GetProgramInfoLog(shaderProgramId);
if (!string.IsNullOrEmpty(infoLog))
{
throw new Exception(infoLog);
}
return new ShaderProgram() {shaderProgramId = shaderProgramId};
}
shaders
vertex
#version 330
layout(location=0) in vec3 vPosition;
out vec4 vertexColor;
void main() {
gl_Position = vec4( vPosition, 1.0);
vertexColor = vec4(0.0,1.0,0.0,1.0);
}
fragment
#version 330
out vec4 FragColor;
in vec4 vertexColor;
void main()
{
FragColor = vertexColor;
}

I see a couple of problems with your code. The main reason you may not see your shader in action is because you do not keep your shaders attached to your shader handle (shaderProgramId). Instead, you detach and delete them right after you compiled and attached them. What you are doing there is basically creating your shader program and then immediately throwing it away.
Another issue (which might not cause your main problem, but yet) may be your usage of a VAO. A VAO is actually meant to preserve the OpenGL states of the objects bound to it across state switches. It is rather a container for VBOs inside it, holding their descriptions. So what you want to do is to create your VAO, bind it, then create, bind and describe (glVertexAttribPointer) your VBOs. After that you can unbind your VAO. When you bind it again, you don't have to do anything extra (like binding VBOs or using glVertexAttribPointer again): What you did when first binding your VAO and adding your VBOs is stored in the VAO. Just bind the VAO, bind your shader (glUseProgram) and happily render away.

Result of rendering into texture is empty only with shader

I think I am very stupid because I can't figure out what causes the issue in this simplest case.
Project type is Windows OpenGL (on Android the issue also persists).
Here is the shader:
//----------------------------------------------------------------------------
float4 PSTest (float2 TexCoords : TEXCOORD0) : COLOR0
{
float4 color = float4 (1, 0, 0, 1);
return color;
}
//----------------------------------------------------------------------------
Technique Test
{
pass Test
{
PixelShader = compile ps_2_0 PSTest();
}
}
It works fine if I draw it to the screen with the Texture (128x128):
Shader.CurrentTechnique = Shader.Techniques["Test"];
GraphicsDevice.SetRenderTarget (null);
GraphicsDevice.Clear (Color.White);
Batch.Begin (SpriteSortMode.Immediate, BlendState.Opaque,
SamplerState.PointClamp, DepthStencilState.None, null, Shader);
Batch.Draw (Texture, Vector2.Zero, Color.White);
Batch.End();
Now I decide to draw it into render target Output created by code:
new RenderTarget2D (GraphicsDevice, 256, 256, false,
SurfaceFormat.Color, DepthFormat.None);
Draw code:
// Render into Output
Shader.CurrentTechnique = Shader.Techniques["Test"];
GraphicsDevice.SetRenderTarget (Output);
GraphicsDevice.Clear (Color.DimGray);
Batch.Begin (SpriteSortMode.Immediate, BlendState.Opaque,
SamplerState.PointClamp, DepthStencilState.None, null, Shader);
Batch.Draw (Texture, Vector2.Zero, Color.White);
Batch.End();
// Draw Output
GraphicsDevice.SetRenderTarget (null);
GraphicsDevice.Clear (Color.White);
Batch.Begin();
Batch.Draw (Output, Vector2.Zero, Color.White);
Batch.End();
But the result is totally absurd:
I have tried different simplest shaders, SurfaceFormat.HdrBlendable, texture sizes - but nothing helps.
Also I have tried to draw Texture (filled with violet) into render target without shader (then to the screen) and it works fine.
It seems like I have missed some basics thing about shaders in XNA but I can't figure out what it is.

Here is the solution: https://monogame.codeplex.com/discussions/530086. The reason is in the shader's input parameters.

Monogame Shader Porting Issues

Ok so I ported a game I have been working on over to Monogame, however I'm having a shader issue now that it's ported. It's an odd bug, since it works on my old XNA project and it also works the first time I use it in the new monogame project, but not after that unless I restart the game.
The shader is a very simple shader that looks at a greyscale image and, based on the grey, picks a color from the lookup texture. Basically I'm using this to randomize a sprite image for an enemy every time a new enemy is placed on the screen. It works for the first time an enemy is spawned, but doesn't work after that, just giving a completely transparent texture (not a null texture).
Also, I'm only targeting Windows Desktop for now, but I am planning to target Mac and Linux at some point.
Here is the shader code itself.
sampler input : register(s0);
Texture2D colorTable;
float seed; //calculate in program, pass to shader (between 0 and 1)
sampler colorTableSampler =
sampler_state
{
Texture = <colorTable>;
};
float4 PixelShaderFunction(float2 c: TEXCOORD0) : COLOR0
{
//get current pixel of the texture (greyscale)
float4 color = tex2D(input, c);
//set the values to compare to.
float hair = 139/255; float hairless = 140/255;
float shirt = 181/255; float shirtless = 182/255;
//var to hold new color
float4 swap;
//pixel coordinate for lookup
float2 i;
i.y = 1;
//compare and swap
if (color.r >= hair && color.r <= hairless)
{
i.x = ((0.5 + seed + 96)/128);
swap = tex2D(colorTableSampler,i);
}
if (color.r >= shirt && color.r <= shirtless)
{
i.x = ((0.5 + seed + 64)/128);
swap = tex2D(colorTableSampler,i);
}
if (color.r == 1)
{
i.x = ((0.5 + seed + 32)/128);
swap = tex2D(colorTableSampler,i);
}
if (color.r == 0)
{
i.x = ((0.5 + seed)/128);
swap = tex2D(colorTableSampler, i);
}
return swap;
}
technique ColorSwap
{
pass Pass1
{
// TODO: set renderstates here.
PixelShader = compile ps_2_0 PixelShaderFunction();
}
}
And here is the function that creates the texture. I should also note that the texture generation works fine without the shader, I just get the greyscale base image.
public static Texture2D createEnemyTexture(GraphicsDevice gd, SpriteBatch sb)
{
//get a random number to pass into the shader.
Random r = new Random();
float seed = (float)r.Next(0, 32);
//create the texture to copy color data into
Texture2D enemyTex = new Texture2D(gd, CHARACTER_SIDE, CHARACTER_SIDE);
//create a render target to draw a character to.
RenderTarget2D rendTarget = new RenderTarget2D(gd, CHARACTER_SIDE, CHARACTER_SIDE,
false, gd.PresentationParameters.BackBufferFormat, DepthFormat.None);
gd.SetRenderTarget(rendTarget);
//set background of new render target to transparent.
//gd.Clear(Microsoft.Xna.Framework.Color.Black);
//start drawing to the new render target
sb.Begin(SpriteSortMode.Immediate, BlendState.Opaque,
SamplerState.PointClamp, DepthStencilState.None, RasterizerState.CullNone);
//send the random value to the shader.
Graphics.GlobalGfx.colorSwapEffect.Parameters["seed"].SetValue(seed);
//send the palette texture to the shader.
Graphics.GlobalGfx.colorSwapEffect.Parameters["colorTable"].SetValue(Graphics.GlobalGfx.palette);
//apply the effect
Graphics.GlobalGfx.colorSwapEffect.CurrentTechnique.Passes[0].Apply();
//draw the texture (now with color!)
sb.Draw(enemyBase, new Microsoft.Xna.Framework.Vector2(0, 0), Microsoft.Xna.Framework.Color.White);
//end drawing
sb.End();
//reset rendertarget
gd.SetRenderTarget(null);
//copy the drawn and colored enemy to a non-volitile texture (instead of render target)
//create the color array the size of the texture.
Color[] cs = new Color[CHARACTER_SIDE * CHARACTER_SIDE];
//get all color data from the render target
rendTarget.GetData<Color>(cs);
//move the color data into the texture.
enemyTex.SetData<Color>(cs);
//return the finished texture.
return enemyTex;
}
And just in case, the code for loading in the shader:
BinaryReader Reader = new BinaryReader(File.Open(#"Content\\shaders\\test.mgfx", FileMode.Open));
colorSwapEffect = new Effect(gd, Reader.ReadBytes((int)Reader.BaseStream.Length));
If anyone has ideas to fix this, I'd really appreciate it, and just let me know if you need other info about the problem.

I am not sure why you have "at" (#) sign in front of the string, when you escaped backslash - unless you want to have \\ in your string, but it looks strange in the file path.
You have wrote in your code:
BinaryReader Reader = new BinaryReader(File.Open(#"Content\\shaders\\test.mgfx", FileMode.Open));
Unless you want \\ inside your string do
BinaryReader Reader = new BinaryReader(File.Open(#"Content\shaders\test.mgfx", FileMode.Open));
or
BinaryReader Reader = new BinaryReader(File.Open("Content\\shaders\\test.mgfx", FileMode.Open));
but do not use both.

I don't see anything super obvious just reading through it, but really this could be tricky for someone to figure out just looking at your code.
I'd recommend doing a graphics profile (via visual studio) and capturing the frame which renders correctly then the frame rendering incorrectly and comparing the state of the two.
Eg, is the input texture what you expect it to be, are pixels being output but culled, is the output correct on the render target (in which case the problem could be Get/SetData), etc.

Change ps_2_0 to ps_4_0_level_9_3.
Monogame cannot use shaders built on HLSL 2.
Also the built in sprite batch shader uses ps_4_0_level_9_3 and vs_4_0_level_9_3, you will get issues if you try to replace the pixel portion of a shader with a different level shader.
This is the only issue I can see with your code.

Why does a valid Texture not reach the Shader? (Direct3D9)

Hello everyone I'm currently trying to create a deferred renderer for my graphics engine using c# and SlimDX. As a resource I use this tutorial which is very helpful eventhough it's intended for XNA.
But right now I'm stuck...
I have my renderer set up to draw all geometry's color, normals and depth to seperate render target textures. This works. I can draw the resulting textures to the restored backbuffer as sprites and I can see that they contain just what they are supposed to. But when I try to pass those Textures to another shader, in this case to create a light map, weirds things happen. Here's how I draw one frame:
public bool RenderFrame(FrameInfo fInfo){
if(!BeginRender()) //checks Device, resizes buffers, calls BeginScene(), etc.
return false;
foreach(RenderQueue queue in fInfo.GetRenderQueues()){
RenderQueue(queue);
}
EndRender(); //currently only calls EndScene, used to do more
ResolveGBuffer();
DrawDirectionalLight(
new Vector3(1f, -1f, 0),
new Color4(1f,1f,1f,1f),
fi.CameraPosition,
SlimMath.Matrix.Invert(fi.ViewProjectionMatrix));
}
private void ResolveGBuffer() {
if(DeviceContext9 == null || DeviceContext9.Device == null)
return;
DeviceContext9.Device.SetRenderTarget(0, _backbuffer);
DeviceContext9.Device.SetRenderTarget(1, null);
DeviceContext9.Device.SetRenderTarget(2, null);
}
private void DrawDirectionalLight(Vector3 lightDirection, Color4 color, SlimMath.Vector3 cameraPosition, SlimMath.Matrix invertedViewProjection) {
if(DeviceContext9 == null || DeviceContext9.Device == null)
return;
DeviceContext9.Device.BeginScene();
_directionalLightShader.Shader.SetTexture(
_directionalLightShader.Parameters["ColorMap"],
_colorTexture);
_directionalLightShader.Shader.SetTexture(
_directionalLightShader.Parameters["NormalMap"],
_normalTexture);
_directionalLightShader.Shader.SetTexture(
_directionalLightShader.Parameters["DepthMap"],
_depthTexture);
_directionalLightShader.Shader.SetValue<Vector3>(
_directionalLightShader.Parameters["lightDirection"],
lightDirection);
_directionalLightShader.Shader.SetValue<Color4>(
_directionalLightShader.Parameters["Color"],
color);
_directionalLightShader.Shader.SetValue<SlimMath.Vector3>(
_directionalLightShader.Parameters["cameraPosition"],
cameraPosition);
_directionalLightShader.Shader.SetValue<SlimMath.Matrix>(
_directionalLightShader.Parameters["InvertViewProjection"],
invertedViewProjection);
_directionalLightShader.Shader.SetValue<Vector2>(
_directionalLightShader.Parameters["halfPixel"],
_halfPixel);
_directionalLightShader.Shader.Technique =
_directionalLightShader.Technique("Technique0");
_directionalLightShader.Shader.Begin();
_directionalLightShader.Shader.BeginPass(0);
RenderQuad(SlimMath.Vector2.One * -1, SlimMath.Vector2.One);
_directionalLightShader.Shader.EndPass();
_directionalLightShader.Shader.End();
DeviceContext9.Device.EndScene();
}
Now when I replace the call to DrawDirectionalLight with some code to draw _colorTexture, _normalTexture and _depthTexture to the screen everything looks ok, but when I use the DrawDirectionalLight function instead I see wild flickering. From the output of PIX it looks like my textures do not get passed to the shader correctly:
Following the tutorial the texture parameters and samplers are defined as follows:
float3 lightDirection;
float3 Color;
float3 cameraPosition;
float4x4 InvertViewProjection;
texture ColorMap;
texture NormalMap;
texture DepthMap;
sampler colorSampler = sampler_state{
Texture = ColorMap;
AddressU = CLAMP;
AddressV = CLAMP;
MagFilter= LINEAR;
MinFilter= LINEAR;
MipFilter= LINEAR;
};
sampler depthSampler = sampler_state{
Texture = DepthMap;
AddressU = CLAMP;
AddressV = CLAMP;
MagFilter= POINT;
MinFilter= POINT;
MipFilter= POINT;
};
sampler normalSampler = sampler_state{
Texture = NormalMap;
AddressU = CLAMP;
AddressV = CLAMP;
MagFilter= POINT;
MinFilter= POINT;
MipFilter= POINT;
};
Now my big question is WHY? There are no error messages printed to debug output.
EDIT:
the rendertargets/textures are created like this:
_colorTexture = new Texture(DeviceContext9.Device,
DeviceContext9.PresentParameters.BackBufferWidth,
DeviceContext9.PresentParameters.BackBufferHeight,
1,
Usage.RenderTarget,
Format.A8R8G8B8,
Pool.Default);
_colorSurface = _colorTexture.GetSurfaceLevel(0);
_normalTexture = new Texture(DeviceContext9.Device,
DeviceContext9.PresentParameters.BackBufferWidth,
DeviceContext9.PresentParameters.BackBufferHeight,
1,
Usage.RenderTarget,
Format.A8R8G8B8,
Pool.Default);
_normalSurface = _normalTexture.GetSurfaceLevel(0);
_depthTexture = new Texture(DeviceContext9.Device,
DeviceContext9.PresentParameters.BackBufferWidth,
DeviceContext9.PresentParameters.BackBufferHeight,
1,
Usage.RenderTarget,
Format.A8R8G8B8,
Pool.Default);
_depthSurface = _depthTexture.GetSurfaceLevel(0);
EDIT 2:
The problems seems to lie in the directionalLightShader itselft since passing other regular textures doesn't work either.

The answer to my problem is as simple as the problem was stupid. The strange behaviour was caused by 2 different errors:
I was just looking at the wrong events in PIX. The textures we passed correctly to the shader but I didn't see it because it was 'hidden' in the BeginPass-event (behind the '+').
The pixel shader which I was trying to execute never got called because vertices of the fullscreen quad I used to render were drawn in clockwise order... my CullMode was also set to clockwise...
Thanks to everyone who read this question!