I have a table called evidence with a trigger which calls a stored procedure which basically does table partitioning by month. However I get an obscure error when I start inserting lots of rows under load:
Npgsql.NpgsqlException: query string argument of EXECUTE is null
Severity: ERROR Code: 22004 at Npgsql.NpgsqlState.<ProcessBackendResponses_Ver_3>d__a.MoveNext() in c:\C#Apps\github.npgsql.Npgsql.stock\src\Npgsql\NpgsqlState.cs:line890 at Npgsql.ForwardsOnlyDataReader.GetNextResponseObject() in c:\C#Apps\github.npgsql.Npgsql.stock\src\Npgsql\NpgsqlDataReader.cs:line 1175 at
Npgsql.ForwardsOnlyDataReader.GetNextRowDescription() in c:\C#Apps\github.npgsql.Npgsql.stock\src\Npgsql\NpgsqlDataReader.cs:line 1191 at
Npgsql.ForwardsOnlyDataReader.NextResult() in c:\C#Apps\github.npgsql.Npgsql.stock\src\Npgsql\NpgsqlDataReader.cs:line 1377 at
Npgsql.NpgsqlCommand.ExecuteNonQuery() in c:\C#Apps\github.npgsql.Npgsql.stock\src\Npgsql\NpgsqlCommand.cs:line523
My system has automatic retry functionality and eventually every record gets inserted into the database, but after many many exceptions when the load is high.
Database is PostgreSQL 9.3 on a CentOS 6 server and client is C# .NET using Npgsql driver.
Table:
CREATE TABLE evidence
(
id uuid NOT NULL,
notification_id uuid NOT NULL,
feedback character varying(200),
result character varying(20),
trigger_action_type character varying(200),
trigger_action_id uuid,
data_type integer NOT NULL,
data bytea,
name character varying(30),
CONSTRAINT pk_evidence PRIMARY KEY (id)
);
Trigger:
CREATE TRIGGER evidence_move_to_partition_tables
BEFORE INSERT
ON evidence
FOR EACH ROW
EXECUTE PROCEDURE partition_evidence_by_month();
Trigger Function:
CREATE OR REPLACE FUNCTION partition_evidence_by_month()
RETURNS trigger AS
$BODY$
DECLARE
_notification_id uuid;
_raised_local_time timestamp without time zone;
_table_name character varying(35);
_start_date timestamp without time zone;
_end_date timestamp without time zone;
_table_space character varying(50) := 'ls_tablespace2';
_query text;
BEGIN
_notification_id := NEW.notification_id;
SELECT raised_local_time FROM notifications WHERE id=_notification_id INTO _raised_local_time;
_start_date := date_trunc('month', _raised_local_time);
_end_date := _start_date + '1 month'::interval;
_table_name := 'evidence-' || to_char(_start_date, 'YYYY-MM');
-- check to see if table already exists
PERFORM 1
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
AND c.relname = _table_name
AND n.nspname = 'public';
-- if the table doesn't exist, then create it now
IF NOT FOUND THEN
-- create partition table
_query := 'CREATE TABLE public.' || quote_ident(_table_name) || ' ( ) INHERITS (public.evidence)';
EXECUTE _query;
-- alter owner
--EXECUTE 'ALTER TABLE public.' || quote_ident(_table_name) || ' OWNER TO postgres';
-- add index
--EXECUTE 'ALTER TABLE public.' || quote_ident(_table_name) || ' ADD PRIMARY KEY (id)';
END IF;
-- move the data to the partition table
EXECUTE 'INSERT INTO public.' || quote_ident(_table_name) || ' VALUES ($1.*)' USING NEW;
RETURN NULL;
END;
$BODY$ LANGUAGE plpgsql VOLATILE COST 100;
Calling Code:
using (var cmd = db.CreateCommand())
{
cmd.CommandText = #"INSERT INTO evidence
(id, notification_id, feedback, result, trigger_action_type,
trigger_action_id, data_type, data, name)
VALUES (#id,#nid,#feedback,#result,#tat,#taid,#dt,#data,#name)";
cmd.Parameters.AddWithValue("#id", evItem.ID);
cmd.Parameters.AddWithValue("#nid", evItem.NotificationID);
cmd.Parameters.AddWithValue("#feedback", evItem.Feedback);
cmd.Parameters.AddWithValue("#result", evItem.Result);
cmd.Parameters.AddWithValue("#tat", evItem.TriggerActionType);
cmd.Parameters.AddWithValue("#taid", evItem.TriggerActionID);
cmd.Parameters.AddWithValue("#dt", (int)evItem.DataType);
cmd.Parameters.AddWithValue("#data", evItem.Data);
cmd.Parameters.AddWithValue("#name", evItem.Name);
cmd.ExecuteNonQuery();
}
Why would this bizarre error appear only when the system is under load? What can I do to prevent it happening?
Thanks!
The error message is
query string argument of EXECUTE is null
You have two EXECUTE commands:
_query := 'CREATE TABLE public.'
|| quote_ident(_table_name) || ' ( ) INHERITS (public.evidence)';
EXECUTE _query;
...
EXECUTE 'INSERT INTO public.'
|| quote_ident(_table_name) || ' VALUES ($1.*)' USING NEW;
The only part that can be NULL is table_name.
The only chance for table_name to become NULL is here:
SELECT raised_local_time FROM notifications WHERE id=_notification_id
INTO _raised_local_time;
So the cause must be one of two reasons:
NEW.notification_id is NULL.
There is no row in notifications for the given NEW.notification_id.
Try this modified trigger function for debugging:
CREATE OR REPLACE FUNCTION partition_evidence_by_month()
RETURNS trigger AS
$func$
DECLARE
_table_name text;
BEGIN
SELECT 'evidence-' || to_char(raised_local_time, 'YYYY-MM')
FROM public.notifications -- schema-qualify to be sure
WHERE id = NEW.notification_id
INTO _table_name;
IF _table_name IS NULL THEN
RAISE EXCEPTION '_table_name is NULL. Should not occur!';
END IF;
IF NOT EXISTS ( -- create table if it does not exist
SELECT 1
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
AND c.relname = _table_name
AND n.nspname = 'public') THEN
EXECUTE 'CREATE TABLE public.'
|| quote_ident(_table_name) || ' ( ) INHERITS (public.evidence)';
END IF;
EXECUTE 'INSERT INTO public.'
|| quote_ident(_table_name) || ' VALUES $1' -- Use NEW row directly
USING NEW; -- write data to the partition table
RETURN NULL;
END
$func$ LANGUAGE plpgsql;
Remove unused variables and simplify code. (This is obviously a simplified example.)
Among other things, you don't need date_trunc() at all. Simply feed the original timestamp to to_char().
No point in using varchar(n). Simply use text or varchar.
Avoid too many assignments where unnecessary - comparatively expensive in PL/pgSQL.
Add a RAISE to check my hypothesis.
If you get the error message, discriminating between the two possible causes would be the next step. Should be trivial ...
Related
Can I define the stored procedure without using the RefCursor ? (like "return refcursor")
I do not want to use OracleDbType.RefCursor because it is not sent as dbparameter in other databases.
Also DbParameter.DbType = OracleDbType.RefCursor; does not supported
I do not want to define "retval IN OUT SYS_REFCURSOR" in the code below. Is there another way?
CREATE OR REPLACE procedure SYSTEM.customer_select_row(
p_email IN CUSTOMER.Email%TYPE,
p_password IN CUSTOMER."Password"%TYPE,
retval IN OUT SYS_REFCURSOR
)
IS
BEGIN
OPEN retval FOR
SELECT CustomerId, FirstName, LastName FROM CUSTOMER
WHERE Email = p_email AND "Password" = p_password
END customer_select_row;
You could use a pipeline Function,
It is a function that works exacltly as a table
you can call it this way
SELECT *
FROM TABLE(TEST_PIPELINE.STOCKPIVOT(10));
the TEST_PIPELINE.STOCKPIVOT(10) is a function
you can build it this way:
create or replace PACKAGE TEST_PIPELINE AS
-- here you declare a type record
type t_record is record
(
field_1 VARCHAR2(100),
field_2 VARCHAR2(100));
-- declare a table type from your previously created type
TYPE t_collection IS TABLE OF t_record;
-- declare that the function will return the collection pipelined
FUNCTION StockPivot(P_LINES NUMBER) RETURN t_collection PIPELINED;
END;
/
create or replace PACKAGE BODY TEST_PIPELINE IS
FUNCTION StockPivot(P_LINES NUMBER) RETURN t_collection PIPELINED IS
-- declare here a type of the record
T_LINE T_RECORD;
BEGIN
-- here is a loop example for insert some lines on pipeline
FOR I IN 1..P_LINES LOOP
-- inser data on your line this way
T_LINE.field_1 := 'LINE - ' || I;
T_LINE.field_2 := 'LINE - ' || I;
-- then insert insert the line for result (this kind of functions should not have a return statement)
PIPE ROW (T_LINE );
END LOOP;
END;
END;
I am having some trouble with the CREATE IF NOT EXISTS clause.
I am using a C# application to create a MySQL table, the connection to DB has been established so it's not a problem.
The error I am getting is an exception when I try to execute the query, I get the message:
MySql.Data.MySqlClient.MySqlException (0x80004005): You have an error
in your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near 'IF NOT EXISTS(price
VARCHAR, time VARCHAR)' at line 1
In debug mode, the immediate window shows my command string as:
CREATE TABLE ticks_14_11_2016 IF NOT EXISTS(price VARCHAR, time
VARCHAR)
From the examples I have seen, this should be the proper syntax. I am not worried about constraints and keys for the time being, I just need the query to execute...
Also, here is the C# code which I use to build the string and execute query:
string tableName = "ticks_" + getTodayString();
if (databaseClient.IsConnect()) {
string tableString = "CREATE TABLE " + tableName +
" IF NOT EXISTS" +
"(price VARCHAR, " +
"time VARCHAR)";
try
{
var command = databaseClient.Connection.CreateCommand();
command.CommandText = tableString;
command.ExecuteNonQuery();
} catch (Exception e)
{
Console.WriteLine(e);
}
}
The variable databaseClient has a member that is the MySQLConnection object
Also, my server version is: 5.6.28-76.1
You have the if not exists in the wrong place, and also, the varchar type needs a mandatory length argument.
A corrected version should be:
CREATE TABLE IF NOT EXISTS ticks_XXXXX (price VARCHAR(10), time VARCHAR(10));
Change the length to whatever is appropriate for you.
For more information see the reference manual.
You have other ways also to check whether table exists in database or not.
IF OBJECT_ID(N'dbo.ticks_14_11_2016', N'U') IS NOT NULL
BEGIN
------Exists
END
IF EXISTS(SELECT 1 FROM sys.Objects WHERE Object_id =
OBJECT_ID(N'dbo.ticks_14_11_2016') AND Type = N'U')
BEGIN
------Exists
END
IF EXISTS(SELECT 1 FROM sys.Tables WHERE Name = N'ticks_14_11_2016 ' AND Type = N'U')
BEGIN
----Exists
END
IF OBJECT_ID('ticks_14_11_2016') IS NOT NULL
BEGIN
-----Exists
END
Use your logic accordingly
I have a package that allows me to create a table from the output of a stored procedure REF_CURSOR variable. I use this with DevPress XPO Source in order to return large results to my client application.
I used to create a solid table, add a key, index it and return the new table name to the client which is provided to the XPO source, and it is working. However, using solid tables is not the best solution, so I started using a GTT.
If I execute the package in TOAD, my data is preserved, but if I execute the command from C#, there is no data in my table right after execution. The connection is not closed yet, so I am not 100% sure why my data is not there.
Is there something in the Connection Context that I can set to make sure that all executions happen in the same session? There is an execute immediate statement to populate the table, and I think that TOAD might use the same context when a I execute the package.
Here is some of my code:
FUNCTION Build_Table_from_Cursor(REF_CURSOR SYS_REFCURSOR, ID NUMBER, AddKeyField CHAR) RETURN VARCHAR2 AS
QueryCursor SYS_REFCURSOR;
CursorNumber NUMBER;
p_tablename varchar2(30);
pk_name varchar2(30);
BEGIN
QueryCursor := REF_CURSOR;
CursorNumber := DBMS_SQL.TO_CURSOR_NUMBER(QueryCursor);
p_tablename := 'TEMPTABLE';
UTIL.create_table_from_cursor(CursorNumber, p_tablename); --This creates the GTT with all the columns
Execute immediate 'TRUNCATE TABLE ' || p_tablename; --To Add the key this must be done otherwise there is an error
pk_name := substr(p_tablename, INSTR(p_tablename, '.') + 1);
IF(AddKeyField = 'Y') THEN --Sometimes the Key field already exists
EXECUTE IMMEDIATE 'ALTER TABLE ' || p_tablename || ' ADD (KEY_FIELD_ NUMBER)';
END IF;
EXECUTE IMMEDIATE 'CREATE UNIQUE INDEX ' || p_tablename || 'KEY_INDEX ON ' || p_tablename || ' (KEY_FIELD_)';
EXECUTE IMMEDIATE 'ALTER TABLE ' || p_tablename || ' ADD CONSTRAINT pk_' || pk_name || ' PRIMARY KEY( KEY_FIELD_ )';
QueryCursor := DBMS_SQL.TO_REFCURSOR(CursorNumber);
PDS.UTIL.POPULATE_TABLE_FROM_CURSOR(QueryCursor, p_tablename, 1000); --This populates the table
EXECUTE IMMEDIATE 'UPDATE ' || p_tablename || ' SET KEY_FIELD_ = ROWNUM';
COMMIT;
return p_tablename;
END Build_Table_from_Cursor;
This works perfectly when I execute in TOAD.
When I run this
using (var conn = factory.CreateConnection(Dal.ConnectionStrings[connectionString].ConnectionString))
{
conn.Open();
using (var cmd = factory.CreateCommand(CommandType.StoredProcedure, storedProcedureName))
{
var storedProcedureRow = commandExecuteDataSet.StoredProcedure[0];
foreach (var parametersRow in commandExecuteDataSet.Parameters)
{
cmd.Parameters.Add(CustomDbProviderFactory.CreateParameter(parametersRow.Name, parametersRow.Value ?? "", GetDBTypeFromString(parametersRow.OracleDbType)));
}
cmd.Parameters.Add(CustomDbProviderFactory.CreateParameter(storedProcedureRow.RefCursorName, DbType.Object, ParameterDirection.Output, true));
cmd.ExecuteNonQuery();
var refCursor = cmd.Parameters[storedProcedureRow.RefCursorName].Value;
cmd.Parameters.Clear();
cmd.CommandText = "SERVERMODE_UTIL.Build_Table_from_Cursor";
cmd.Parameters.Add(CustomDbProviderFactory.CreateParameter("REF_CURSOR", refCursor));
cmd.Parameters.Add(CustomDbProviderFactory.CreateParameter("ID", biID));
cmd.Parameters.Add(CustomDbProviderFactory.CreateParameter("AddKeyField", "Y"));
cmd.Parameters.Add(CustomDbProviderFactory.CreateParameter("p_tablename", DbType.String, ParameterDirection.ReturnValue));
cmd.ExecuteNonQuery();
var tempTableName = cmd.Parameters["p_tablename"].Value.ToString();
tempTableName = tempTableName.Substring(tempTableName.IndexOf(".") + 1);
}
}
As part of a larger package, this is the code that is executed to create the GTT
l_statement := 'CREATE GLOBAL TEMPORARY TABLE ' || l_tablename || ' (' || CHR(13) || CHR(10) || l_statement || CHR(13) || CHR(10) || ') ON COMMIT PRESERVE ROWS';
execute immediate l_statement;
I have a situation in such a way that variable value has to be returned based on multiple condition. Initially it is set to passed parameter value. Below is just an algo,
create proc checkFlag (ppleid int, pflag bit, loginid int)
begin
declare vflag bit;
declare vhasRights bit;
declare vIsLocked bit;
declare vlockedUid int;
set vflag=pflag; // assign as passed param
if (vflag = 0)
then
select id, name,vflag as 'IsEditable'
from peopletable
where id=ppleid;
else
-- algo to check whether loginid has rights to edit
if (vhasrights = 1)
then
-- sql statements to check whether record is not write locked into vIsAlreadyLocked
if (vIsAlreadyLocked = 1)
then
--- sql statements to check whether locked user is login user in vlockedUid
if (vlockedUid = ploginid)
then
set vflag =1;
else
set vflag=0;
end if;
else
set vflag=0;
end if;
select id, name,vflag as 'IsEditable'
from peopletable
where id=ppleid;
end if;
end;
A. Sp. call withing sqleditor
call checkflag(values.....); returns appropriate value
B. Sp. call from c#
internal void ReadPpl()
{
bool initFlag=<flag based on client end condition>;
. code for creating connection and command of type storedproc
.
MysqlParameter prm=new parameter("pflag", initFlag);
.
.
. cmd.parameter(prm)
using (mysqldatareader rdr=cmd.ExecuteReader())
{
if (rdr.HasRows())
{
rdr.Read();
**bool IsWritable=Convert.ToBoolean(Convert.ToInt32(rdr["IsEditable"])); // This is the line that returns value as passed parameter but not based on conditions in db and params **
}
Any help will be appreciated.
Thanks in advance.
Got problem solved.
call proc(#flag...)
resulted #flag other than 0. So the else condition was executed.
But
MySqlParameter prm=new(....,Value=initValue);
considered intiValue as 0 hence, the only the first if condition in stored proc. was executed every time.
I'm trying to create a new stored procedure programatically using the following code:
using (MySqlConnection conn = new MySqlConnection(connectionString))
{
conn.Open();
using (MySqlTransaction trans = conn.BeginTransaction())
{
using (MySqlCommand command = conn.CreateCommand())
{
command.CommandText = query;
command.ExecuteNonQuery();
}
trans.Commit();
}
}
And the following text as the create statement copied from Mysql workbench:
static string query = #"
delimiter $$
CREATE PROCEDURE `GetParentIds`(IN `tempTableName` VARCHAR(255), IN `id` int)
BEGIN
DECLARE parId INT;
DECLARE curId INT;
DROP TEMPORARY TABLE IF EXISTS tempTableName;
CREATE TEMPORARY TABLE tempTableName (
node_id INT NOT NULL PRIMARY KEY
);
set curId := id;
get_parents_loop: LOOP
set parId := null;
set parId = (select ParentID from {TableName} where ID = curId);
IF parId is NULL THEN
LEAVE get_parents_loop;
END IF;
INSERT INTO tempTableName(node_id) Values (parId);
set curId := parId;
END LOOP get_parents_loop;
SELECT *
FROM tempTableName;
END$$";
This procedure is passed an ID of an object which has a parentID and it gets all of the parentIDs of all the parents of the given object and returns them. The problem comes when I try to run it and I get the following message:
"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'delimiter $$
CREATE PROCEDURE GetParentIds(IN tempTableName VARCHAR(255),' at line 1"
Any and all Ideas are welcome!
* EDIT **
Thanks to all the answers below, this is what finally worked:
CREATE PROCEDURE GetParentIds(IN tempTableName VARCHAR(255), IN id int)
BEGIN
DECLARE parId INT;
DECLARE curId INT;
DROP TEMPORARY TABLE IF EXISTS tempTableName;
CREATE TEMPORARY TABLE tempTableName (node_id INT NOT NULL PRIMARY KEY );
set curId := id;
get_parents_loop: LOOP
set parId := null;
set parId = (select ParentID from TDOs where TDOID = curId);
IF parId is NULL THEN
LEAVE get_parents_loop;
END IF;
INSERT INTO tempTableName(node_id) Values (parId);
set curId := parId;
END LOOP get_parents_loop;
SELECT *
FROM tempTableName;
END;
Remove DELIMITER $$ at the beginning and $$ after last END
DELIMITER is a mysql client command that enables you to change its statement terminator temporarily while you define a stored routine.
If you are defining a stored routine from within a programming interface that does not use the semicolon as a statement terminator, semicolons within stored routine definitions do not present any special issues.