We Keep Coding

Reading file with Apache Arrow ArrowFileReader in .net

I am trying to read the content of an arrow file but I was not able to find the functions to get the actual data from it. I am not able to find any useful example to read the data too. For example here.
The code example for writing and reading in C#:
// Write
var recordBatch = new Apache.Arrow.RecordBatch.Builder(memoryAllocator)
.Append("Column A", false, col => col.Int32(array => array.AppendRange(Enumerable.Range(5, 15))))
.Build();
using (var stream = File.OpenWrite(filePath))
using (var writer = new Apache.Arrow.Ipc.ArrowFileWriter(stream, recordBatch.Schema, true))
{
await writer.WriteRecordBatchAsync(recordBatch);
await writer.WriteEndAsync();
}
// Read
var reader = Apache.Arrow.Ipc.ArrowFileReader.FromFile(filePath);
var readBatch = await reader.ReadNextRecordBatchAsync();
var col = readBatch.Column(0);
By debugging the code, I can see the values in the col Values property but I have no way of accessing this information in the code.
Am I missing anything or is there a different approach to read the data?

The Apache.Arrow package does not do any compute today. It will read in the file and you will have access to the raw buffers of data. This is sufficient for a number of intermediary tasks (e.g. services that shuttle data to and from or aggregate data files). So if you want to do a lot of operations on the data you may want some kind of dataframe library.
One such library is the Microsoft.Data.Analysis library which has added a DataFrame type which can be created from an Arrow RecordBatch. There is some explanation and examples of the library in this blog post.
I haven't worked with that library much but I was able to put together a short example of reading an Arrow file and printing the data:
using System;
using System.Diagnostics;
using System.IO;
using System.Threading.Tasks;
using Apache.Arrow.Ipc;
using Microsoft.Data.Analysis;
namespace DataframeExperiment
{
class Program
{
static async Task AsyncMain()
{
using (var stream = File.OpenRead("/tmp/test.arrow"))
using (var reader = new ArrowFileReader(stream))
{
var recordBatch = await reader.ReadNextRecordBatchAsync();
Console.WriteLine("Read record batch with {0} column(s)", recordBatch.ColumnCount);
var dataframe = DataFrame.FromArrowRecordBatch(recordBatch);
var columnX = dataframe["x"];
foreach (var value in columnX)
{
Console.WriteLine(value);
}
}
}
static void Main(string[] args)
{
AsyncMain().Wait();
}
}
}
I created the test file with a small python script:
import pyarrow as pa
import pyarrow.ipc as ipc
tab = pa.Table.from_pydict({'x': [1, 2, 3], 'y': ['x', 'y', 'z']})
with ipc.RecordBatchFileWriter('/tmp/test.arrow', schema=tab.schema) as writer:
writer.write_table(tab)
You could presumably also create the test file using C# with Apache.Arrow's array builders.
Update (Using Apache.Arrow directly)
On the other hand, if you want to use Apache.Arrow directly, and still get access to the data, then you can use typed arrays (e.g. Int32Array, Int64Array). You will first need to determine the type of your array somehow (either through prior knowledge of the schema or as / is style checks or pattern matching).
Here is an example using Apache.Arrow alone:
using System;
using System.IO;
using System.Threading.Tasks;
using Apache.Arrow;
using Apache.Arrow.Ipc;
namespace ArrayValuesExperiment
{
class Program
{
static async Task AsyncMain()
{
using (var stream = File.OpenRead("/tmp/test.arrow"))
using (var reader = new ArrowFileReader(stream))
{
var recordBatch = await reader.ReadNextRecordBatchAsync();
// Here I am relying on the fact that I know column
// 0 is an int64 array.
var columnX = (Int64Array) recordBatch.Column(0);
for (int i = 0; i < columnX.Values.Length; i++)
{
Console.WriteLine(columnX.Values[i]);
}
}
}
static void Main(string[] args)
{
AsyncMain().Wait();
}
}
}

Adding to the second approach proposed by Pace, an utility function like below can be used to get the values
private static dynamic GetArrayData(IArrowArray array)
{
return array switch
{
Int32Array int32array =>int32array.Values.ToArray(),
Int16Array int16array => int16array.Values.ToArray(),
StringArray stringArray => stringArray.Values.ToArray(),
FloatArray floatArray => floatArray.Values.ToArray(),
Int64Array int64Array => int64Array.Values.ToArray(),
DoubleArray doubleArray => doubleArray.Values.ToArray(),
Time32Array time32Array => time32Array.Values.ToArray(),
Time64Array time64Array => time64Array.Values.ToArray(),
BooleanArray booleanArray => booleanArray.Values.ToArray(),
Date32Array date32Array => date32Array.Values.ToArray(),
Date64Array date64Array => date64Array.Values.ToArray(),
Int8Array int8Array => int8Array.Values.ToArray(),
UInt16Array uint6Array => uint6Array.Values.ToArray(),
UInt8Array uInt8Array => uInt8Array.Values.ToArray(),
UInt64Array uInt64Array => uInt64Array.Values.ToArray(),
_ => throw new NotImplementedException(),
};
}
then iterate over the recordBatch as
object[,] results = new Object[recordBatch.Length, recordBatch.ColumnCount];
var col = 0;
foreach (var array in recordBatch.Arrays)
{
var row = 0;
foreach (var data in GetArrayData(array))
{
results[row++, col] = data;
}
col++;
}
return results;
Worth noting however that StringArrays return Bytes so you need to convert to back to string for example using
System.Text.Encoding.Unicode.GetString(stringArray.Values)

FlatFile library, delimited layout, wrong parsing when multiple fields are empty at the end of the row

We use in some of our applications the FlatFile library (https://github.com/forcewake/FlatFile) to parse some files delimited with separator (";"), since a lot of time without problems.
We faced yesterday a problem receiving files having multiple fields empty at the end of the row.
I replicated the problem with short console application to show and permit you to verify in a simple way:
using FlatFile.Delimited;
using FlatFile.Delimited.Attributes;
using FlatFile.Delimited.Implementation;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace FlatFileTester
{
class Program
{
static void Main(string[] args)
{
var layout = GetLayout();
var factory = new DelimitedFileEngineFactory();
using (MemoryStream ms = new MemoryStream())
using (FileStream file = new FileStream(#"D:\shared\dotnet\FlatFileTester\test.csv", FileMode.Open, FileAccess.Read))
{
byte[] bytes = new byte[file.Length];
file.Read(bytes, 0, (int)file.Length);
ms.Write(bytes, 0, (int)file.Length);
var flatFile = factory.GetEngine(layout);
ms.Position = 0;
List<TestObject> records = flatFile.Read<TestObject>(ms).ToList();
foreach(var record in records)
{
Console.WriteLine(string.Format("Id=\"{0}\" - DescriptionA=\"{1}\" - DescriptionB=\"{2}\" - DescriptionC=\"{3}\"", record.Id, record.DescriptionA, record.DescriptionB, record.DescriptionC));
}
}
Console.ReadLine();
}
public static IDelimitedLayout<TestObject> GetLayout()
{
IDelimitedLayout<TestObject> layout = new DelimitedLayout<TestObject>()
.WithDelimiter(";")
.WithQuote("\"")
.WithMember(x => x.Id)
.WithMember(x => x.DescriptionA)
.WithMember(x => x.DescriptionB)
.WithMember(x => x.DescriptionC)
;
return layout;
}
}
[DelimitedFile(Delimiter = ";", Quotes = "\"")]
public class TestObject
{
[DelimitedField(1)]
public int Id { get; set; }
[DelimitedField(2)]
public string DescriptionA { get; set; }
[DelimitedField(3)]
public string DescriptionB { get; set; }
[DelimitedField(4)]
public string DescriptionC { get; set; }
}
}
This is an example of file:
1;desc1;desc1;desc1
2;desc2;desc2;desc2
3;desc3;;desc3
4;desc4;desc4;
5;desc5;;
So the first 4 rows are parsed as expected:
All fields with values in the first and second row
empty string for third field of third row
empty string for fouth field of fourth row
in the fifth row we expect empty string on third and fourth field, like this:
Id=5
DescriptionA="desc5"
DescriptionB=""
DescriptionC=""
instead we receive this:
Id=5
DescriptionA="desc5"
DescriptionB=";" // --> THE SEPARATOR!!!
DescriptionC=""
We can't understand if is a problem of configuration, bug of the library, or some other problem in the code...
Anyone have some similar experiences with this library, or can note some problem in the code above not linked with the library but causing the error...?

I took a look and debug the source code of the open source library: https://github.com/forcewake/FlatFile.
It seems there's a problem, in particular in this case, in witch there are 2 empty fields, at the end of a row, the bug take effects on the field before the last of the row.
I opened an issue for this libray, hoping some contributor of the library could invest some time to investigate, and, if it is so, to fix: https://github.com/forcewake/FlatFile/issues/80
For now we decided to fix the wrong values of the list, something like:
string separator = ",";
//...
//...
//...
records.ForEach(x => {
x.DescriptionC = x.DescriptionC.Replace(separator, "");
});
For our case, anyway, it make not sense to have a character corresponding to the separator as value of that field...
...even if it would be better to have bug fixing of the library

To do searching in an array with some keyword

I have an array which is like
books={'java 350','Photoshop 225','php 210','JavaScript 80','python 180','jquery 250'}
my input for search as be like "ph2" it retrieve both Photoshop 225,php 210 in drop-down menu what is the exact string function to do this task or any set codes available to do this task.
I'm using some build in function like
if (array.Any(keyword.Contains))
and
if (array.Contains(keyword))
it's doesn't help what exactly i want any one pls help me to solve this thanks in advance.....

More flexible approach:
using Microsoft.VisualBasic; // add reference
using Microsoft.VisualBasic.CompilerServices;
using System.Linq;
namespace ConsoleApplication
{
class Program
{
static void Main (string[] args)
{
string[] books = { "java 350", "Photoshop 225", "php 210", "JavaScript 80", "python 180", "jquery 250" };
string input = "*" + string.Join ("*", "ph2".ToArray ()) + "*"; // will be *p*h*2*
var matches = books.Where (x => LikeOperator.LikeString (x, input, CompareMethod.Text));
}
}
}

Ordering writeline outputs from text file using C# in console

I wonder if you could help me at all. Essentially my program has to scan through a text file and then print the lines. Each line that is printed must be alphabetized also, if possible. I could do with being able to point at any file through cmd rather than automatically pointing it at a specific file and in a specific location.
I have this so far as I wanted to get things working in a basic form.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace Program
{
class Program
{
static void Main(string[] args)
{
String line;
try
{
//We Have to pass the file path and packages.txt filename to the StreamReader constructor
StreamReader sr = new StreamReader("D:\\Users\\James\\Desktop\\packages.txt");
//Instruction to read the first line of text
line = sr.ReadLine();
//Further Instruction is to to read until you reach end of file
while (line != null)
{
//Instruction to write the line to console window
Console.WriteLine(line);
//The read the next line
line = sr.ReadLine();
}
//Finally close the file
sr.Close();
Console.ReadLine();
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message);
}
finally
{
Console.WriteLine("Executing finally block.");
}
}
}
}
I hope you guys can help me, I am very rusty!
My thoughts were to convert the string into an char array? then modify and sort using array.sort method.
OK guys. On your advice I have made a few changes. I get an exception thrown at me now as we are trying to get it to accept an argument in order for us to point it at any text file, not a specific one.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace Program
{
class Program
{
static void Main (params string[] args)
{
string PathToFile = args[1];
string TargetPackages = args[2];
try
{
string[] textLines = File.ReadAllLines(PathToFile);
List<string> results = new List<string>();
foreach (string line in textLines)
{
if (line.Contains(TargetPackages))
{
results.Add(line);
}
Console.WriteLine(results);
}
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message);
}
finally
{
Console.WriteLine("Executing finally block.");
}
}
}
}

If you just want to sort by the first word and output, you need to read all the lines into memory (I hope your file isn't too large), sort the lines, and then write them back out.
There are many ways to do all that. The File class has some helper functions that make reading and writing text files line-by-line very simple, and LINQ's OrderBy method makes quick work of sorting things.
File.WriteAllLines(
outputFileName,
File.ReadLines(inputFileName).OrderBy(line => line));
See File.WriteAllLines and File.ReadLines for information on how they work.
If you want to load each line, sort the first word, and then re-output the line:
File.WriteAllLines(
outputFileName,
File.ReadLines(inputFileName)
.Select(line =>
{
var splits = line.Split(new [] {' '}};
var firstWord = new string(splits[0].OrderBy(c => c));
var newLine = firstWord + line.Substring(firstWord.Length);
return newLine;
}));
Note that this loads and processes one line at a time, so you don't have to hold the entire file in memory.

You should be better off by reading all lines at once and then looking at each line separately, like this:
List<string> allLines = System.IO.File.ReadLines(pathToFile).ToList();
allLines = allLines.OrderBy(line => line).ToList(); //this orders alphabetically all your lines
foreach (string line in allLines)
{
Console.WriteLine(line);
}
This will print all the lines in the file ordered alphabetically.
You can also parametrize the path by using the args parameter you receive when opening the application:
CMD> pathToYourExe.exe path1 path2 path3
You can mock this by using the DEBUG arguments in the project's Debug menu

How to sort DBML objects alphabetically?

I've got a DBML file in my project with all my LinqToSql objects. Initially I imported them from the DB, and all was well. Now as my DB has been growing, I've been adding the new tables to the diagram in the O/R Designer, but they always get appended to the end of the XML. This is a bit of a pain, because when I'm defining foreign keys, it always lists the available tables in the order in which they appear in the XML.
Any ideas how to sort the XML table declarations alphabetically according to the table name?

I know this is old, but I also want to sort the tables and functions in my DBML to make it more manageable in Git. The following console application code seems to work pretty well. You can drag and drop a DBML file onto the exe, or you could set up a bat file or build event in your project(s).
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Xml.Linq;
namespace DbmlSorter
{
class Program
{
static void Main(string[] args)
{
if (args.Length == 0)
return;
var fileName = args[0];
try
{
if (!File.Exists(fileName))
return;
SortElements(fileName);
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
Console.WriteLine();
Console.WriteLine("Press any key to exit...");
Console.ReadKey();
}
private static void SortElements(string fileName)
{
var root = XElement.Load(fileName);
var connections = new SortedDictionary<string, XElement>();
var tables = new SortedDictionary<string, XElement>();
var functions = new SortedDictionary<string, XElement>();
var others = new SortedDictionary<string, XElement>();
foreach (var element in root.Elements())
{
var key = element.ToString();
if (key.StartsWith("<Connection"))
connections.Add(key, element);
else if (key.StartsWith("<Table"))
tables.Add(key, element);
else if (key.StartsWith("<Function"))
functions.Add(key, element);
else
others.Add(key, element);
}
root.RemoveNodes();
foreach (var pair in connections)
{
root.Add(pair.Value);
Console.WriteLine(pair.Key);
}
foreach (var pair in tables)
{
root.Add(pair.Value);
Console.WriteLine(pair.Key);
}
foreach (var pair in functions)
{
root.Add(pair.Value);
Console.WriteLine(pair.Key);
}
foreach (var pair in others)
{
root.Add(pair.Value);
Console.WriteLine(pair.Key);
}
root.Save(fileName);
}
}
}

A possible solution is to write a small application that reads in the XML, sorts it to your liking and outputs the updated version.