I have a list of keys that I want to get from Redis. I wrote a function like this but it returns everything:
public IOrderedEnumerable<Fields> GetValues(List<string> symbols)
{
var retVal = new List<Fields>();
var patternStr = "[";
int count = 0;
foreach (var symbol in symbols)
{
patternStr += (symbol);
if (++count != symbols.Count)
{
patternStr += ", ";
}
}
patternStr += "]*";
foreach (var ep in redis.GetEndPoints())
{
var server = redis.GetServer(ep);
var keysList = server.Keys(database: 0, pattern: patternStr).ToList();
var keys = keysList.ToArray();
Console.WriteLine("Number of Symbols in this range{0} ", keys.Length);
foreach (var rk in keys)
{
var myValTask = db.StringGetAsync(rk.ToString());
var myVal = myValTask.Result;
var jsonStr = myVal.ToString();
...
}
...
}
...
The section of code I believe is the problem. I just want Redis to return the subset of keys, and I am building a pattern by seperating them by "," :
var patternStr = "[";
int count = 0;
foreach (var symbol in symbols)
{
patternStr += (symbol);
if (++count != symbols.Count)
{
patternStr += ", ";
}
}
patternStr += "]*";
I know I can get all the keys, then filter them once I get them, but I want to avoid the network thrashing...
Edit 1
BTW, the keys look like this:
127.0.0.1:6379> keys *
1) "BBWI_2022-08-19"
2) "ABBV_2023-01-20"
3) "ZTS_2022-10-21"
You can use Scan command on redis server like this:
public async Task<List<string>> ScanKeysAsync(string match, string count)
{
var schemas=new List<string>();
int nextCursor = 0;
do
{
RedisResult redisResult =await _redisServer.ExecuteAsync("SCAN", nextCursor.ToString(), "MATCH", match, "COUNT", count);
var innerResult = (RedisResult[])redisResult;
nextCursor = int.Parse((string)innerResult[0]);
List<string> resultLines = ((string[])innerResult[1]).ToList();
schemas.AddRange(resultLines);
}
while (nextCursor != 0);
return schemas;
}
and in your case would be something like this:
var keys=await ScanKeysAsync("BBWI*",10);//return max 10 occurance of pattern
But I recommend to use Scan in very special scenarios because it acts as a cursor and will iterate between all keys in redis to find match and also please read this https://redis.io/commands/scan
Test
var redis = scope.ServiceProvider.GetRequiredService<IDatabase>();
await redis.StringSetAsync("BBWI_2022-08-19", "test1",TimeSpan.FromMinutes(5));
await redis.StringSetAsync("BBWI_20fd22-08-19", "test2", TimeSpan.FromMinutes(5));
await redis.StringSetAsync("ABBV_2023-08-19", "test3", TimeSpan.FromMinutes(5));
var foundKeys = await ScanKeysAsync("BBW*", "10");
//BBWI_2022-08-19
//BBWI_20fd22-08-19
If I understand correctly, you have a set of discrete keys that you want to fetch in a single batch. If so, you can fetch all items (within reason, say < 1000 at a time) by passing a RedisKey[] array to StringGetAsync:
var keys = symbols.Select(symbol => (RedisKey)symbol).ToArray();
var values = await db.StringGetAsync(keys);
// .. use values
Related
I need to pull a bunch of key value pairs based on a predefined pattern from a string. An example of what I would need is this:
Pattern: {value1}-{value2}
String: Example-String
Result KVP:
{ Key: value1, value: Example },
{ Key: value2, value: String }
The catch is that the pattern could be pretty much anything (although the values I'd need to extract would always be surrounded in curly brackets), ie:
Pattern: {test1}\{test2}={value}
String: Example\Value=Result
Result KVP:
{ Key: test1, value: Example },
{ Key: test2, value: Value },
{ Key: value, value: Result }
What I have done so far isn't quite working and I'm quite certain that there has to be a more elegant way of doing this as opposed to my solution anyway so I thought I'd see if anyone here would have a good idea.
EDIT:
Here is essentially what I have so far (it's working, but IMO it's really ugly):
public List<KeyValuePair<string, string>> Example(string pattern, string input)
{
var values = new List<KeyValuePair<string, string>>();
var r1 = Regex.Matches(input, #"(\{[A-Z,a-z]*\})");
string newregex = string.Empty;
foreach (Match item in r1)
{
newregex = newregex.Replace(item.Value, "(.*?)"); //updates regex so that it adds this as a group for use later, ie: "{item1}-{item2}" will become "(.*?)-{item2}"
string field = item.Value.Substring(1, item.Value.Length - 2); // {test1} will return "test1"
values.Add(new KeyValuePair<string, string>(field, string.Empty));
}
newregex = $"{newregex}\\z"; // ensures that it matches to end of input
var r2 = Regex.Match(input, newregex);
// KVP index (used below)
int val = 0;
foreach (Group g in r2.Groups)
{
if (g.Value == input)
continue; // first group will be equal to input, ignore
values[val] = new KeyValuePair<string, string>(values[val].Key, g.Value); // update KVP at index with new KVP with the value
val++;
}
return values;
}
Unfortunately I don't know regular expressions very well, but one way to solve this is to walk through each character of the pattern string and create a list of keys and delimeters, after which we can walk through the search string, and find the index of each delimeter to get the current value, and then add a new KeyValuePair to a list.
Here's a rough sample that assumes good input:
public static List<KeyValuePair<string, string>> GetKVPs(string pattern, string search)
{
var results = new List<KeyValuePair<string, string>>();
var keys = new List<string>();
var delimeters = new List<string>();
var currentKey = string.Empty;
var currentDelimeter = string.Empty;
var processingKey = false;
// Populate our lists of Keys and Delimeters
foreach (var chr in pattern)
{
switch (chr)
{
case '}':
{
if (currentKey.Length > 0)
{
keys.Add(currentKey);
currentKey = string.Empty;
}
processingKey = false;
break;
}
case '{':
{
if (currentDelimeter.Length > 0)
{
delimeters.Add(currentDelimeter);
currentDelimeter = string.Empty;
}
processingKey = true;
break;
}
default:
{
if (processingKey)
{
currentKey += chr;
}
else
{
currentDelimeter += chr;
}
break;
}
}
}
if (currentDelimeter.Length > 0) delimeters.Add(currentDelimeter);
var lastDelim = -1;
// Find our Values based on the delimeter positions in the search string
for (int i = 0; i < delimeters.Count; i++)
{
var delimIndex = search.IndexOf(delimeters[i], lastDelim + 1);
if (delimIndex > -1)
{
var value = search.Substring(lastDelim + 1, delimIndex - lastDelim - 1);
results.Add(new KeyValuePair<string, string>(keys[i], value));
lastDelim = delimIndex + delimeters[i].Length - 1;
}
}
// Add the item after the final delimeter if it exists:
if (lastDelim > -1 && lastDelim < search.Length - 1)
{
results.Add(new KeyValuePair<string, string>(keys.Last(),
search.Substring(lastDelim + 1)));
}
return results;
}
And an example of it in action:
public static void Main(string[] args)
{
var results = GetKVPs(
"{greeting}, {recipient}, this is {sender}.",
"Hello, Dolly, this is Louis.");
foreach (var kvp in results)
{
Console.WriteLine($"{kvp.Key} = {kvp.Value}");
}
GetKeyFromUser("\nDone! Press any key to exit...");
}
Output
Here is my string:
www.stackoverflow.com/questions/ask/user/end
I split it with / into a list of separated words:myString.Split('/').ToList()
Output:
www.stackoverflow.com
questions
ask
user
end
and I need to rejoin the string to get a list like this:
www.stackoverflow.com
www.stackoverflow.com/questions
www.stackoverflow.com/questions/ask
www.stackoverflow.com/questions/ask/user
www.stackoverflow.com/questions/ask/user/end
I think about linq aggregate but it seems it is not suitable here. I want to do this all through linq
You can try iterating over it with foreach
var splitted = "www.stackoverflow.com/questions/ask/user/end".Split('/').ToList();
string full = "";
foreach (var part in splitted)
{
full=$"{full}/{part}"
Console.Write(full);
}
Or use linq:
var splitted = "www.stackoverflow.com/questions/ask/user/end".Split('/').ToList();
var list = splitted.Select((x, i) => string.Join("/", a.Take(i + 1)));
Linq with side effect:
string prior = null;
var result = "www.stackoverflow.com/questions/ask/user/end"
.Split('/')
.Select(item => prior == null
? prior = item
: prior += "/" + item)
.ToList();
Let's print it out
Console.WriteLine(string.Join(Environment.NewLine, result));
Outcome:
www.stackoverflow.com
www.stackoverflow.com/questions
www.stackoverflow.com/questions/ask
www.stackoverflow.com/questions/ask/user
www.stackoverflow.com/questions/ask/user/end
Linq without side effects ;)
Enumerable.Aggregate can be used here if we use List<T> as a result.
var raw = "www.stackoverflow.com/questions/ask/user/end";
var actual =
raw.Split('/')
.Aggregate(new List<string>(),
(list, word) =>
{
var combined = list.Any() ? $"{list.Last()}/{word}" : word;
list.Add(combined);
return list;
});
without Linq write below code,
var str = "www.stackoverflow.com/questions/ask/user/end";
string[] full = str.Split('/');
string Result = string.Empty;
for (int i = 0; i < full.Length; i++)
{
Console.WriteLine(full[i]);
}
for (int i = 0; i < full.Length; i++)
{
if (i == 0)
{
Result = full[i];
}
else
{
Result += "/" + full[i];
}
Console.WriteLine(Result);
}
I am trying print documents by simplex/duplex and then by envelope type (pressure seal or regular)
I have Boolean fields for Simplex and for PressureSeal in my Record class.
All pressure seal are simplex, then there are regular simplex and duplex documents.
I can currently print the pressure seal documents separate from the regular simplex. I need to be able to create the regular duplex documents.
I have some lines commented out that caused all documents to be duplicated.
So, I am looking for something that works like so:
if (Simplex)
if (pressureseal)
create output file
else
create regular simplex output file
else
create duplex output file
Here is my existing code
#region Mark Records By Splits
//splits - 3,7,13
var splits = Properties.Settings.Default.Splits.Split(',');
Dictionary<int, int> splitRanges = new Dictionary<int, int>();
int lastSplit = 0;
foreach (var split in splits)
{
// Attempt to convert split into a integer and skip it if we can't.
int splitNum;
if (!int.TryParse(split, out splitNum))
continue;
splitRanges.Add(lastSplit, splitNum);
lastSplit = Math.Max(lastSplit, splitNum + 1);
}
// Assign record splits.
foreach (var range in splitRanges)
{
var recordsInRange = NoticeParser.records
.Where(x => x.Sheets >= range.Key && x.Sheets <= range.Value)
.ToList();
recordsInRange.ForEach(x => x.Split = string.Format("{0}-{1}", range.Key, range.Value));
}
var unassignedRecords = NoticeParser.records.Where(x => x.Sheets >= lastSplit).ToList();
unassignedRecords.ForEach(x => x.Split = string.Format("{0}up", lastSplit));
#endregion
#region Sort out Pressure Seal records
var recordsGroupedByPressureSeal = NoticeParser.records
.GroupBy(x=>x.PressureSeal);
//var recordsGroupedBySimplex = NoticeParser.records.GroupBy(x => x.Simplex);
#endregion
int fCount = 0;
int nsCount = 0;
//foreach (var simdupGroup in recordsGroupedBySimplex)
//{
// var recordsGroupedBySimDup = simdupGroup.GroupBy(x => x.Split).OrderBy(x => x.Key).ToDictionary(x => x.Key, x => x.ToList());
foreach (var pressureGroup in recordsGroupedByPressureSeal)
{
var recordsGroupedBySplit = pressureGroup.GroupBy(x => x.Split).OrderBy(x => x.Key).ToDictionary(x => x.Key, x => x.ToList());
foreach (var recordsInSplit in recordsGroupedBySplit.Values)
{
string processingExecutable = Path.Combine(Properties.Settings.Default.RootFolder, Properties.Settings.Default.ProcessingExecutable);
string toProcessingFile = string.Format(Properties.Settings.Default.OutputFolder + "{0}_" + "toBCC.txt", fCount);
string fromProcessingFile = string.Format(Properties.Settings.Default.OutputFolder + "IBC_LN_Sort_FromBCC.txt");
// If a sortation executable is specified, run it.
if (recordsInSplit.Count >= Properties.Settings.Default.MinimumSortationCount &&
File.Exists(processingExecutable))
{
// log.Info("Sorting records...");
var processedRecords = recordsInSplit.ProcessAddresses<Record, RecordMap>(
processingExecutable,
toProcessingFile,
fromProcessingFile);
// Update records with the sortation fields.
recordsInSplit.UpdateAddresses(processedRecords);
}
else
{
toProcessingFile = string.Format(Properties.Settings.Default.OutputFolder + "{0}_no_sort_toBCC.txt", nsCount);
fromProcessingFile = string.Format(Properties.Settings.Default.OutputFolder + "IBC_LN_NoSort_FromBCC.txt");
//var processedRecords = recordsInSplit.ProcessAddresses<Record, RecordMap>(
// processingExecutable,
// toProcessingFile,
// fromProcessingFile);
// Update records with the sortation fields.
// recordsInSplit.UpdateAddresses(processedRecords);
// If not sorted, provide our own sequence number.
int sequence = 1;
recordsInSplit.ForEach(x => x.SequenceNumber = sequence++);
recordsInSplit.ForEach(x => x.TrayNumber = 1);
nsCount++;
}
fCount++;
}
}
//}
NoticeWriter noticeWriter = new NoticeWriter(noticeParser.reader);
#region Print by PressureSeal or Regular
//foreach (var simdupGroup in recordsGroupedBySimplex)
//{
// string printType = null;
// if (simdupGroup.Key)
// printType = "Simplex";
// else
// printType = "Duplex";
foreach (var splitGroup in recordsGroupedByPressureSeal)
{
string envType = ""; // envelope type
if (splitGroup.Key)
envType = "PressureSeal";
else
envType = "Regular";
var recordsGroupedBySplit = splitGroup.GroupBy(x => x.Split).OrderBy(x => x.Key).ToDictionary(x => x.Key, x => x.ToList());
foreach (var recordsInSplit in recordsGroupedBySplit)
{
string outputName = string.Format("IBC_Daily_Notices_{0}_{1}",envType, /*printType,*/ recordsInSplit.Key);
noticeWriter.WriteOutputFiles(Properties.Settings.Default.OutputFolder, outputName, recordsInSplit.Value, Properties.Settings.Default.RecordsPerBatch);
}
}
//}
#endregion
What I'm trying to do is the following - I have hundreds of log files, that I need to search through and do some counting. The basic idea is this, take a .txt file, read every line, if search item 1 is found, increment the counter for search item 1, if search item 2 is found, increment the counter for search item 2 and so on.. For example, if the file contained something like...
a b c
d e f
g h i
j k h
And If I specified the searchables to be e & h, the output should say
e : 1
h : 2
The number of search terms is expandable, basically the user can give either 1 search number or 10, so i'm not sure how I can implement n number of counters based on the number of searchables.
The below is what I have so far, its just a basic approach to see what works and what doesnt... Right now, it only keeps the count for one of the search terms. At the moment, I am writing the results to the console to just test, ultimately, It will be written to a .txt or .xlsx. any help will be appreciated!
string line;
int Scounter = 0;
int Mcounter = 0;
List<string> searchables = new List<string>();
private void search_Log(string p)
{
searchables.Add("S");
searchables.Add("M");
StreamReader reader = new StreamReader(p);
while ((line = reader.ReadLine()) != null)
{
for (int i = 0; i < searchables.Count(); i++)
{
if (line.Contains(searchables[i]))
{
Scounter++;
}
}
}
reader.Close();
Console.WriteLine("# of S: " + Scounter);
Console.WriteLine("# of M: " + Mcounter);
}
A common approach to this is to use a Dictionary<string, int> to track the values and counts:
// Initialise the dictionary:
Dictionary<string, int> counters = new Dictionary<string, int>();
Then later:
if (line.Contains(searchables[i]))
{
if (counters.ContainsKey(searchables[i]))
{
counters[searchables[i]] ++;
}
else
{
counters.Add(searchables[i], 1);
}
}
Then, when you are finished processing:
// Add in any searches which had no results:
foreach (var searchTerm in searchables)
{
if (counters.ContainsKey(searchTerm) == false)
{
counters.Add(searchTerm, 0);
}
}
foreach (var item in counters)
{
Console.WriteLine("Value {0} occurred {1} times", item.Key, item.Value);
}
you could use a class for the searchables like:
public class Searchable
{
public string searchTerm;
public int count;
}
then
while ((line = reader.ReadLine()) != null)
{
foreach (var searchable in searchables)
{
if (line.Contains(searchable.searchTerm))
{
searchable.count++;
}
}
}
This would be one of many ways to track multiple search terms and their counts.
You can make use of linq here:
string lines = reader.ReadtoEnd();
var result = lines.Split(new string[]{" ","\r\n"},StringSplitOptions.RemoveEmptyEntries)
.GroupBy(x=>x)
.Select(g=> new
{
Alphabet = g.Key ,
Count = g.Count()
}
);
Input:
a b c
d e f
Output :
a: 1
b: 1
c: 1
d: 1
e: 1
f: 1
This version will count 1^n search terms that occur 1^n times per file line. It accounts for the possibility of a term existing more than once on one line.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApplication5
{
class Program
{
static void Main(string[] args)
{
Func<string, string[], Dictionary<string, int>> searchForCounts = null;
searchForCounts = (filePathAndName, searchTerms) =>
{
Dictionary<string, int> results = new Dictionary<string, int>();
if (string.IsNullOrEmpty(filePathAndName) || !File.Exists(filePathAndName))
return results;
using (TextReader tr = File.OpenText(filePathAndName))
{
string line = null;
while ((line = tr.ReadLine()) != null)
{
for (int i = 0; i < searchTerms.Length; ++i)
{
var searchTerm = searchTerms[i].ToLower();
var index = 0;
while (index > -1)
{
index = line.IndexOf(searchTerm, index, StringComparison.OrdinalIgnoreCase);
if (index > -1)
{
if (results.ContainsKey(searchTerm))
results[searchTerm] += 1;
else
results[searchTerm] = 1;
index += searchTerm.Length - 1;
}
}
}
}
}
return results;
};
var counts = searchForCounts("D:\\Projects\\ConsoleApplication5\\ConsoleApplication5\\TestLog.txt", new string[] { "one", "two" });
Console.WriteLine("----Counts----");
foreach (var keyPair in counts)
{
Console.WriteLine("Term: " + keyPair.Key.PadRight(10, ' ') + " Count: " + keyPair.Value.ToString());
}
Console.ReadKey(true);
}
}
}
Input:
OnE, TwO
Output:
----Counts----
Term: one Count: 7
Term: two Count: 15
I have written a method in C# which retrieves tweets from mongoDB and would like to count and sort authors by the number of retweets.
Right now, the method already performs map and reduce and returns unsorted results the following way:
public void RetweetsCount()
{
string wordMap = #"function wordMap() {
var usernameOrigin = this.text.match(/\brt\s*#(\w+)/i);
if (usernameOrigin === null) {
return;
}
// loop every word in the document
emit(usernameOrigin[1], { count : 1 });
}";
string wordReduce = #"function wordReduce(key, values) {
var total = 0;
for (var i = 0; i < values.length; i++) {
total += values[i].count;
}
return { count : total };
}";
var options = new MapReduceOptionsBuilder();
options.SetOutput(MapReduceOutput.Inline);
var results = collection.MapReduce(wordMap, wordReduce, options);
foreach (var result in results.GetResults())
{
Console.WriteLine(result.ToJson());
}
}
Does anyone know how sort results by descending count value (number of retweets)?
Here's the solution. After retrieving results from MapReduce, I first converted the IEnumerable to list and then ordered the list the folliwing way:
var results = collection.MapReduce(wordMap, wordReduce, options);
IEnumerable<BsonDocument> resultList = results.GetResults();
List<BsonDocument> orderedList = resultList.ToList().OrderByDescending(x => x[1]).ToList();