I need to make a frequency analysis console program using c#. It has to show the 10 most frequent letters from a textfile. I have managed to display the first 10 letters read by the program and the frequency of each character. I, however, don't know how to sort the dictionary. This is the code I have so far.
I must also give the user the option to the frequency analysis in case sensitive mode (as it is right now) and case insensitive. Help with this issue will also be appreciated. Thank You!
static void Main(string[] args)
{
// 1.
// Array to store frequencies.
int[] c = new int[(int)char.MaxValue];
// 2.
// Read entire text file.
// string root = Server.MapPath("~");
// string FileName = root + "/App_Data/text.txt";
//string s = File.ReadAllText(FileName);
foreach (string line in File.ReadLines(#"c:\Users\user\Documents\Visual Studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\App_Data\text.txt", Encoding.UTF8)) {
var fileStream = new FileStream(#"c:\Users\user\Documents\Visual Studio 2015\Projects\ConsoleApplication1\ConsoleApplication1\App_Data\text.txt", FileMode.Open, FileAccess.Read);
using (var streamReader = new StreamReader(fileStream, Encoding.UTF8))
{
string line2;
while ((line2 = streamReader.ReadLine()) != null)
{
// process the line
// 3.
// Iterate over each character.
foreach (char t in line)
{
// Increment table.
c[(int)t]++;
}
// 4.
// Write all letters found.
int counter = 0;
for (int i = 0; i < (int)char.MaxValue; i++)
{
if (c[i] > 0 && counter < 11 &&
char.IsLetterOrDigit((char)i))
{
++counter;
Console.WriteLine("Letter: {0} Frequency: {1}",
(char)i,
c[i]);
}
}
}
}
Console.ReadLine();
}
}
If all you want to do is to found frequencies, you don't want any dictionaries, but a Linq. Such tasks are ones Linq has been designed for:
...
using System.Linq;
...
static void Main(string[] args) {
var result = File
.ReadLines(#"...", Encoding.UTF8)
.SelectMany(line => line) // string into characters
.Where(c => char.IsLetterOrDigit(c))
.GroupBy(c => c)
.Select(chunk => new {
Letter = chunk.Key,
Count = chunk.Count() })
.OrderByDescending(item => item.Count)
.ThenBy(item => item.Letter) // in case of tie sort by letter
.Take(10)
.Select(item => $"{item.Letter} freq. {item.Count}"); // $"..." - C# 6.0 syntax
Console.Write(string.Join(Environment.NewLine, result));
}
I like #Dmitry Bychenko's answer because it's very terse. But, if you have a very large file then that solution may not be optimal for you. The reason being, that solution has to read the entire file into memory to process it. So, in my tests, I got up to around 1GB of memory usage for a 500MB file. The solution below, while not quite as terse, uses constant memory (basically 0) and runs as fast or faster than the Linq version in my tests.
Dictionary<char, int> freq = new Dictionary<char, int>();
using (StreamReader sr = new StreamReader(#"yourBigFile")) {
string line;
while ((line = sr.ReadLine()) != null) {
foreach (char c in line) {
if (!freq.ContainsKey(c)) {
freq[c] = 0;
}
freq[c]++;
}
}
}
var result = freq.Where(c => char.IsLetterOrDigit(c.Key)).OrderByDescending(x => x.Value).Take(10);
Console.WriteLine(string.Join(Environment.NewLine, result));
It would be easier to use the actual Dictionary type in C# here, rather than an array:
Dictionary<char, int> characterCountDictionary = new Dictionary<char, int>();
You add a key if it doesn't exist already (and insert a value of 1), or you increment the value if it does exist. Then you can pull out the keys of your dictionary as a list and sort them, iterating to find the values. If you do case insensitive you'd just convert all upper case to lower case before inserting into the dictionary.
Here's the MSDN page for the examples for Dictionary: https://msdn.microsoft.com/en-us/library/xfhwa508(v=vs.110).aspx#Examples
Related
So the aim is to sort a word into letters and give it a score based on each letters worth. i have this working and have put it int a Dictionary using the score as the key and word as the value.
I need to find the highest scoring word that uses only the letters provided (each only used once), and if there are multiple with the same score then the first that was added to the dictionary needs to be printed. the code is as below;
Dictionary<string, int> wordHolder = new Dictionary<string, int>();
int N = int.Parse(Console.ReadLine());
for (int i = 0; i < N; i++)
{
string W = Console.ReadLine();
char[] splitWord = W.ToCharArray();
Console.Error.WriteLine(W);
int points = splitWord.Sum(c => letterPoints.First(kvp => kvp.Value.Contains(c)).Key);
wordHolder.Add(W, points);
Console.Error.WriteLine(points);
}
string LETTERS = Console.ReadLine();
Console.Error.WriteLine(LETTERS);
Letters is the provided characters given in a single string
You can order by descending and get first element:
wordHolder.OrderByDescending(x => x.Value).First();
Or Order by ascending and get last element:
wordHolder.OrderBy(x => x.Value).Last();
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApplication9
{
internal static class Program
{
private static void Main()
{
// --- building up the data so the next (repeated) steps are efficient: ---
// your words and points
var wordHolder = new Dictionary<string, int>();
// INSERT DATA INTO wordHolder HERE
// create a dictionary where you can access all words and their points
// efficiently by the letters you have available:
var efficientAccessByLetters = wordHolder.Select(entry =>
new
{
Index = new string(entry.Key.OrderBy(c => c).ToArray()),
Word = entry.Key,
Points = entry.Value
})
.GroupBy(x => x.Index)
.ToDictionary(x => x.Key,
x => x.OrderByDescending(p => p.Points).ToList());
// --- Repeat this steps as many times as you like: ---
while (true)
{
Console.WriteLine("Enter letters available:");
// get letters that you want to build he best word from:
var availableLetters = Console.ReadLine();
// normalize the letters for fast index search by sorting them
var normalized = new string(availableLetters.OrderBy(c => c).ToArray());
if (!efficientAccessByLetters.ContainsKey(normalized))
{
Console.WriteLine("Sorry, no matching words found.");
}
else
{
// find all words and their respective points by index access
var wordsThatMatchLetters = efficientAccessByLetters[normalized];
// as words are sorted by points, just get the first
var best = wordsThatMatchLetters.First();
// output
Console.WriteLine("Best Match: {0} for {1} points", best.Word, best.Points);
}
}
}
}
}
The best of way of doing it without a dictionary.
public static string High(string s)
{
return s.Split(' ').OrderByDescending(a => a.Select(b => b - 96).Sum()).Last();
}
I have been trying to get this to work for the past two hours with quite a lot of success but it is not working 100%. It auto suggests lines from the file to the nearest 1-2 words I believe which is not what I want. I want it to suggest it as best as possible.
An example is here: https://gyazo.com/6cb86e78b95026aeeb3f8cc96ca03163
As you can see, it gives me two suggestions in the console but prints the right one to me due to where it is placed in the text document. Howcome it is not giving me the only one I want even though it's word-for-word and therefore has more keywords in common than the one without stattrak.
This is my code:
string[] suggestContents = File.ReadAllLines("csgo_items.txt");
int lineCount = 0;
int mostSimilar = 0;
int currentSimilar = 0;
string[] splitLine = message.Split("_".ToCharArray());
Dictionary<int, int> suggestItem = new Dictionary<int, int>();
foreach (string line in suggestContents)
{
for(int i = 0; i < splitLine.Length; i++)
{
if(line.ToLower().Contains(splitLine[i].ToLower()))
{
currentSimilar++;
}
}
if(currentSimilar > mostSimilar)
{
List<int> suggestList = new List<int>();
foreach (KeyValuePair<int, int> entry in suggestItem)
{
if(entry.Value == mostSimilar)
{
suggestList.Add(entry.Key);
}
}
foreach(int intRemove in suggestList)
{
suggestItem.Remove(intRemove);
}
mostSimilar = currentSimilar;
suggestItem.Add(lineCount, mostSimilar);
}
else if(currentSimilar == mostSimilar)
{
suggestItem.Add(lineCount, mostSimilar);
}
lineCount++;
currentSimilar = 0;
}
Thanks, any help is appreciated.
Let me rephrase your code (that looks correct btw) in a more idiomatic translation
var res =suggestContents.Select(
(x,i) => new Tuple<int, string,int>
(i, x , message.Split('_')
.Count(z => x.ToLower().Contains(z.ToLower())))
).GroupBy(t => t.Item3).OrderByDescending(t => t.Key).First();
You may want to change it to
var res = suggestContents.Select(
(x, i) => new Tuple<int, string, int>
(i, x, message.Split('_')
.Count(z =>
(x.ToLower().Contains(z.ToLower()))
||
(x.ToLower().Split(' ').Any(w =>
z.ToLower().Contains(w.ToLower())))
))
).GroupBy(t => t.Item3).OrderByDescending(t => t.Key).First();
The idea is to test not ony if the line contains a word but also if the word (stattraktm) contains part (stattrakt) of any split of the line.
It is hard to help without seeing your TXT file. It seems strange that your console says stattrakt, yet on the right you can see stattrak™ which is totally different word and will not be a match in .Contain or anything.
Without changing your code much, although I support Machine Learning solution, I prefer keep the answer as close to the original code-
string[] suggestContents = File.ReadAllLines("csgo_items.txt");
int lineCount = 0;
int mostSimilar = 0;
int currentSimilar = 0;
string[] splitLine = message.Split("_".ToCharArray());
//Dictionary<int, int> suggestItem = new Dictionary<int, int>();
List<string> suggestedItems = new List<string>();
foreach (string line in suggestContents)
{
for(int i = 0; i < splitLine.Length; i++)
{
if(line.ToLower().Contains(splitLine[i].ToLower()))
{
currentSimilar++;
}
}
if(currentSimilar > mostSimilar)
{
//We clear the current list, it is no longer needed, we have better match
//List<int> suggestList = new List<int>();
suggestedItems.Clear();
mostSimilar = currentSimilar;
//add current line to array.
suggestedItems.Add(line);
}
else if(currentSimilar == mostSimilar)
{
//if another match simply add to list without clearing it.
suggestedItems.Add(line);
}
lineCount++;
currentSimilar = 0;
}
Might have mistakes cause I did it on the editor here.
Another option might be replacing the foreach with regular for loop, and save the index of the line instead of the line itself if it more important to you in a same way only List of int.
This method takes the most frequent words from string array.
It works very slowly for big arrays (like 190.000 milliseconds for 70.000 strings).
I've measured (using Stopwatch()) that its first part is the slowest one:
public static List<WordDouble> MostFrequentWords(double count, string[] words)
{
var wordsAndNumbers = new List<WordDouble>();
foreach (var word in words)
{
if (wordsAndNumbers.Exists(e => e.Word == word.ToLower()))
wordsAndNumbers[wordsAndNumbers.FindIndex(e => e.Word == word.ToLower())].Count++;
else
{
var addWord = new WordDouble();
addWord.Word = word.ToLower();
addWord.Count = 1;
wordsAndNumbers.Add(addWord);
}
}
/*method goes on, other parts work fast and do not need improvement */
...
return something;
}
public class WordDouble
{
public string Word;
public double Count;
}
How can I improve performance of this method?
Checking for an item using Exists in a list is an O(n) operation, while checking for an item in a dictionary is an O(1) operation.
This runs in a fraction of the time (actually in about 1/2200 of the time):
Dictionary<string, int> wordsAndNumbers = new Dictionary<string, int>();
foreach (string word in words) {
if (wordsAndNumbers.ContainsKey(word.ToLower())) {
wordsAndNumbers[word.ToLower()]++;
} else {
wordsAndNumbers.Add(word.ToLower(), 1);
}
}
Here is the result of a test run with 70000 strings, for the original code, my code, and Console's code, respectively:
00:01:21.0804944
00:00:00.0360415
00:00:00.1060375
You can even speed it up a little more by doing ToLower only once in the loop:
var wordsAndNumbers = new Dictionary<string, int>();
foreach (var word in words) {
string s = word.ToLower();
if (wordsAndNumbers.ContainsKey(s)) {
wordsAndNumbers[s]++;
} else {
wordsAndNumbers.Add(s, 1);
}
}
Test run:
00:00:00.0235761
First of all why do you use a double to count words?
Use a long Dictionary and never cast to lower just for comparison.
Dictionary<string,long> wordsAndNumbers = new
Dictionary<string,long>(StringComparer.OrdinalIgnoreCase);
foreach(var word in words)
{
if (!wordsAndNumbers.ContainsKey(word))
wordsAndNumbers[word] = 1;
else
wordsAndNumbers[word]++;
}
with 70000 Words i get the following runtime: 00:00:00.0152345 which is significant faster then the to lower solution on my machine which takes 00:00:00.0320127
I get many lines of String as an Input that look like this. The Input is a String that comes from
theObjects.Runstate;
each #VAR;****;#ENDVAR; represents one Line and one step in the loop.
#VAR;Variable=Speed;Value=Fast;Op==;#ENDVAR;#VAR;Variable=Fabricator;Value=Freescale;Op==;#ENDVAR;
I split it, to remove the unwanted fields, like #VAR,#ENDVAR and Op==.
The optimal Output would be:
Speed = Fast;
Fabricator = Freescale; and so on.
I am able to cut out the #VAR and the#ENDVAR. Cutting out the "Op==" wont be that hard, so thats now not the main focus of the question. My biggest concern right now is,thatI want to print the Output as a Text-File. To print an Array I would have to loop over it. But in every iteration, when I get a new line, I overwrite the Array with the current splitted string. I think the last line of the Inputfile is an empty String, so the Output I get is just an empty Text-File. It would be nice if someone could help me.
string[] w;
Textwriter tw2;
foreach (EA.Element theObjects in myPackageObject.Elements)
{
theObjects.Type = "Object";
foreach (EA.Element theElements in PackageHW.Elements)
{
if (theObjects.ClassfierID == theElements.ElementID)
{
t = theObjects.RunState;
w = t.Replace("#ENDVAR;", "#VAR;").Replace("#VAR;", ";").Split(new string[] { ";" }, StringSplitOptions.RemoveEmptyEntries);
foreach (string s in w)
{
tw2.WriteLine(s);
}
}
}
}
This linq-query gives the exptected result:
var keyValuePairLines = File.ReadLines(pathInputFile)
.Select(l =>
{
l = l.Replace("#VAR;", "").Replace("#ENDVAR;", "").Replace("Op==;", "");
IEnumerable<string[]> tokens = l.Split(new[]{';'}, StringSplitOptions.RemoveEmptyEntries)
.Select(t => t.Split('='));
return tokens.Select(t => {
return new KeyValuePair<string, string>(t.First(), t.Last());
});
});
foreach(var keyValLine in keyValuePairLines)
foreach(var keyVal in keyValLine)
Console.WriteLine("Key:{0} Value:{1}", keyVal.Key, keyVal.Value);
Output:
Key:Variable Value:Speed
Key:Value Value:Fast
Key:Variable Value:Fabricator
Key:Value Value:Freescale
If you want to output it to another text-file with one key-value pair on each line:
File.WriteAllLines(pathOutputFile, keyValuePairLines.SelectMany(l =>
l.Select(kv => string.Format("{0}:{1}", kv.Key, kv.Value))));
Edit according to your question in the comment:
"What would I have to change/add so that the Output is like this. I
need AttributeValuePairs, for example: Speed = Fast; or Fabricator =
Freescale ?"
Now i understand the logic, you have key-value pairs but you are interested only in the values. So every two key-values belong together, the first value of a pair specifies the attibute and the second value the value of that attribute(f.e. Speed=Fast).
Then it's a little bit more complicated:
var keyValuePairLines = File.ReadLines(pathInputFile)
.Select(l =>
{
l = l.Replace("#VAR;", "").Replace("#ENDVAR;", "").Replace("Op==;", "");
string[] tokens = l.Split(new[]{';'}, StringSplitOptions.RemoveEmptyEntries);
var lineValues = new List<KeyValuePair<string, string>>();
for(int i = 0; i < tokens.Length; i += 2)
{
// Value to a variable can be found on the next index, therefore i += 2
string[] pair = tokens[i].Split('=');
string key = pair.Last();
string value = null;
string nextToken = tokens.ElementAtOrDefault(i + 1);
if (nextToken != null)
{
pair = nextToken.Split('=');
value = pair.Last();
}
var keyVal = new KeyValuePair<string, string>(key, value);
lineValues.Add(keyVal);
}
return lineValues;
});
File.WriteAllLines(pathOutputFile, keyValuePairLines.SelectMany(l =>
l.Select(kv=>string.Format("{0} = {1}", kv.Key, kv.Value))));
Output in the file with your single sample-line:
Speed = Fast
Fabricator = Freescale
I really cant understand how to make a simple algorithm on C# to solve my problem. So, we have a sentences:
{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}.
So, my program should make a lot of sentences looks like:
Hello my mate.
Hello my m8.
Hello my friend.
Hello my friends.
Hi my mate.
...
Hi-Hi my friends.
I know, there are a lot of programs which could do this, but i'd like to make it myself. Ofcourse, it should work with this too:
{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}, {i|we} want to {tell|say} you {hello|hi|hi-hi}.
Update I just wasn't too happy about my using the regexen to parse so simple input; yet I disliked the manual index manipulation jungle found in other answers.
So I replaced the tokenizing with a Enumerator-based scanner with two alternating token-states. This is more justified by the complexity of the input, and has a 'Linqy' feel to it (although it really isn't Linq). I have kept the original Regex based parser at the end of my post for interested readers.
This just had to be solved using Eric Lippert's/IanG's CartesianProduct Linq extension method, in which the core of the program becomes:
public static void Main(string[] args)
{
const string data = #"{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}, {i|we} want to {tell|say} you {hello|hi|hi-hi}.";
var pockets = Tokenize(data.GetEnumerator());
foreach (var result in CartesianProduct(pockets))
Console.WriteLine(string.Join("", result.ToArray()));
}
Using just two regexen (chunks and legs) to do the parsing into 'pockets', it becomes a matter of writing the CartesianProduct to the console :) Here is the full working code (.NET 3.5+):
using System;
using System.Text;
using System.Text.RegularExpressions;
using System.Linq;
using System.Collections.Generic;
namespace X
{
static class Y
{
private static bool ReadTill(this IEnumerator<char> input, string stopChars, Action<StringBuilder> action)
{
var sb = new StringBuilder();
try
{
while (input.MoveNext())
if (stopChars.Contains(input.Current))
return true;
else
sb.Append(input.Current);
} finally
{
action(sb);
}
return false;
}
private static IEnumerable<IEnumerable<string>> Tokenize(IEnumerator<char> input)
{
var result = new List<IEnumerable<string>>();
while(input.ReadTill("{", sb => result.Add(new [] { sb.ToString() })) &&
input.ReadTill("}", sb => result.Add(sb.ToString().Split('|'))))
{
// Console.WriteLine("Expected cumulative results: " + result.Select(a => a.Count()).Aggregate(1, (i,j) => i*j));
}
return result;
}
public static void Main(string[] args)
{
const string data = #"{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}, {i|we} want to {tell|say} you {hello|hi|hi-hi}.";
var pockets = Tokenize(data.GetEnumerator());
foreach (var result in CartesianProduct(pockets))
Console.WriteLine(string.Join("", result.ToArray()));
}
static IEnumerable<IEnumerable<T>> CartesianProduct<T>(this IEnumerable<IEnumerable<T>> sequences)
{
IEnumerable<IEnumerable<T>> emptyProduct = new[] { Enumerable.Empty<T>() };
return sequences.Aggregate(
emptyProduct,
(accumulator, sequence) =>
from accseq in accumulator
from item in sequence
select accseq.Concat(new[] {item}));
}
}
}
Old Regex based parsing:
static readonly Regex chunks = new Regex(#"^(?<chunk>{.*?}|.*?(?={|$))+$", RegexOptions.Compiled);
static readonly Regex legs = new Regex(#"^{((?<alternative>.*?)[\|}])+(?<=})$", RegexOptions.Compiled);
private static IEnumerable<String> All(this Regex regex, string text, string group)
{
return !regex.IsMatch(text)
? new [] { text }
: regex.Match(text).Groups[group].Captures.Cast<Capture>().Select(c => c.Value);
}
public static void Main(string[] args)
{
const string data = #"{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}, {i|we} want to {tell|say} you {hello|hi|hi-hi}.";
var pockets = chunks.All(data, "chunk").Select(v => legs.All(v, "alternative"));
The rest is unchanged
Not sure what you need Linq (#user568262) or "simple" recursion (#Azad Salahli) for. Here's my take on it:
using System;
using System.Text;
class Program
{
static Random rng = new Random();
static string GetChoiceTemplatingResult(string t)
{
StringBuilder res = new StringBuilder();
for (int i = 0; i < t.Length; ++i)
if (t[i] == '{')
{
int j;
for (j = i + 1; j < t.Length; ++j)
if (t[j] == '}')
{
if (j - i < 1) continue;
var choices = t.Substring(i + 1, j - i - 1).Split('|');
res.Append(choices[rng.Next(choices.Length)]);
i = j;
break;
}
if (j == t.Length)
throw new InvalidOperationException("No matching } found.");
}
else
res.Append(t[i]);
return res.ToString();
}
static void Main(string[] args)
{
Console.WriteLine(GetChoiceTemplatingResult(
"{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}, {i|we} want to {tell|say} you {hello|hi|hi-hi}."));
}
}
As others have noted, you can solve your problem by splitting up the string into a sequence of sets, and then taking the Cartesian product of all of those sets. I wrote a bit about generating arbitrary Cartesial products here:
http://blogs.msdn.com/b/ericlippert/archive/2010/06/28/computing-a-cartesian-product-with-linq.aspx
An alternative approach, more powerful than that, is to declare a grammar for your language and then write a program that generates every string in that language. I wrote a long series of articles on how to do so. It starts here:
http://blogs.msdn.com/b/ericlippert/archive/2010/04/26/every-program-there-is-part-one.aspx
You can use a Tuple to hold index values of each collection.
For example, you would have something like:
List<string> Greetings = new List<string>()
{
"Hello",
"Hi",
"Hallo"
};
List<string> Targets = new List<string>()
{
"Mate",
"m8",
"friend",
"friends"
};
So now you have your greetings, let's create random numbers and fetch items.
static void Main(string[] args)
{
List<string> Greetings = new List<string>()
{
"Hello",
"Hi",
"Hallo"
};
List<string> Targets = new List<string>()
{
"Mate",
"m8",
"friend",
"friends"
};
var combinations = new List<Tuple<int, int>>();
Random random = new Random();
//Say you want 5 unique combinations.
while (combinations.Count < 6)
{
Tuple<int, int> tmpCombination = new Tuple<int, int>(random.Next(Greetings.Count), random.Next(Targets.Count));
if (!combinations.Contains(tmpCombination))
{
combinations.Add(tmpCombination);
}
}
foreach (var item in combinations)
{
Console.WriteLine("{0} my {1}", Greetings[item.Item1], Targets[item.Item2]);
}
Console.ReadKey();
}
This doesn't look trivial. You need to
1. do some parsing, to extract all the lists of words that you want to combine,
2. obtain all the actual combinations of these words (which is made harder by the fact that the number of lists you want to combine is not fixed)
3. rebuild the original sentence putting all the combinations in the place of the group they came from
part 1 (the parsing part) is probably the easiest: it could be done with a Regex like this
// get all the text within {} pairs
var pattern = #"\{(.*?)\}";
var query = "{Hello|Hi|Hi-Hi} my {mate|m8|friend|friends}.";
var matches = Regex.Matches(query, pattern);
// create a List of Lists
for(int i=0; i< matches.Count; i++)
{
var nl = matches[i].Groups[1].ToString().Split('|').ToList();
lists.Add(nl);
// build a "template" string like "{0} my {1}"
query = query.Replace(matches[i].Groups[1].ToString(), i.ToString());
}
for part 2 (taking a List of Lists and obtain all resulting combinations) you can refer to this answer
for part 3 (rebuilding your original sentence) you can now take the "template" string you have in query and use String.Format to substitute all the {0}, {1} .... with the combined values from part 2
// just one example,
// you will need to loop through all the combinations obtained from part 2
var OneResultingCombination = new List<string>() {"hi", "mate"};
var oneResult = string.Format(query, OneResultingCombination.ToArray());