We Keep Coding

Track Data Input Through Application Code and System Libraries

I am a security dude, and I have done extensive research on this one, and at this point I am looking for guidance on where to go next.
Also, sorry for the long post, I bolded the important parts.
What I am trying to do at a high level is simple:
I am trying to input some data into a program, and "follow" this data, and track how it's processed, and where it ends up.
For example, if I input my login credentials to FileZilla, I want to track every memory reference that accesses, and initiate traces to follow where that data went, which libraries it was sent to, and bonus points if I can even correlate it down to the network packet.
Right now I am focusing on the Windows platform, and I think my main question comes down to this:
Are there any good APIs to remote control a debugger that understand Windows forms and system libraries?
Here are the key attributes I have found so far:
The name of this analysis technique is "Dynamic Taint Analysis"
It's going to require a debugger or a profiler
Inspect.exe is a useful tool to find Windows UI elements that take input
The Windows automation framework in general may be useful
Automating debuggers seems to be a pain. IDebugClient interface allows for more rich data, but debuggers like IDAPro or even CheatEngine have better memory analysis utilities
I am going to need to place memory break points, and track the references and registers that are associated with the input.
Here are a collection of tools I have tried:
I have played with all the following tools: WinDBG (awesome tool), IDA Pro, CheatEngine, x64dbg, vdb (python debugger), Intel's PIN, Valgrind, etc...
Next, a few Dynamic Taint Analysis tools, but they don't support detecting of .NET components or other conveniences that Windows debugging framework provides natively provided by utilities like Inspect.exe:
https://github.com/wmkhoo/taintgrind
http://bitblaze.cs.berkeley.edu/temu.html
I then tried writing my own C# program using IDebugClient interface, but the it's poorly documented, and the best project I could find was from this fellow, and is 3 years old:
C# app to act like WINDBG's "step into" feature
I am willing to contribute code to an existing project that fits this use case, but at this point I don't even know where to start.
I feel like as a whole dynamic program analysis and debugging tools could use some love... I feel kind of stuck, and don't know where to move from here. There are so many different tools and approaches to solving this problem, and all of them are lacking in some manner of another.
Anyway, I appreciate any direction or guidance. If you made it this far thanks!!
-Dave

If you insist on doing this at runtime, Valgrind or Pin might be your best bet. As I understand it (having never used it), you can configure these tools to interpret each machine instruction in an arbitrary way. You want to trace dataflows through machine instructions to track tainted data (reads of such data, followed by writes to registers or condition code bits). A complication will likely be tracing the origin of an offending instruction back to a program element (DLL? Link module? Named subroutine) so that you can complain appropriately.
This a task you might succeed at doing as an individual in terms of effort.
This should work for applications.
I suspect one of your problems will be tracing where goes in the OS. That's a lot harder although the same principle applies; your difficulty will be getting the OS supplier to let you track insructions executed in the OS.
Doing this as runtime analysis has the downside that if a malicious application doesn't do anything bad on your particular execution, you won't find any problems. That's the classic shortcoming of dynamic analysis.
You could consider tracking the data the source code level using classic compiler techniques. This requires that you have access to all the source code that might be involved (that's actually really hard if your application depends on a wide variety of libraries), that you have tools that can parse and track dataflows through source modules, and that these tools talk to each other for different languages (assembler, C, Java, SQL, HTML, even CSS...).
As static analysis, this has the chance of detecting an undesired dataflow no matter which execution occurs. Turing limitations means that you likely cannot detect all such issues. THat's the shortcoming of static analysis.
Building your own tools, or even integrating individual ones, to do this is likely outside what you can reasonably do as an individual. You'll need to find uniform framework for building such tools. [Check my bio for one].

How to find unused parts of a large .NET application?

Consider a large multi-tier enterprise web application and many services with very complex functionality, mostly written in .NET (C#) on the server side and obviously html and javascript on the client, consisting of many hundred pages with the amount of service calls (actions) well in the thousands, hosted on multiple servers and developed over 15 years. Some parts are very new and modern, other parts are legacy.
Some parts of this application are obsolete and nobody actually uses those parts anymore. Whether these are whole unused sub-applications, unused pages, files, service calls, methods or even lines of code, doesn't matter. Older parts do not provide any usage statistics but do use dependency injection.
How can one automatically find out, based on access to production servers, which parts are unused, without changing the actual source code? So the question is not finding unreferenced / unreachable code. It's about finding parts that users don't actually use anymore.
One option could be looking at query logs. This discovers unused pages, but it is very difficult (a tedious manual process) to find out which parts in the background are used by those pages only.
Another option could probably be monitoring file access on servers. Does that make sense? Would that be feasible?
Yet another thought is doing something like test coverage tools do, but not during testing. Could coverage (something like lines of code executed) be measured in a live C#.NET application, assuming that debug symbols are available?

It is hard to give an answer without really knowing the situation. However, I do not think there is some automatic or easy way. I do not know the best solution, but I can tell you what I would do. I would start with collecting all log files from the (IIS?) server (for at least a year, code could be used once a year) and analyze those. This should give you the best insight on which parts are called externally. You do have those logs?
Also check the eventlogs. Sometimes there are messages like 'Directory does not exist', which could mean that the service isn't working for years but nobody noticed. And check for redundant applications, perhaps applications are active on multiple servers.
Check inside tables with time indications and loginfo for recent entries.
Checking the dates on files and analyzing the database may provide additional information, but I don't think it will really help.
Make a list of all applications that you think are obsolete, based on user input or applications that should be obsolete.
Use your findings to create a list based on the probability that application /code is obsolete. Next steps, based on your list, could be:
remove redundant applications.
look for changes in the datamodel of filesystem and check if these still match with the code.
analyze the database for invalid queries. This could indicate that the datamodel has changed, causing the application to stop working. If nobody noticed then this application or functionality is obsolete.
add logging to the code where you have doubts.
look at application level and start with marking calls as obsolete, comment / removing unused code or redirect to (new) equivalent code.
turn off applications and monitor what happens. If there is a dependency then you can take action to remove this dependency or choose to let the application live.
Monitoring the impact of your actions will help you to sort things out. I hope this answer gives you some ideas.
-- UPDATE --
There may be logging available, but collecting, reading and interpreting may be hard and time consuming. To make it easier to monitor you could think of the following:
monitor database: you can use the profiler tool, but it may be easier to create a trigger that logs all CRUD operations with all the information you need. Create a program that can read the scheme of the database and filter the log by table, stored procedure, view to determine what isn't used. I didn't investigate, but perhaps you can monitor rollbacks and exceptions as well.
monitor IIS. There are off course the log files, but you can also think of adding a Module to the website where you can write custom code to monitor whatever you want. All traffic passes the module. Take a look here: https://www.iis.net/learn/develop/runtime-extensibility/developing-iis-modules-and-handlers-with-the-net-framework. If I am not mistaken all you have to do is add the module to the website and configure the website to use the module. Create a program to filter the log on url, status, ip, identification, etc. to determine what is used.
I think that is sufficient for first analysis. It then comes to interpreting the logs. Perhaps you'll see a way to combine the logs so you can link a request to certain database actions, without having to look in or change the code. Just some thoughts.

You can use ReSharper. It will tell you such problems while you're coding.
However you can also detect problems afterwards. In the Menu you will find the entry "ReSharper > Inspect > Code Issues in Solution".
It will create a report, there you will find it under "Redundancies in Code".

How to detect which ecommerce software is being used

I am making an webcrawler in C# which needs to find webshops. The problem i'm having is that I need to detect if the webpage is a webshop. If it is I need to find out what type of e-commerse software it is using. But the problem is that I don't know how you can detect it in the source code.
I also have just a Chrome plugin called builtwith which can detect all kinds of software. But I have yet to find out how they are doing that.
It would be nice if someone could help me with this problem

Before giving you an actual answer, it's worth noting that what you're proposing could be in violation of the terms of use for many websites out there. You should take the time to investigate what legal liability you might be exposing yourself and your organization to.
This is going to be a lot of time consuming work, but it's not difficult. Your crawler is just going to need to simply work using a rules-based approach to detect signatures in the payload of the page.
Find the specific software that you're intending to detect.
Find 2-3 sites that are definitely using the software.
Review the HTML payload to see what scripts, CSS, and HTML patterns they have that are common across the sites.
Build a code-based rule that can detect those patterns consistently. For example: if (html.Contains("widgetName")) isPlatformName = true;
Test that patterns across more sites that you know for certain are using that software.
Repeat for each software vendor.
The more complicated thing will be when the targets have multiple versions and you need to adapt your rules to know and be aware of the various versions, or when platforms are very similar.
I think the most complicated part of this is having a well-thought-out bot issue detection, reporting, and throttling architecture in place. You should probably spend the bulk of your time planning that.
That's it.

There are a couple different ways to determine the technologies a site is using. Firstly, if you are technically savvy, you can right click on an eCommerce page (either catalog, checkout page, etc) and look at the source code. Many platforms will have hints in the source code that will give you an idea what the site is running.
You can also look at the DNS/hosting information, which would help you determine if the eCommerce solution is hosted or SaaS (like Shopify, for example).
You can also try using InterNIC and enter the domain name. The results will return the nameservers which could point you in the right direction.
Finally, if all that sleuthing seems too difficult, there’s an easier way! Try BuiltWith. It’s generally pretty reliable, as long as the system you're looking up isn’t custom/proprietary. Enter a domain into BuiltWith and it will show you the platform, widgets used, analytics and tracking codes, CDNs, CMS, payment processors, and more.

What considerations need to be made when transitioning an application to support?

I will be taking on the role of support for a complex application that is transitioning from the development team. This application is a sharepoint solution that connects to several (7) web services. The development team is rolling off almost immediately and will be available only for small questions.
I'm new to this role so I'm wondering what suggestions you have for me as I take on this large project. What are some considerations that should be made so that the transition to support is smooth and uninterupted?
I've been reading the documentation but I can already see some gaps that need to be filled. The applicaiton is very (perhaps overly) configurable and there is lots of injected code. Stepping through the code is about the only way I can gain an understanding of what is actually happening.

It sounds like you've already got your environment set up if you're able to debug the application, so that's the first thing I was going to suggest in a knowledge-transfer situation. Some general things that I would get from the developers before they depart:
A list of third-party components that the application uses, along with license information and website logins if applicable.
Access to every part of the environment that this thing runs on, both production and development. That means the source code management system, database server(s), etc. It sounds like you have some of these already but make sure you get access to absolutely everything.
If your development environment was given to you "as is" (i.e. you took it over from one of the departing developers, make sure you know how to rebuild it from scratch. They might have a document that describes the process of building a development box, but if not maybe you can get them to show you how to set up a fresh machine.
Three will go a long way towards this, but if setting up a server to run the application is different in any way from setting up a development environment, you'd want to know how so you can diagnose server configuration issues if they crop up, or even rebuild a server. Although this sort of thing may be someone else's responsibility depending on your organization.
Once you have those, you probably want to get some understanding of why the application does the things that it does. That will give you the context you need to understand support and enhancement requests when they come in.
Are the original developers the only source of this information, or are there business people who you will be working with after the developers leave? One of the first things I try to do when starting on an existing application that's new to me is to find someone who knows the business well and have them give me a high-level run-down of the application's purpose in life. From there you can go into more detail on individual components/features/whatever as needed. The business people may be a better source for this information than the developers are, so you may want to try them first.
Hopefully some of that helps.

If you're not the systems admin (as opposed to the SharePoint admin), develop an understanding with them of what tasks you are able to do and what you need of them.
This may include things like stopping and starting services (IIS, Timer Service, etc.) and filesystem and DB monitoring and maintenance. Getting this sorted out up front saves a lot of pain later.
If the sys admins don't have some understanding of SharePoint, educate them. They will need to know what the deal is with things like code deployments.
It's best not to feel my pain.

Implementing a fully-functional prototype for a web-based application

I am building a prototype for a web-based application and was considering building the front-end in HTML, which can then be reused later for the actual application. I had done a Flash-based prototype earlier, which embedded the .swf into a C# executable. Flash made for rapid turnaround time while the Windows application provided unlimited access to fancy API's for DB access and sound.
I want to consider something similar for this one too. Does this approach make sense? I am particularly concerned about the way the HTML would communicate with the container app. From what I understand out of preliminary research, it would be only through JavaScript, which might quickly get unwieldy. This is especially so because unlike the Flash-based prototype which implemented a lot of its functionality in the .swf, the HTML UI will depend entirely upon the shell to maintain state. Also, I don't need anything more than access to a database. So a desktop application might be overkill.
Another alternative that comes to mind is to build the prototype using PHP and deploy it with a portable server stack such as Server2Go or XAMPP. But I've never done something like this before. Anybody here shed some light on drawbacks of this approach?
The key requirement is rapid iterations of the UI, reusable front-end code and simplified deployment without any installations or configuration.

Some of the best programming advice I've seen came from Code Complete, and was along the lines of, "evolutionary prototypes are fine things, and throwaway prototypes are fine things, but you run into trouble when you try to make one from the other." That is, know which type of prototype you're developing, and respect it. If you're developing a throwaway prototype, don't permit yourself to use any of it, however tempting it may be, in the production system. And if you're developing an evolutionary prototype - one intended to become the production system - don't compromise quality in any way.
It sounds like you're trying to get both, the rapid development of a throwaway and the reusability of an evolutionary prototype - and you can't. Make up your mind, and stand by it. You can't have your cake and eat it, too.

I think you off to the wrong start, here. Why would you want your prototype to be fully functional? A prototype is intended to be throw-away and to help flesh out requirements and UI. If you need full functionality, why not just skip to the final product? If prototyping is really something you want to do, I suggest looking into a specialized prototyping tool.

Are you prototyping the user interface for a customer? If you are, consider something less unwieldy like paper prototypes or presentation software (like PowerPoint) until you get the UI nailed down. If you can establish the UI and are clear about the customer's requirements, you can then develop the application in whatever the actual platform is going to be with a clear model in mind.
In my current project, I prototyped the UI in PowerPoint first. In a subsequent iteration, I used static web pages and some jQuery plugins to simulate actual user interaction. That proved to be very effective in demonstrating the interface, and I didn't have to build the application first.

I would join in on folks suggesting paper prototyping as the "idea", but not necessarily the implementation. The biggest point here is that tools such as HTML or Flash let you get "bogged down" in the details - what does this color look like? What's the text on this thing? Lots of time can pass by that way. Instead, what you should be focusing on is user flows.
One tool that keeps the spirit of paper prototyping without all the "paper" drawbacks is Balsamiq: http://www.balsamiq.com/demos/mockups/Mockups.html. It was covered by Jeff and Joel in one of the Stack Overflow podcasts; I've been using it for my own projects for a while. It's freeware, and it does its job magnificently.

If you know C# then another option you can look at is Silverlight. You can then leverage your knowledge of C# and/or JavaScript and interact with a rich object model.
Would that do what you are looking for? The installation would be minimal on the part of the client - download and install the Silverlight plugin

If prototyping is something you truly wish to accomplish here, paper and pencil will be your best friends. You can draw out as many iterations as necessary. While none of this is ultimately useful later on once you begin coding, it is as quick and rapid is it goes.
As mentioned previously, there are many prototyping tools which have a bit of a learning curve, but an alternative to consider would be using a framework such as CakePHP or Ruby on Rails which make for fast application logic and leave customizing the front end being the primary hard work left. And plus, you're left with a mostly functional application when you're done with your prototyping which can be tweaked as needed.
In either scenario, you're paying with your time either upfront (in the case with learning a new framework), of over time in payments (with the case of prototyping on paper or coding by hand).