We Keep Coding

How can I scrape instagram followers?

there are a lot of websites where you can get a list of all followers from an Instagram profile. For example the profile of Jennifer Lopez. If I click on followers and scroll the hole list down, I only see round about 1000 users. Is there any way to get a list of all followers, or something in the range between 10 thousand and 100 thousand users? How do the others do it?
Here are a few pages where it seems to work:
crowdbabble
Instagram Scraper
magimetrics
I would be very grateful if you could help me!

I believe most of the pages you are seeing is using the Instagram API (or the method described below). However, that is a bit hard to get access to without an application that they are happy with. As far as I have understood it, you will have to make the application before you know if you will have access, which is a bit stupid. I guess they are trying to stop new users from using it while they keep letting the people already using it keep using it.
The documentation for their API seems to be missing a lot of what was available earlier, and right now there is no endpoint to get followers(that might be something temporarily wrong with the documentation page: https://www.instagram.com/developer/endpoints/).
You could get the followers the same way the Instagram webpage is doing it. However, it seems only to work if you request up to around 5000-6000 followers at a time, and you might get rate limited.
They are making a GET request to: https://www.instagram.com/graphql/query/ with the query parameters query_hash and variables.
The query_hash I guess is a hash of the variables. However, I might be wrong since it will keep working even tho you change the variables. The same hash might not work forever, so its possible you would have to get the same way the Instagram page is doing it. You will get that even tho you are not logged in, so I would not think it would be very hard.
The variables parameter is an URL encoded JSON object containing your search variables.
The JSON should look like this:
{
"id":"305701719",
"first":20
}
The id is the user's id. The first is the number of followers you want.
The URL would look like this when you encode it. https://www.instagram.com/graphql/query/?query_hash=bfe6fc64e0775b47b311fc0398df88a9&variables=%7B%22id%22%3A%22305701719%22%2C%22first%22%3A20%7D
This will return a json object like this:
"data": {
"user": {
"edge_followed_by": {
"count": 73785285,
"page_info": {
"has_next_page": true,
"end_cursor": "AQDJzGlG3jGfM6KGYF7oOhlMqDm9_-db8DW_8gKYTeKO5eIca7cRqL1ODK1SsMA33BYBbAZz3BdC3ImMT79a1YytB1j9z7f-ZaTIkQKEoBGepA"
},
"edges": [
{
"node": {}
}
]
}
}
}
The edges array will contain a list of node elements containg user info about people that are following the person you where searching for.
To get the next x number of followers, you would have to change the json used in the variables query to something like this:
{
"id":"305701719",
"first":10,
"after":"AQDJzGlG3jGfM6KGYF7oOhlMqDm9_-db8DW_8gKYTeKO5eIca7cRqL1ODK1SsMA33BYBbAZz3BdC3ImMT79a1YytB1j9z7f-ZaTIkQKEoBGepA"
}
after would be what you received as an end_cursor in the previous request.
and your new URL would look like this: https://www.instagram.com/graphql/query/?query_hash=bfe6fc64e0775b47b311fc0398df88a9&variables=%7B%22id%22%3A%22305701719%22%2C%22first%22%3A10%2C%22after%22%3A%22AQDJzGlG3jGfM6KGYF7oOhlMqDm9_-db8DW_8gKYTeKO5eIca7cRqL1ODK1SsMA33BYBbAZz3BdC3ImMT79a1YytB1j9z7f-ZaTIkQKEoBGepA%22%7D
This way you can keep looping until has_next_page is false in the response.

EDIT 23/08/2018
Instagram seems to have blocked any scrolling/query hash request to get followers list/likers list on a post, at least on desktop, even for your own account.
https://developers.facebook.com/blog/post/2018/01/30/instagram-graph-api-updates/
It should be although still possible from a phone, maybe following a Selenium-like for mobile, using Appmium : http://appium.io/
Maybe some reverse app engineering may also be the key, if any idea from that side :
https://www.blackhatworld.com/seo/journey-instagram-app-reverse-engineer.971468/
EDIT 25/08/2018
It seems to be back... if any information about it ?

Communication with client and server using sockets

So I have written the code so that I can communicate with server and client.
The first question is how does the server identify that its communicating with an actual client, not someone else who's using the port, I've heard that browsers verify with servers using SHA hashing.
Second question is about the best way to send and receive data in variables, and also identifying which is which, because the current method of splitting data doesn't seem very elegant.
Server side code to receive and send data:
NetworkStream NetStream1 = TCPSocket.GetStream();
NetStream.Read(Buffer, 0, Buffer.Length);
ReceivedData = System.Text.Encoding.ASCII.GetString(Buffer);
string[] splitter = ReceivedData.Split('-');
Variable1 = splitter[0];
Variable2 = splitter[1];
//send response
SendBuffer = Encoding.ASCII.GetBytes(ResultINT1+"-"+ResultINT2);
NetStream.Write(SendBuffer, 0, SendBuffer.Length);
NetStream.Flush();
Client code to send and receive
NetworkStream SendStream = ClientSocket.GetStream();
byte[] SendBuffer = System.Text.Encoding.ASCII.GetBytes(V1+"-"+V2);
SendStream.Write(SendBuffer, 0, SendBuffer.Length);
SendStream.Flush();
//response
SendStream.Read(RecieveBuffer, 0, RecieveBuffer.Length);
string ResultString = System.Text.Encoding.ASCII.GetString(RecieveBuffer);
string[] splitted = ResultString.Split('-');
int R1 = Convert.ToInt32(splitted[0]);
int R2 = Convert.ToInt16(splitted[1]);

Provide some authentication mechanism
Use some serializer.

Your first question concerns authentication which is a huge subject and has many possible implementations although I'm not sure exactly what you mean by "someone else who's using the port". Your server should always be on the same port - that is how the client identifies a service.
Regarding your second question there are again many possibilities but I would suggest that the simplest for a beginner would be using XmlSerializer and a simple message envelope.
Create an XmlSerializable class either just using simple public properties or perhaps decorating with XmlElementAttribute, XmlRootAttribute etc.
Serialize to a MemoryStream
Write the bytes from the memory stream wrapped in an envelope (see later)
Receive a complete envelope into a byte array.
construct a MemoryStream from the byte array
Use XmlSerializer to reconstruct a copy of your original object.
The envelope is critical. The simplest one is just the binary length of the serialized object. Most protocols will typically extend that with CRC to handle possible corruption but since Ethernet uses a strong CRC and TCP is a reliable transport (albeit with a weak CRC) that is usually overkill. The key point that beginners miss is that TCP is a streaming protocol not a message based protocol thus it is perfectly possible for a sender to make a single write of say 1000 bytes and yet the receiver receives this as a number of smaller chunks. This is why you need some way to detect the end of a message such as using a length and why the receiver needs to accumulate received chunks until a complete message (and possibly part of the next) is received and can be deserialized.
This may seem complicated but unfortunately, at the TCP level, it doesn't get any simpler than that :(

The first question is how does the server identify that its communicating with an actual client, not someone else who's using the port, I've heard that browsers verify with servers using SHA hashing.
The server can identify different client by their IP addresses. See StreamReader.ReadToEnd
Second question is about the best way to send and receive data in variables, and also identifying which is which, because the current method of splitting data doesn't seem very elegant.
It depends on your protocol architecture, but a portable way to exchange values on network is to keep them in text format (this way no problem of endianness, type size...).
Said that, be careful of your variable separator : a '-' might be difficult to use with negative numbers, ' ' or ';' are more common.

You might want to define a communication protocol of some kind - a text-based protocol would be most straightforward to begin with - you can then read and write the "commands" each on a separate line.
First, there would be a "handshake", where the client would send something like "HELLO my-awesome-protocol-v1\n" and the server would respond similarly. This way you will be sure that the other person is a client, who understands the protocol or you can close a connection, which does not implement the protocol.
Then there could be some way of sending the values of variables with commands like "VAR variableName 123.45\n". You can read https://en.wikipedia.org/wiki/Text-based_protocol and see http://www.ncftp.com/libncftp/doc/ftp_overview.html for inspiration.

XML Delimiters over TCP/IP Socket

Ok, so I have done a little searching and found the answer to my question about how to properly frame my incoming XML data over my TCP socket is to either send the XML with a header indicating size beforehand, or use delimiting escape sequences. Since I am basically writing a client for a very old C server which consequently communicates with a number of other clients which I'm not interested in having to tinker with I chose the latter. Previously I was actually stripping the '\n' out of the stream because I was ignorant to what its purpose was. Now when my asynchronous OnDataReceived method is called I do get a nice chunk of data with the d elimiters, here is a sample picked from the stream and stored in a string variable:
<?xml version=\"1.0\"?><message><type>SERVER</type><user>TestDeleteOrKillMe</user> <cmd>PRIVATE_MSG</cmd><host>65.255.81.81</host><msg>4111|3.16C (UNIX)|</msg></message>\n<?xml version=\"1.0\"?><message><type>SERVER</type><user>TestDeleteOrKillMe</user> <cmd>PRIVATE_MSG</cmd><host>65.255.81.81</host><msg>4362|copyright 1993 by James D. Bennett|</msg></message>\n
My question is now what is the procedure for pulling the string values out of the individual XML "statements?" so that I can send these to my ParseMessage(string) method and start processing them? At first I was thinking of possibly some RegEx to maybe start at the beginning until it finds the first \n and then select and strip all of the previous text out? I'd like to hear some input on what the common(or uncommon) practices for this problem is before I begin some possibly overly convoluted approach. Much thanks.

C# best way to differentiate socket messages

Im new to sockets, and Im creating a tictactoe online, I know how to make the connections with the clients and the server, but I will make a chat too.
Then I doing this, when a user chat I send a message with a prefix "CHAT: HELLO WORLD"
and when a user make a move I send a message without the prefix... this is the best way?
THX!!!

In defining a wire protocol over a stream-based protocol like TCP, you have a few options for constructing messages:
Fixed-length
All messages are the same length; every sequence of x bytes represents a new message.
Length-prefixed (variable length)
The first byte(s) of the message represent the length of the payload to follow.
String-terminated (variable length)
Read bytes from the stream until you come to a specified byte-string that represents the end of a message, i.e. the newline character \n.
If you ever intend on changing the protocol (protip: you will, even if you don't think you will), it is crucial that you include an identifier for the protocol version in each message to prevent issues when dealing with clients using an older iteration of the protocol. Clearly, this is the first thing you must determine before deciphering the rest of the payload, so this should be the first byte(s) of the message (following any length-prefix) - how could we determine the version if we don't know where it is located in every message we receive?

Typically you would go with a format that includes a packet length, type and payload.
In your case you could go with a Byte (type), Int16 (length), Byte[] (payload).
The type can be represented in code as an enum. Length would just represent the length of the payload.
public enum Byte PacketType {
PlayerMove = 1,
PlayerChat = 2
}

You need to define a protocol. Remember to allow room for additional features :-).
Eg. using regular expressions over complete lines (end with selected line terminator):
Matching ^:[a-c][1-3]:: is a move (colon, position, colon user name).
Matching ^!.*?:: is a chat message (exclamation point, name, colon, text).
and anything else (in V1) is an error.
Remember:
Data is sent in packets, you might need multiple reads from the socket to get a complete message.
Avoid ambiguity: resolving it might be x or y is hard.
Specify a text encoding (eg. UTF-8).

I assume you're using TCP?
You need to make sure you 'frame' both messages so you can identify them and also avoid potential blocking issues (in case the client stops sending while you are still expecting to read CHAT: or whatever you define). With TCP your byte order is guaranteed but reading does not guarantee a complete 'packet' so you'll need to implement some way of building up a buffer and identifying when your 'message' is complete.
A reasonably simple way of doing this is to make sure each 'message' has a header with the type and size specified.
EG:
Enumerate your message types (move and chat currently), so say 'chat' is 0x01 and your message is 1020 bytes. You can prefix your 'message' with 0x0103FC so the server knows how many bytes to expect, and build up a buffer using async socket calls until the 1020 bytes are read (or you arbitrarily decide that the client is not sending anymore)

RightFax C# through RFCOMAPILib - Attachments

I'm trying to send faxes through RightFax in an efficient manner.
My users need to fax PDFs and even though the application is working fine, it is very slow for bulk sending (> 20 recipients, taking abt 40 seconds per fax).
// Fax created
fax.Attachments.Add(#"C:\\Test Attachments\\Products.pdf", BoolType.False);
fax.Send();
RightFax has this concept of *Library Documents, so what I thought we could do was to store a PDF document as a Library Document on the server and then reuse it, so there is no need to upload this PDF for n users.
I can create Library Documents without problems (I can retrieve them, etc.), but how do I add a PDF to this? (I have rights on the server.)
LibraryDocument doc2 = server.LibraryDocuments.Create;
doc2.Description = "Test Doc 1";
doc2.ID = "568"; // tried ints everything!
doc2.IsPublishedForWeb = BoolType.True;
doc2.PageCount = 2;
doc2.Save();
Also, once I created a fax, the API gives you an option to "StoreAsNewLibraryDocument", which is throwing an exception when run. System.ArgumentException: Value does not fall within the expected range
fax.StoreAsNewLibraryDocument("PRODUCTS","the products");
What matters for us is how to send say 500 faxes in the most efficient way possible using the API through RFCOMAPILib. I think that if we can reuse the PDF attached, it would greatly improve perfomance. Clearly, sending a fax in 40 seconds is unacceptable when you have hundreds of recipients.
How do we send faxes with attachments in the most efficient mode through the API?

StoreAsNewLibraryDocument() is the only practical way to store LibraryDocuments using the RightFax COM API, but assuming you're not using a pre-existing LibraryDocument, you have to call the function immediately after sending the first fax, which will have a regular file (not LibraryDoc) attachment.
(Don't create a LibraryDoc object on the server yourself, as you do above - you'd only do that if you have an existing file on the server that isn't a LibraryDocument, and you want to make it into one. You'll probably never encounter such a scenario.)
The new LibraryDocument is then referenced (in subsequent fax attachments) by the ID string you specify as the first argument of StoreAsNewLibraryDocument(). If that ID isn't unique to the RightFax Server's LibraryDocuments collection, you'll get an error. (You could use StoreAsLibraryDocumentUpdate() instead, if you want to actually replace the file on the server.) Also, remember to always specify the AttachmentType.
In theory, this should be all you really have to do:
' First fax:
fax.Attachments.Add(#"C:\\Test Attachments\\Products.pdf", BoolType.False);
fax.Attachments.Item(1).AttachmentType = AttachmentType.aFile;
fax.Send();
fax.StoreAsNewLibraryDocument("PRODUCTS", "The Products");
server.LibraryDocuments("PRODUCTS").IsPublishedForWeb = BoolType.True;
' And for all subsequent faxes:
fax.Attachments.Add(server.LibraryDocuments("PRODUCTS"));
fax.Attachments.Item(1).AttachmentType = AttachmentType.aLibraryDocument;
fax.Send();
The reason I say "in theory" is because this doesn't always work. Sometimes when you call StoreAsNewLibraryDocument() you end up with a LibraryDoc with a PageCount of zero. This happens seemingly at random, and is probably due to a bug in RightFax, or possibly a server misconfiguration. So it's a very good idea to check for...
server.LibraryDocuments("PRODUCTS").PageCount = 0
...before you send any of the subsequent faxes, and if necessary retry until it works, or (if it won't) store the LibraryDoc some other way and give up on StoreAsNewLibraryDocument().
Whereas, if you don't have that problem, you can usually send a mass-fax in about a 1/10th of the time it takes when you attach (and upload) the local file each time.
If someone from OpenText/RightFax reads this and can explain why StoreAsNewLibraryDocument() sometimes results in zero-page faxes, an additional answer about that would be appreciated quite a bit!