Browse
Category
DiscussionsOff Topic

[Java] Getting information from a website.

Open
SnipahShot

It is something I find very useful but yet I can't seem to figure out how to do so..

I've been looking at Jsoup all day and having a bit of problem with it. I am trying to get game names from http://steam.tools/cards/ just to play around with it but can't manage to do so.
Anyone has a clue how to do so?

Basically the code starts like this:

String url = "http://steam.tools/cards/";
Document doc = Jsoup.connect(url).get();

Next should be getting elements but that part seems to be what fails for me.
Elements items = doc.select("span[class=game]");

9 years ago*

Comment has been collapsed.

9 Comments
uchristensen

Your actual hurdle is that the game names aren't actually in the HTML page itself (check the source, i.e. the document your program retrieved). The page is coded as a template, and JavaScript is used to populates the table of games via some JavaScript DOM-manipulating libraries. So you need to find out how and from where the page gets the list.

Most browsers today will tell a "lie", when you open the page in developer tools, because the HTML source shown is the generated HTML source. That is, it shows you how the DOM looks after the DOM-manipulating JavaScripts have run.

9 years ago
Permalink

Comment has been collapsed.
SnipahShot

Yeah, I actually tested it before and couldn't figure out why it fails to work. When I tried retrieving all the links from the page, it just gives generic ones like hub and feedback, log in and PayPal. Your point actually makes a lot of sense.

9 years ago
Permalink

Comment has been collapsed.
BackslashN

There you go: http://steam.tools/cards/set_data.json

Use a json parser or regexp the games out of there.

9 years ago
Permalink

Comment has been collapsed.
SnipahShot

That is what I call raw data. How did you find it though?

9 years ago
Permalink

Comment has been collapsed.
patetico

It's actually called JSON ("raw data" is usually used for the binary data and can't always be converted to text). It's really easy to learn it, but you don't have to: java most certainly have a parser to convert it to an object/dictionary/whateverjavauses.

And imo the easiest way to find it is to check the network tab on the developer tools on Firefox or Chrome. Both have a filter for XHR which should have this kind of stuff.

9 years ago
Permalink

Comment has been collapsed.
SnipahShot

Yeah, I actually looked it up but where ever I try, dev tools doesn't show any information that isn't written on the website itself.

9 years ago
Permalink

Comment has been collapsed.
uchristensen

Due to the way the script is written, it may not be detected. Hence you'd have to look for it manually by checking every linked script file. Naturally you can skip all the ones containing the JavaScript libraries (unless everything is combined into one JavaScript file, which isn't the case here).

You are looking for the glue, and in this case you'll find it in "main.js" included at line 188, and in "main.js" you'll find the request on line 354.

9 years ago
Permalink

Comment has been collapsed.
Noe

Posting this for EshanKia as he doesn't have an account here.

Hey guys, creator of steam.tools here to answer a few questions:

The reason the data doesn't show up if you download the page directly is because
the database is loaded afterwards through Javascript. As someone else pointed,
that JSON link is the database being loaded, so all the information should be there.
Dev Tools should also show it, it does for me: http://i.imgur.com/retamO6.png

There are various JSON parsers for Java. Google's gson is my personal favorite.
I'm not too knowledgable with Java though, I could help you more if you were working in Python.

If you want to tell me what exactly you're trying to do, I can point you towards
the right direction and where you can get the data you need. Use the "Feedback" button on the site to reach me.

9 years ago
Permalink

Comment has been collapsed.
SnipahShot

Yeah, I just realized my mistake.. I had something searched when I used the dev tools. Basically, what I am trying to do is match names of games with cards with a list of games Tremor Games has, mostly for myself. Kinda make a list of all the games they have that have cards. Mostly I do it for myself to practice getting data from websites.

9 years ago
Permalink

Comment has been collapsed.

Sign in through Steam to add a comment.