Encyclopaedia Metallum: The Metal Archives

Message board

* FAQ    * Register   * Login 



Reply to topic
Author Message Previous topic | Next topic
doomedlord
Mallcore Kid

Joined: Fri May 13, 2011 6:33 am
Posts: 2
Location: Ireland
PostPosted: Fri May 13, 2011 6:35 am 
 

Hi,

I was wondering if there's an open/public API available for Metal Archives? Searching, retrieving band & discography information, etc.

Thanks,
B.

Top
 Profile  
Evenfiel
Heavy Metal Hunter

Joined: Tue May 27, 2003 9:50 am
Posts: 4619
Location: Brazil
PostPosted: Fri May 13, 2011 7:28 am 
 

So far there is only an Android API:
http://www.androidzoom.com/android_appl ... _fzey.html

Top
 Profile  
doomedlord
Mallcore Kid

Joined: Fri May 13, 2011 6:33 am
Posts: 2
Location: Ireland
PostPosted: Fri May 13, 2011 9:13 am 
 

What I want to know is: how does the Android app access the Metal Archives data? Is it through 1.) connecting directly to the MI.com database, 2.) screen scraping or 3.) querying an API? If it's the latter, I would like to know how I could go about querying it.

Top
 Profile  
Evenfiel
Heavy Metal Hunter

Joined: Tue May 27, 2003 9:50 am
Posts: 4619
Location: Brazil
PostPosted: Fri May 13, 2011 11:24 am 
 

Oh, I see. If you're serious programming an app for MA, you should talk to Morrigan and / or Hellblazer.

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Fri May 13, 2011 11:39 am 
 

... though I should nonchalantly point out I'm already working on an app for the iPhone.

Just putting that out there. 0=)

Top
 Profile  
Merlinthewizard
Mallcore Kid

Joined: Sun Aug 21, 2011 8:00 am
Posts: 13
Location: Germany
PostPosted: Sun Aug 21, 2011 8:02 am 
 

I want an API too!

Top
 Profile  
HellBlazer
Metalhead

Joined: Sat Aug 10, 2002 6:48 am
Posts: 2121
Location: Canada
PostPosted: Sun Aug 21, 2011 6:25 pm 
 

There is no API at present time. The Android app just parses the pages to get its data. We will probably have a formal API at some point in the future but I couldn't really say when; the priority is always on improving the site itself.

Top
 Profile  
Merlinthewizard
Mallcore Kid

Joined: Sun Aug 21, 2011 8:00 am
Posts: 13
Location: Germany
PostPosted: Fri Aug 26, 2011 6:06 pm 
 

And exactly there is the problem: if you change the site just a bit, like adding a new color, that won't work anymore.

I would be happy help you to develop an API.

Top
 Profile  
Cursarion
Metalhead

Joined: Mon Nov 21, 2005 6:56 am
Posts: 785
Location: Finland
PostPosted: Fri Aug 26, 2011 8:53 pm 
 

^ Umm... You can't parse information from a page if its colour is changed? Yeeaah.
_________________
Empä mie semmone ou niiku sie luulet

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Sat Aug 27, 2011 1:07 am 
 

Quote:
And exactly there is the problem: if you change the site just a bit, like adding a new colour, that won't work anymore.

Uhm...
Quote:
I would be happy help you to develop an API.

Erh, mate... not to sound judgemental, but if you write your code based on ad-hoc structure, you're in no position to offer help with coding. o0;

Top
 Profile  
Merlinthewizard
Mallcore Kid

Joined: Sun Aug 21, 2011 8:00 am
Posts: 13
Location: Germany
PostPosted: Sat Aug 27, 2011 3:31 am 
 

An example:

There are only some methods to parse a website.
You can handle a website just as text with 'marks'(links and other information) that you wnat to get.
With that you have to search then for the next site.
So... if you search with an regular expression you are able to get smth like: <dd>active</dd> - thats easy
But if the site will change just a bit to make it more dynamic like it is now <dd class="active">Active</dd>
then you can't parse anymore for the bandstatus.

Also you have to get if you search for a band the answer from the server which comes with JSon. Sure again regularexpressions but in that case you must be sure that the link structure hasn't changed... Etc. etc. etc....

So Alhadis tell he how should i wrote code not in ad-hoc structure? Sure i could parse JavaScript too but that would change that much.

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Sat Aug 27, 2011 4:00 am 
 

Quote:
But if the site will change just a bit to make it more dynamic like it is now <dd class="active">Active</dd>

Then simply equip your RegExp to deal with optional attributes. E.g,
<dd([^\x00]+)?>(\w+)</dd>

In either case, you probably shouldn't be looking so closely at the page's source to extract your data. Simply breaking apart the text content of <div id="band_info"> .. </div> into recognised property values using <dt><dd> would be my approach. :)

If the structure changes in future, than it's your responsibility to remain vigilant and refactor your code. But you're right, we do need a formal API to make the job easier and less hackish.

Top
 Profile  
Merlinthewizard
Mallcore Kid

Joined: Sun Aug 21, 2011 8:00 am
Posts: 13
Location: Germany
PostPosted: Sun Aug 28, 2011 8:03 am 
 

@Alhadis thats how i did it with the old Excyclopedia Medtallum Design (That was quite Simple :))

I'll now try to parse a site with a DOM Parser...

Top
 Profile  
Alhadis
Madder Max

Joined: Fri Sep 05, 2008 8:35 am
Posts: 4014
Location: Australia
PostPosted: Sun Aug 28, 2011 8:11 am 
 

And exactly how much has changed, structure-wise?

Swap around a few variables, c'mon, big deal. Image

Top
 Profile  
Merlinthewizard
Mallcore Kid

Joined: Sun Aug 21, 2011 8:00 am
Posts: 13
Location: Germany
PostPosted: Mon May 07, 2012 3:13 am 
 

My Java-API: https://code.google.com/p/metalarchives/

Top
 Profile  
GOREG4SM
Mallcore Kid

Joined: Sun Sep 16, 2007 4:24 pm
Posts: 3
PostPosted: Wed Jan 25, 2017 6:34 pm 
 

I realize this topic is five years old now, but it would be great to have an API so you could do things like get a listing of all bands of a genre, sort that list based on average reviews, and then filter out any bands below the 80% mark. Just an idea. I would really like to use the data but at this point I would just have to make a crawler and that would get messy fast.

Top
 Profile  
heavymetalbunny
Mallcore Kid

Joined: Sun Nov 15, 2015 9:50 am
Posts: 1
Location: Belgium
PostPosted: Mon Mar 13, 2017 4:10 am 
 

I'm currently writing a metal music app which is using Metal Archives as one of the data sources. I've built a rudimentary scraper [1], but it would be great to have proper API access to the website. +1

[1] https://github.com/floriandejonckheere/metal_archives

Top
 Profile  
HellBlazer
Metalhead

Joined: Sat Aug 10, 2002 6:48 am
Posts: 2121
Location: Canada
PostPosted: Fri Mar 17, 2017 4:18 pm 
 

Someone made this recently as well: http://em.wemakesites.net/

Top
 Profile  
Merlinthewizard
Mallcore Kid

Joined: Sun Aug 21, 2011 8:00 am
Posts: 13
Location: Germany
PostPosted: Sat Jan 02, 2021 8:05 pm 
 

https://github.com/Loki-Afro/metalarchives

works very stable now for nearly 10 years, received some updates recently

Top
 Profile  
Hellow2
Mallcore Kid

Joined: Sun Dec 12, 2021 12:49 pm
Posts: 8
PostPosted: Sun Dec 12, 2021 12:52 pm 
 

Hello,
I wanna make a "big data" project, where I want to create a whole map of all metal genres and their relation to each other. Ofc I could do this with web scraping using selenium or whatever, but an API would be really handy.
Anyway. I wanna use python, If anyone wanna help with this project, just let me know and reply here.

:)

Edit: If you want to use something that behaves like a basic REST API, then take a look at this.

https://www.metal-archives.com/search/ajax-advanced/searching/bands/?bandName=&genre=&country=&yearCreationFrom=&yearCreationTo=&bandNotes=&status=&themes=&location=&bandLabelName=&sEcho=1&iColumns=3&sColumns=&iDisplayStart=0&iDisplayLength=200&mDataProp_0=0&mDataProp_1=1&mDataProp_2=2&_=1639479306089is

Metal Archives uses this to perform the advanced search, so it should be fairly quick.
If I break any rules, please let me know, and I will write a blog article about that in the future.

Top
 Profile  
mihies
Mallcore Kid

Joined: Fri Nov 08, 2019 6:10 am
Posts: 1
Location: Slovenia
PostPosted: Thu Jul 06, 2023 4:24 am 
 

As a developer I'd say a proper API is required if ecosystem is supposed to happen. I can imagine applications and what not. Imagine following a bunch of bands and getting a push notification when a new album is out...
Scrapping is not the way to do it as it is too fragile and too resource consuming as others noted above.
OTOH if admins don't want that to happen, it's fine, but it should be clearly stated I guess.

Top
 Profile  
Display posts from previous:  Sort by  
Reply to topic


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

 
Jump to:  

Back to the Encyclopaedia Metallum


Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group