<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	>

<channel>
	<title>Official DoeDoe Blog</title>
	<atom:link href="http://www.doedoegames.com/blog/feed" rel="self" type="application/rss+xml" />
	<link>http://www.doedoegames.com/blog</link>
	<description></description>
	<pubDate>Sat, 11 Apr 2009 23:19:20 +0000</pubDate>
	<generator>http://wordpress.org/?v=2.7.1</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Evolving Challenges of Casual Games</title>
		<link>http://www.doedoegames.com/blog/evolving-challenges-of-casual-games</link>
		<comments>http://www.doedoegames.com/blog/evolving-challenges-of-casual-games#comments</comments>
		<pubDate>Sat, 11 Apr 2009 23:19:20 +0000</pubDate>
		<dc:creator>Dane Baker</dc:creator>
		
		<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.doedoegames.com/blog/?p=151</guid>
		<description><![CDATA[Once a race to produce the most visually realistic titles, today&#8217;s games industry is courting perhaps the most diverse audience ever — Nintendo&#8217;s Wii and DS handheld proved just how successful a game publisher can be in targeting this diverse audience, even to the detriment of the competition (witness Electronic Arts&#8217; recent troubles, which can [...]]]></description>
			<content:encoded><![CDATA[<p>Once a race to produce the most visually realistic titles, today&#8217;s games industry is courting perhaps the most diverse audience ever — Nintendo&#8217;s Wii and DS handheld proved just how successful a game publisher can be in targeting this diverse audience, even to the detriment of the competition (witness Electronic Arts&#8217; recent troubles, which can be blamed in part on their failure to embrace Nintendo&#8217;s platform).</p>
<p>Those who pick up a Wii controller or even a PSP handheld expect to devote a certain chunk of time to playing games; even the most casual titles deliver a certain level of experience and accompanying learning curve. Enter Apple&#8217;s iPhone, which is redefining the category even further: iPhone users rarely &#8220;sit down&#8221; to play an iPhone game, no matter how engaging — they play waiting in line at the post office or during a quiet moment at work or while writing blog posts (just kidding). This isn&#8217;t to say there&#8217;s no room for more structured, traditional games — Subatomic Studios&#8217; super-successful tower defense title Fieldrunners comes to mind — but for many iPhone developers, &#8220;casual&#8221; is taking on a new meaning.</p>
<p>Apple recently released a <a href="http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewCustomPage?name=page1BillionApps&#038;v0=WWW-NAUS-ITSBILLION-TOP20APPS">list of the top 20 paid apps</a> (link launches iTunes) — of those, 15 are games, most of which can be launched and delved into in mere seconds. Generally speaking, this has no effect on game quality, but it does present unique challenges to iPhone developers. Namely, creating an experience that is familiar with little or no learning curve yet interesting despite abrupt pauses in gameplay.</p>
<p>This early in the iPhone lifespan many games&#8217; uniqueness turns on interfaces and control schemes that utilize the iPhone&#8217;s unique features (accelerometer controlling character movement, for example). As the App Store begins to mature, developers will be able to rely on such simple solutions less and less; hardware features will change, as will user expectations. The challenge will be how best to provide a more varied experience — using the accelerometer to control one aspect of movement while touch gestures control another, for example — that can still be picked up and enjoyed almost instantly by the end user. Even now, with the third iPhone interation on the horizon, gamers expect a certain level of novelty; witness the tepid response of Konami&#8217;s Metal Gear Solid Touch, which provided simplistic touch-to-shoot gameplay that elicited mostly yawns from critics.</p>
<p>Obviously change of this sort will happen gradually as the iPhone market ages. The winners of tomorrow will be the developers who can adapt the quickest to meet these evolving challenges head on.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.doedoegames.com/blog/evolving-challenges-of-casual-games/feed</wfw:commentRss>
		</item>
		<item>
		<title>Scraping iTunes: Finishing Up</title>
		<link>http://www.doedoegames.com/blog/scraping-itunes-finishing-up</link>
		<comments>http://www.doedoegames.com/blog/scraping-itunes-finishing-up#comments</comments>
		<pubDate>Thu, 26 Mar 2009 21:03:22 +0000</pubDate>
		<dc:creator>Bill Soistmann</dc:creator>
		
		<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.doedoegames.com/blog/?p=137</guid>
		<description><![CDATA[Last week I started a series of posts detailing how I scraped the iTunes store and how you might do the same for your needs.
Today I will add just one last thing and we will call this a wrap. 

In order to grab the rating for another country&#8217;s iTunes store, you simply pass in a [...]]]></description>
			<content:encoded><![CDATA[<p>Last week I <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store#more-6">started a series</a> of posts detailing how I scraped the iTunes store and how you might do the same for your needs.</p>
<p>Today I will add just one last thing and we will call this a wrap. </p>
<p><span id="more-137"></span></p>
<p>In order to grab the rating for another country&#8217;s iTunes store, you simply pass in a numeric country code. To make things nice and user friendly, we can add a hash that maps each code to a string that makes more sense to us.</p>
<div class="code"><code>
<pre>
my %stores = (&quot;US&quot;, &quot;143441&quot;,
&quot;Australia&quot;, &quot;143460&quot;,
&quot;Brazil&quot;, &quot;143503&quot;,
&quot;Chile&quot;, &quot;143483&quot;,
&quot;Columbia&quot;, &quot;143501&quot;,
&quot;Croatia&quot;, &quot;143494&quot;,
&quot;Denmark&quot;, &quot;143458&quot;,
&quot;ElSalvador&quot;, &quot;143506&quot;,
&quot;Finland&quot;, &quot;143447&quot;,
&quot;Greece&quot;, &quot;143448&quot;,
&quot;HongKong&quot;, &quot;143463&quot;,
&quot;India&quot;, &quot;143467&quot;,
&quot;Ireland&quot;, &quot;143449&quot;,
&quot;Italy&quot;, &quot;143450&quot;,
&quot;Kuwait&quot;, &quot;143493&quot;,
&quot;Luxemburg&quot;, &quot;143451&quot;,
&quot;Mexico&quot;, &quot;143468&quot;,
&quot;NewZealand&quot;, &quot;143461&quot;,
&quot;Austria&quot;, &quot;143445&quot;,
&quot;Panama&quot;, &quot;143485&quot;,
&quot;Philippines&quot;, &quot;143474&quot;,
&quot;Portugal&quot;, &quot;143453&quot;,
&quot;Romania&quot;, &quot;143487&quot;,
&quot;SaudiArabia&quot;, &quot;143479&quot;,
&quot;Singapore&quot;, &quot;143464&quot;,
&quot;Slovenia&quot;, &quot;143499&quot;,
&quot;SriLanka&quot;, &quot;143486&quot;,
&quot;Taiwan&quot;, &quot;143470&quot;,
&quot;Turkey&quot;, &quot;143480&quot;,
&quot;UnitedKingdom&quot;, &quot;143444&quot;,
&quot;Vietnam&quot;, &quot;143471&quot;,
&quot;Belgium&quot;, &quot;143446&quot;,
&quot;China&quot;, &quot;143465&quot;,
&quot;Czech&quot;, &quot;143489&quot;,
&quot;Spain&quot;, &quot;143454&quot;,
&quot;Guatemala&quot;, &quot;143504&quot;,
&quot;Indonesia&quot;, &quot;143476&quot;,
&quot;Korea&quot;, &quot;143466&quot;,
&quot;Malaysia&quot;, &quot;143473&quot;,
&quot;Norway&quot;, &quot;143457&quot;,
&quot;Peru&quot;, &quot;143507&quot;,
&quot;Qatar&quot;, &quot;143498&quot;,
&quot;Switzerland&quot;, &quot;143459&quot;,
&quot;SouthAfrica&quot;, &quot;143472&quot;,
&quot;Thailand&quot;, &quot;143475&quot;,
&quot;Venezuela&quot;, &quot;1435021&quot;,
&quot;Canada&quot;, &quot;143455&quot;,
&quot;Germany&quot;, &quot;143443&quot;,
&quot;Hungary&quot;, &quot;143482&quot;,
&quot;Lebanon&quot;, &quot;143497&quot;,
&quot;Pakistan&quot;, &quot;143477&quot;,
&quot;Russia&quot;, &quot;143469&quot;,
&quot;Sweden&quot;, &quot;143456&quot;,
&quot;Japan&quot;, &quot;143462&quot;,
&quot;CostaRica&quot;, &quot;143495&quot;,
&quot;Israel&quot;, &quot;143491&quot;,
&quot;Poland&quot;, &quot;143478&quot;,
&quot;UnitedArabEmirates&quot;, &quot;143481&quot;,
&quot;France&quot;, &quot;143442&quot;,
&quot;Slovenia&quot;, &quot;143496&quot;,
&quot;Netherlands&quot;, &quot;143452&quot;,
&quot;Argentina&quot;, &quot;143505&quot;
);
</pre>
<p></code></div>
</p>
<p>Then we change this line</p>
<div class="code"><code>
<pre>
$coCode = $data{&#x27;country&#x27;};
</pre>
<p></code></div>
<p>to this</p>
<div class="code"><code>
<pre>
$coCode = ($data{&#x27;country&#x27;})?$data{&#x27;country&#x27;}:&quot;US&quot;;
</pre>
<p></code></div>
<p>and this line</p>
<div class="code"><code>
<pre>
$store = 143441;
</pre>
<p></code></div>
<p>to this</p>
<div class="code"><code>
<pre>
$store = $stores[$coCode];
</pre>
<p></code></div>
</p>
<p>That&#8217;s all there is to it. I saved this for last because of all the extra code, but it&#8217;s a very easy edit.</p>
<p>You can grab the <a href="http://doedoegames.com/getstars4.txt">current version</a> and give it a shot. Have fun.</p>
<p><em>This is the last post in a series of posts which start <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store">here</a>.</em></p>
]]></content:encoded>
			<wfw:commentRss>http://www.doedoegames.com/blog/scraping-itunes-finishing-up/feed</wfw:commentRss>
		</item>
		<item>
		<title>Scraping iTunes: Touch of Generalization</title>
		<link>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-4</link>
		<comments>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-4#comments</comments>
		<pubDate>Tue, 24 Mar 2009 03:06:10 +0000</pubDate>
		<dc:creator>Bill Soistmann</dc:creator>
		
		<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.doedoegames.com/blog/?p=121</guid>
		<description><![CDATA[At this point our script (if you haven&#8217;t been following along) gives us the xhtml for five images which we can include in our webpage. This works nicely but requires that we name and store our images a certain way. What if we want to use different image names?

In a &#8220;real world&#8221; situation it might [...]]]></description>
			<content:encoded><![CDATA[<p>At this point our script (<a href="">if you haven&#8217;t been following along</a>) gives us the xhtml for five images which we can include in our webpage. This works nicely but requires that we name and store our images a certain way. What if we want to use different image names?</p>
<p><span id="more-121"></span></p>
<p>In a &#8220;real world&#8221; situation it might be just as easy to rename the script, modify the images and path names; but we can generalize this a bit in just a few minutes. Follow along below and you should be able to take either approach.</p>
<p>
First, initialize a few more variables</p>
<div class="code"><code>
<pre>
my $star;
my $empty_star;
my $half_star;
my $image_path;
my $ext;
</pre>
<p></code></div>
<p>then add a few more lines to our <em>splitVars</em> function to see if options are passed in query string</p>
<div class="code"><code>
<pre>
$star=($data{&#x27;star&#x27;})?$data{&#x27;star&#x27;}:&#x27;star_on&#x27;;
$empty_star=$data{&#x27;empty_star&#x27;}?$data{&#x27;empty_star&#x27;}:&#x27;star&#x27;;
$half_star=$data{&#x27;half_star&#x27;}?$data{&#x27;half_star&#x27;}:&#x27;star_half&#x27;;
$image_path=$data{&#x27;image_path&#x27;}?$data{&#x27;image_path&#x27;}:&#x27;images&#x27;;
$ext=$data{&#x27;ext&#x27;}?$data{&#x27;ext&#x27;}:&#x27;gif&#x27;;
</pre>
<p></code></div>
</p>
<p>
Finally, edit the <em>printHTML</em> function. Find these four lines</p>
<div class="code"><code>
<pre>
    $thisimg=&#x27;star_on&#x27;;
    $thisimg=&#x27;star_half&#x27;;
    $thisimg=&#x27;star&#x27;;
	print &quot;&lt;img src=\&quot;images/$thisimg.gif\&quot; alt=\&quot;$t Star\&quot; /&gt;\n&quot;;
</pre>
<p></code></div>
<p>and change them to this</p>
<div class="code"><code>
<pre>
    $thisimg=$star;
    $thisimg=$half_star;
    $thisimg=$empty_star;
	print &quot;&lt;img src=\&quot;$image_path/$thisimg.$ext\&quot; alt=\&quot;$t Star\&quot; /&gt;\n&quot;;
</pre>
<p></code></div>
</p>
<p>Now you can pass in the following options via the query string</p>
<ul>
<li>image_path - directory containing your images</li>
<li>ext - extension for images</li>
<li>star - basename for full star</li>
<li>half_star - basename for half star</li>
<li>empty_star - basename for empty star (no star)</li>
</ul>
<p>Grab the whole script <a href="http://doedoegames.com/getstars3.txt">here</a> or give it a try at <a href="http://bsoist.freeshell.org/itunes/getstars3.cgi?id=304570595&#038;html=true&#038;ext=png&#038;image_path=my_images&#038;star=full_star&#038;half_star=half_star&#038;empty_star=empty_star">http://bsoist.freeshell.org/itunes/getstars3.cgi?id=304570595&#038;html=true&#038;ext=png&#038;image_path=my_images&#038;star=full_star&#038;half_star=half_star&#038;empty_star=empty_star</a></p>
<p><em>This is part of a series of posts which start <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store">here</a>. The next post is <a href="http://www.doedoegames.com/blog/scraping-itunes-finishing-up">here</a>.</em></p>
]]></content:encoded>
			<wfw:commentRss>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-4/feed</wfw:commentRss>
		</item>
		<item>
		<title>Scraping iTunes: Returning HTML</title>
		<link>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-3</link>
		<comments>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-3#comments</comments>
		<pubDate>Fri, 20 Mar 2009 19:50:26 +0000</pubDate>
		<dc:creator>Bill Soistmann</dc:creator>
		
		<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.doedoegames.com/blog/?p=13</guid>
		<description><![CDATA[Today I want to take a look at how to modify our Perl script so that it returns the html we need instead of a number. The current version of our script can be found here. Before I get ahead of myself, I want to mention that we will continue to ignore the country code [...]]]></description>
			<content:encoded><![CDATA[<p>Today I want to take a look at how to modify our Perl script so that it returns the html we need instead of a number. The current version of our script can be found <a href="/getstars.txt">here</a>. Before I get ahead of myself, I want to mention that we will continue to ignore the country code for now. Adding an option for grabbing the data from another iTunes store adds quite a bit of code so I will cover that in my last post.
</p>
<p>
	So what we want to do today is add an option for grabbing the data as html instead of a number. It makes sense to leave the number as an option in the event someone still wants to use it for that purpose. At this point, we know we can add two arguments to the URL</p>
<p><span id="more-13"></span></p>
<ol>
<li>country=US - which the script currently ignores </li>
<li>id=304570595 - the application id</li>
</ol>
<p>
For our new addition we will use an argument named <em>html</em> and pass in &#x27;true&#x27; for true and anything else (including nothing) for false. The script will assume false if no argument is passed in.
</p>
<p>While we are at it, let&#8217;s go ahead and add an option for half stars.</p>
<p>We start by adding these lines</p>
<div class="code">
		<code></p>
<pre>
my $htmlformat;
my $halves;
			</pre>
<p>		</code>
	</div>
<p>with the rest of the variable assignments. Then, we add more to our <em>splitVars</em> function. This should do it</p>
<div class="code">
		<code></p>
<pre>
elsif(index($item, &quot;html&quot;, 0) &gt;= 0){
	$htmlformat = substr($item, index($item, &quot;html&quot;, 0) + 5, length($item));
}
			</pre>
<p>		</code>
	</div>
</p>
<p>Now is a good time to improve on the code that grabs the arguments. The current version is fine for a couple of args but we can see that it will quickly become unwieldy. Let&#8217;s clean it up a bit and rewrite the <em>splitVars</em> function as follows:</p>
<div class="code">
		<code></p>
<pre>
sub splitVars{
	if (length ($ENV{&#x27;QUERY_STRING&#x27;}) &gt; 0){
	      my $buffer = $ENV{&#x27;QUERY_STRING&#x27;};
	      my @pairs = split(/&amp;/, $buffer);
	      foreach my $pair (@pairs){
	           (my $name, my $value) = split(/=/, $pair);
	           $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack(&quot;C&quot;, hex($1))/eg;
	           $data{$name} = $value;
	      }
		$currentSoftware = $data{&#x27;id&#x27;};
		$htmlformat = $data{&#x27;html&#x27;};
		$coCode = $data{&#x27;country&#x27;};
		$halves = $data{&#x27;halves&#x27;};
	 }
}
		</pre>
<p>		</code>
	</div>
</p>
<p>
Now we can change the main logic a bit so we have this:</p>
<div class="code">
		<code></p>
<pre>
print &quot;Content-type: text/html\n\n&quot;;
splitVars();
$store = 143441;
getReviews();
if($htmlformat eq &#x27;true&#x27;) {
	printHTML();
} else {
	if ($halves eq &#x27;true&#x27;) {
		print $result * 2;
	} else {
		print $result;
	}
}
			</pre>
<p>		</code>
	</div>
<p>Well, now we can clean up some variables we don&#8217;t need but we also have a function we haven&#8217;t written yet - <em>printHTML</em>.
</p>
<p>Let&#8217;s get the function done and then I will show you our new version. This function will do exactly what we did with the PHP - with differences in syntax, of course. We end up with something like this.</p>
<div class="code">
		<code></p>
<pre>
sub printHTML{
    my $t; my $thisimg;
	$result=$result*2;
	my $half=$result%2;
	$result=$result/2;
	for (my $i=0;$i&lt;5;$i++) {
                if($result&gt;0) {
                    $thisimg=&#x27;star_on&#x27;;
                    if($half==1) {
                         $thisimg=&#x27;star_half&#x27;;
                         $t=&#x27;Half&#x27;;
                         $half=0;
                    }
                } else {
                    $t=&#x27;No&#x27;;
                    $thisimg=&#x27;star&#x27;;
                }
		print &quot;&lt;img src=\&quot;images/$thisimg.gif\&quot; alt=\&quot;$t Star\&quot; /&gt;\n&quot;;
		$result--;
	}
}
			</pre>
<p>		</code>
	</div>
</p>
<p>What we have now is a version with several options. </p>
<ol>
<li> id - REQUIRED - pass in app id</li>
<li>country - OPTIONAL - pass in country code - default is US</li>
<li>halves - OPTIONAL - pass in true for number of half stars - default is false, has no effect if next argument is true</li>
<li>html - OPTIONAL - pass in true for html instead of a number - default is false</li>
</ol>
<p>Grab the whole thing <a href="http://doedoegames.com/getstars2.txt">over here</a> if you&#8217;d like. </p>
<p><em>This is part of a series of posts which start <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store">here</a>. The next post is <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store-part-4">here</a>.</em></p>
]]></content:encoded>
			<wfw:commentRss>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-3/feed</wfw:commentRss>
		</item>
		<item>
		<title>Scraping iTunes: Deciding Direction</title>
		<link>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-2</link>
		<comments>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-2#comments</comments>
		<pubDate>Thu, 19 Mar 2009 15:18:58 +0000</pubDate>
		<dc:creator>Bill Soistmann</dc:creator>
		
		<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.doedoegames.com/blog/?p=9</guid>
		<description><![CDATA[I finished off my last post by sharing a Perl script which will output the number of stars for an application given the app id. Today I will explain what I hoped to accomplish when I came up with the idea and then the direction in which I ended up moving. Feel free to grab [...]]]></description>
			<content:encoded><![CDATA[<p>I finished off my <a href="http://www.doedoegames.com/blog/?p=6">last post</a> by sharing a Perl script which will output the number of stars for an application given the app id. Today I will explain what I hoped to accomplish when I came up with the idea and then the direction in which I ended up moving. Feel free to grab the <a href="/getstars.txt">script</a> and use it for your own purposes.</p>
<p>
For the sake of demonstration, let&#8217;s look at an example using 3&#189; stars.</p>
<p><span id="more-9"></span></p>
<p><strong>UPDATE:</strong><br />
I obviously neglected to cover an important point when I posted this. For this to work, I will need to modify our Perl script to return the number of half stars. I did that in my script with </p>
<div class="code">
		<code></p>
<pre>
print $result*2;
			</pre>
<p>		</code>
	</div>
</p>
<p>but I will add an option for this in Friday&#8217;s post.
</p>
<p>
Here is the html for the stars:</p>
<div class="code">
		<code></p>
<pre>
&lt;img src=&quot;images/star_on.gif&quot; alt=&quot;Star&quot; /&gt;
&lt;img src=&quot;images/star_on.gif&quot; alt=&quot;Star&quot; /&gt;
&lt;img src=&quot;images/star_on.gif&quot; alt=&quot;Star&quot; /&gt;
&lt;img src=&quot;images/star_half.gif&quot; alt=&quot;Half Star&quot; /&gt;
&lt;img src=&quot;images/star.gif&quot; alt=&quot;No Star&quot; /&gt;
			</pre>
<p>		</code>
	</div>
</p>
<p>We&#8217;re using PHP for this site so if I have the number of half stars, I can set up something like this:	</p>
<div class="code">
		<code></p>
<pre>
&lt;?php
$stars=&#x27;7&#x27;;
$half=$stars%2;
$stars=floor($stars/2);
for($i=0;$i&lt;5;$i++) {
	$thisimg=($stars&gt;0)?&#x27;star_on&#x27;:&#x27;star&#x27;;
	$t=($star==&#x27;star_on&#x27;)?&#x27;&#x27;:&#x27;No&#x27;;
	if ($thisimg==&#x27;star&#x27; &amp;&amp; $half==1) {
		$thisimg=&#x27;star_half&#x27;;
		$t=&#x27;Half&#x27;;
		$half=0;
	}
	echo &quot;&lt;img src=\&quot;images/$thisimg.gif\&quot; alt=\&quot;$t Star\&quot; /&gt;\n&quot;;
	$stars--;
}
?&gt;
			</pre>
<p>		</code>
	</div>
</p>
<p>So, now all I have to do is pull the star number before I do that and I&#8217;m all set. We replace this line</p>
<div class="code">
		<code></p>
<pre>
$stars=&#x27;7&#x27;;
			</pre>
<p>		</code>
	</div>
<p>With these two</p>
<div class="code">
		<code></p>
<pre>
$url=http://mydomain.com/path/to/script/getstars.cgi?id=304570595
$stars=file_get_contents($url);
			</pre>
<p>		</code>
	</div>
<p>So the whole thing looks like this</p>
<div class="code">
		<code></p>
<pre>
&lt;?php
$url=http://mydomain.com/path/to/script/getstars.cgi?id=304570595
$stars=file_get_contents($url);
$half=$stars%2;
$stars=floor($stars/2);
for($i=0;$i&lt;5;$i++) {
	$thisimg=($stars&gt;0)?&#x27;star_on&#x27;:&#x27;star&#x27;;
	$t=($thisimg==&#x27;star_on&#x27;)?&#x27;&#x27;:&#x27;No&#x27;;
	if ($thisimg==&#x27;star&#x27; &amp;&amp; $half==1) {
		$thisimg=&#x27;star_half&#x27;;
		$t=&#x27;Half&#x27;;
		$half=0;
	}
	echo &#x27;&lt;img src=&quot;images/$thisimg.gif&quot; alt=&quot;$t Star&quot; /&gt;\n&#x27;;
	$stars--;
}
?&gt;
			</pre>
<p>		</code>
	</div>
</p>
<p>If you have any experience with this sort of thing, you will know that there are a couple of things to consider. The first is that it may make more sense to return the html instead of the number in the first place and include that html in the page. This is very easy to do in PHP. The second, more important, consideration is that this will call the getstars script AND request data from the iTunes store every time someone visits our page. I wanted to post this info today in case any of you can benefit from it, but I will cover the first of these considerations tomorrow.</p>
<p><em>This is part of a series of posts which start <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store">here</a>. The next post is <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store-part-3">here</a>.</em></p>
]]></content:encoded>
			<wfw:commentRss>http://www.doedoegames.com/blog/scraping-the-itunes-store-part-2/feed</wfw:commentRss>
		</item>
		<item>
		<title>Scraping iTunes: Setup</title>
		<link>http://www.doedoegames.com/blog/scraping-the-itunes-store</link>
		<comments>http://www.doedoegames.com/blog/scraping-the-itunes-store#comments</comments>
		<pubDate>Wed, 18 Mar 2009 04:35:33 +0000</pubDate>
		<dc:creator>Bill Soistmann</dc:creator>
		
		<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.doedoegames.com/blog/?p=6</guid>
		<description><![CDATA[With the release of Bailout America we wanted to find a way to display the game&#8217;s App Store rating on the website without editing it manually, so we decided to scrape the information from iTunes automatically. I figured if I could pull the number of stars as a numeral I could iterate and build out [...]]]></description>
			<content:encoded><![CDATA[<p>With the release of <a href="http://doedogames.com/bailout.php">Bailout America</a> we wanted to find a way to display the game&#8217;s App Store rating on <a href="http://doedogames.com/bailout.php">the website</a> without editing it manually, so we decided to scrape the information from iTunes automatically. I figured if I could pull the number of stars as a numeral I could iterate and build out the star images.</p>
<p>What I&#8217;ve ended up with is much more and I thought someone else might benefit from it. Over the next several days I will post details about how I ended up with our current version as well as some idea how you can put it to use for your own benefit. Today we will start with the basics - how to scrape the iTunes store.</p>
<p>The first step is to decide what data we need and on what page we can find it. We launch iTunes and browse to our application. We find the link that reads <em>4 Reviews for all versions</em> and click it which takes us to a page that shows the number of stars near the top right hand side of the page. If I can grab that, we&#8217;ll be in business. So, we click the back button, find that link again, control-click on it and copy the iTunes URL.
</p>
<p><span id="more-6"></span></p>
<p><img src="/blog-images/copyurl.gif" alt="Copy URL" /></p>
<p>Then we launch a text editor and paste in the URL and not that the important part of the URL is the application id which you find in the query string:</p>
<div class="code">
		<code>http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewContentsUserReviews?id=<span class="note">304570595</span>&#038;pageNumber=0&#038;sortOrdering=1&#038;type=Purple+Software</code>
	</div>
</p>
<p>Now we use curl and see what kind of data we get from iTunes. We will use two command line options to ensure we can grab the data:</p>
<ol>
<li>-A<br/><br />
			to set the User-Agent.<br/><br />
			I use iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)
		</li>
<li>-H<br/><br />
			to send another header<br/><br />
			X-Apple-Store-Front: 143441-1<br/><br />
			143441 is the country code for the U.S.
		</li>
</ol>
<div class="code">
		<code>curl -A "iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2" -H "X-Apple-Store-Front: 143441-1" 	  'http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/viewContentsUserReviews?id=304570595&#038;pageNumber=0&#038;sortOrdering=2&#038;type=Purple+Software' > ~/Desktop/foo.html</code>
	</div>
</p>
<p>Now we open ~/Desktop/foo.html in our favorite text editor and take a look at lines 667 - 674:</p>
<div class="code">
		<code>
<pre>&lt;TextView topInset=&quot;0&quot; truncation=&quot;right&quot; leftInset=&quot;0&quot; styleSet=&quot;basic13&quot; textJust=&quot;left&quot; maxLines=&quot;1&quot;&gt;&lt;SetFontStyle normalStyle=&quot;textColor&quot;&gt;<span class="note">Average rating for all versions: </span>&lt;/SetFontStyle&gt;&lt;/TextView&gt;

 &lt;View alt=&quot;&quot; stretchiness=&quot;1&quot;&gt;&lt;/View&gt;
&lt;/VBoxView&gt;
   &lt;VBoxView alt=&quot;&quot;&gt;
 &lt;View alt=&quot;&quot; stretchiness=&quot;1&quot;&gt;&lt;/View&gt;

     &lt;HBoxView <span class="note">rightInset=&quot;6&quot; alt=&quot;4 stars&quot;</span> leftInset=&quot;6&quot;&gt;</pre>
<p></code>
	</div>
</p>
<p>What we are after is the part that reads <code class="fragment">4 stars</code> on line 674. We could use regular expressions or some other technique to pull out what we need. Easiest thing in this case is to split the file up. Let&#8217;s use Perl for this. First we call use curl as we did before and assign the output to a variable by using the backticks. We will also throw in the -s option to suppress the progress indicators:</p>
<div class="code">
			<code>
<pre>my $riz = `curl -s -A &quot;iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2&quot; -H &quot;X-Apple-Store-Front: 143441-1&quot; &#x27;http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/viewContentsUserReviews?id=304570595&amp;pageNumber=0&amp;sortOrdering=2&amp;type=Purple+Software&#x27;`;	</pre>
<p></code>
		</div>
</p>
<p>Then we split using the section <code class="fragment">Average rating for</code>. Some applications will have <code class="fragment">Average rating for all versions:</code> and others will have <code class="fragment">Average rating for the current version:</code> so by using just <code class="fragment">Average rating for</code> we will get what we want in all cases:</p>
<div class="code">
			<code>
<pre>$riz = substr($riz, index($riz, &#x27;Average rating for&#x27;, 0), 300);</pre>
<p></code>
		</div>
</p>
<p>So, we split it and keep the second half. Then, we split again just before the 4 by using <code class="fragment">rightInset=&quot;6&quot; alt=&quot;</code> and keep the second half. Next, we split at a simple <code class="fragment">"</code>, keep the first half and then grab the first character:</p>
<div class="code">
			<code>
<pre>$riz = substr($riz, index($riz, &quot;rightInset=\&quot;6\&quot; alt=\&quot;&quot;, 0)+20, 100);
$riz = substr($riz, 0, index($riz, &#x27;&quot;&#x27;, 0));
$result = substr($riz, 0, 1);</pre>
<p></code>
		</div>
</p>
<p>The only thing to do now is look for any half stars. If this app had 4&#189; stars, we would have found <code class="fragment">4 and a half stars</code> so we need to check for that and then print the result:</p>
<div class="code">
			<code>
<pre>if(index($riz, &#x27;and a half&#x27;, 1) &gt; 0){$result += 0.5;}
print $result;</pre>
<p></code>
		</div>
</p>
<p>Anytime we run the script we will get a numeral indicating the number of stars. We can throw in a bit more to generalize it so we can grab ratings for other apps or other stores. This should do the trick:</p>
<div class="code">
			<code>
<pre>#! /usr/pkg/bin/perl -w
#! /usr/pkg/bin/curl

use strict;
use warnings;

my $result = -1;
my $store;
my $coCode;
my $appId;
my $counter = 0;
my $currentSoftware;
my $temp;
my $flag = 0;

print &quot;Content-type: text/html\n\n&quot;;
splitVars();
$currentSoftware = $appId;
$store = 143441;getReviews();
print $result;		

sub splitVars{
my $item = &quot;&quot;;
my $riz = $ENV{&#x27;QUERY_STRING&#x27;};
if($riz){
	my @rizray = split(&quot;&amp;&quot;, $riz);
	foreach $item (@rizray){
		if(index($item, &quot;id&quot;, 0) &gt;= 0){
			$appId = substr($item, index($item, &quot;id&quot;, 0)+3, length($item));
		}
		elsif(index($item, &quot;country&quot;, 0) &gt;= 0){
			$coCode = substr($item, index($item, &quot;country&quot;, 0) + 8, length($item));
		}
	}
}
}

sub getReviews
{
    my $riz = &quot;test&quot;;
    $riz = `curl -s -A &quot;iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2&quot; -H &quot;X-Apple-Store-Front: <span class="note">$store</span>-1&quot; &#x27;http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/viewContentsUserReviews?id=<span class="note">$currentSoftware</span>&amp;pageNumber=0&amp;sortOrdering=2&amp;type=Purple+Software&#x27;`;

   $riz = substr($riz, index($riz, &#x27;Average rating for&#x27;, 0), 300);
   $riz = substr($riz, index($riz, &quot;rightInset=\&quot;6\&quot; alt=\&quot;&quot;, 0)+20, 100);
   $riz = substr($riz, 0, index($riz, &#x27;&quot;&#x27;, 0));
   if(length($riz) != 0){
	if($result == -1){$result = substr($riz, 0, 1);}
	else{$result += int(substr($riz, 0, 1));}
	if(index($riz, &#x27;and a half&#x27;, 1) &gt; 0){$result += 0.5;}
	$counter++;
   }
}</pre>
<p></code>
		</div>
</p>
<p>Now we save it as getstars.cgi, make it executable, and put it in our webspace and we&#8217;re all set. Hit the URL with http://yourdomain.com/path/getstars.cgi?id=304570595 and we get the number of stars.</p>
<p>Ignore the country code in the Perl for now — I will add more tomorrow to take advantage of that and I didn&#8217;t feel like stripping it out for today&#8217;s post.</p>
<p><em>This is part of a series of posts which start <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store">here</a>. The next post is <a href="http://www.doedoegames.com/blog/scraping-the-itunes-store-part-2">here</a>.</em></p>
]]></content:encoded>
			<wfw:commentRss>http://www.doedoegames.com/blog/scraping-the-itunes-store/feed</wfw:commentRss>
		</item>
	</channel>
</rss>
