Giving up on NextBus

My bus arrival timings web-app, NextBus, has been performing very poorly for the past several weeks. I've tried everything I can to fix it but unfortunately I can't. Given the circumstances, I have decided to stop further development on the app.

I'll try and describe why the site is broken but the summary is this: I have reason to believe that SBS Transit is rejecting network connections from my app to its servers. In the face of this, there is no way that I can keep NextBus functional.

To regular users who have come to rely on NextBus, I am sorry for letting you down. I have also let down the folks behind third-party apps such as iSingeo and SG Buses which relied on NextBus. I feel really bad that they spent time integrating their applications with NextBus which is now a waste.

I'll leave the site running and it will at least work for the 200 odd stops for which arrival information is available via PublicTransport@SG. iPhone users: you may want to try tranSGuide, an app that shows bus arrival timings. It's developed by a friend and it still works :-)

The Details

As you know, the core functionality of NextBus is to provide you with arrival information of all buses at any bus stop. To do this, it has to communicate with the SBS Transit mobile iris website which provides the information, although in a rather inconvenient manner.

Starting sometime in the last week of March, several connection requests to the SBS Transit servers from the Google App Engine environment (the infrastructure that hosts NextBus), started timing out. I didn't notice it initially since the stops that I personally use NextBus for are handled by the Public Transport@SG service, which NextBus also supports. When I did notice the problem from the server logs, I didn't have sufficient time to investigate the cause since I was to leave town for a week. So the site suffered while I was away.

When I got back, it took me a day to figure out what the problem was (requests to SBS servers timing out) and another day to implement a work-around. My work-around was simple - I implemented a simple web proxy that just fetches pages from SBS servers and returns them as is. I put up this proxy on an Amazon EC2 server instance hosted in the US. I modified the NextBus code to request pages from this proxy instead of requesting them directly from SBS servers. This worked! SBS pages were flowing back to NextBus via the proxy. So I quickly put this new proxy-powered code into production and the NextBus site was working reliably again. I thought it was kind-of weird that direct connections from the App Engine environment were not working but connections from Amazon servers were working. But hey, it was working now! So I didn't give it much thought.

Until a week later.

A week after implementing the proxy, requests from the Amazon EC2 server hosting the proxy also started timing out! I then moved the proxy service to my shared hosting server; the same server that shows you this blog. So now requests were going from NextBus to my shared hosting server to SBS servers and all the way back. NextBus was working again.

This lasted for three days.

Interestingly enough, when the proxy service on my shared hosting server stopped working, it wasn't because of connections to SBS timing out. No, the connection attempts were simply getting dropped! They still are, as a matter of fact.

Then I asked couple of my friends to help me out. They kindly agreed to host the proxy service on their servers. You can guess how that went.

Given what I've seen so far, I think the reasonable conclusion to draw is that the folks at SBS are denying connections from NextBus. It may be intentional or it may just be an automated response by some network monitoring tool that is banning any IP that originates an unusually high number of requests. Either way, I don't have an endless supply of proxy servers at my disposal to try and work around the connection bans. And even if I did have that supply, I have no intention of actually doing it since it makes no sense to go on with NextBus when SBS does not like what I am doing here.

So to all of NextBus' users, so long and thanks for all the fish!

Comments

Thanks for all the effort and trying to get the workarounds.

said midnightjazz at 11:16 a.m. on 15th May, 2009 ¶

This is very sad. Have you tried emailing the SBS Transit folks with these details?

said Ashwin Nanjappa at 11:48 a.m. on 15th May, 2009 ¶

It was great while it lasted.

If SBS Transit are blocking as you suspect, I wonder what sort of perverted logic would make them obstruct something that they got for free and that provided a great service to their customers

said Singeo at 3:26 p.m. on 15th May, 2009 ¶

No I didn't. If someone can hook me up with a contact at SBS, I'd be glad to explain the situation to them.

said deepak at 4:15 p.m. on 15th May, 2009 ¶

It may not be something intentional or malicious on SBS' part.

It is very likely that the number of requests to SBS mobileiris that originate from a single NextBus IP are way-way more than the average number of requests per IP. So if they are running some networking monitoring gear, it could interpret this as a DoS attack and automatically trigger a ban.

Of course, sitting outside, we can only speculate on the reason.

PS: Where have you been? No updates on singeo.com.sg in a while!

said deepak at 4:21 p.m. on 15th May, 2009 ¶

As I post, NextBus is working! SBS unblock?

said choozm at 4:36 p.m. on 15th May, 2009 ¶

Which stop number? It is probably on of the 214 stops for which info is available from PublicTransport@SG

said deepak at 4:43 p.m. on 15th May, 2009 ¶

Sorry to hear this. Why not try contacting SBS thru their website to see if anyone can help you with this.

http://www.sbstransit.com.sg/contact/hotline.aspx

Really hope they could resolve this issue with you asap.

said DK at 6:42 p.m. on 15th May, 2009 ¶

"Similarly, no derivative works shall be created without the prior written consent of SBS Transit"

http://www.sbstransit.com.sg/conditions.aspx

At least they haven't sent a cease and desist order.

said Hitesh at 10:54 p.m. on 15th May, 2009 ¶

Too bad the word "NextBus" is trademarked and you are infringing on that trademark. Pick a new name.

said NextBus not allowed at 6:50 a.m. on 16th May, 2009 ¶

perhaps a cease & desist letter would've been better. At least I wouldn't have spent hours implementing workarounds!

said deepak at 10:41 a.m. on 16th May, 2009 ¶

Is it really? Your comment made me curious and I looked up the IPOS site but didn't find any registration by SBS for NextBus. Check for yourself: http://tmsearch.ipos.gov.sg/eTMSearch/eRegister.jsp

said deepak at 10:53 a.m. on 16th May, 2009 ¶

Pack your app into an applet or swf or j2me midlet and let it run direct from the browser

said Thamkngw at 1:32 p.m. on 16th May, 2009 ¶

easier said than done!

said deepak at 2:27 p.m. on 16th May, 2009 ¶

All the more reason for you to get in touch with them. This is an app which is very useful for a lot of us, who are their customers after all.

said Ashwin Nanjappa at 8:11 p.m. on 17th May, 2009 ¶

Hi deepak, Thanks so much for the really useful app, sure better than wat mobileiris offers.. I've been an advocator of NextBus. It's sad that it's now limited to 214 stops, no wonder i've been getting errors. Thanks for coming up with such a great app, keep your spirits high, and don't just stop here.

said saffyz at 1:20 p.m. on 20th May, 2009 ¶

thanks for the work you did!

sadly that it would have ceast. no wonder i can't access my bus stops as usual =(

said zhu at 10:09 a.m. on 21st May, 2009 ¶

Supporter of nextbus. Everyone i show this app too find it useful. Sad that it had to end. now I have to go back to using the troublesome mobileiris

Why can't those people at SBS think of making their app to work like this in the 1st place?

said tiger4 at 4:27 p.m. on 22nd May, 2009 ¶

We need ppl like u!! selflessly setting up stuff that benefit the mass. It's a shame SBS can't or does mot want to tap on your knowledge (or maybe u may not even want to consider?!)

said Anonymous at 10:46 a.m. on 27th May, 2009 ¶

I see. I tried 16061 and 76139. Bus stop number 16061 is working with your NextBus. While for 76139 , it is 'retry'...

I guess 16061 info is from PublicTransport@SG.

Anyway, thank you for your effort!

said choozm at 1:15 p.m. on 27th May, 2009 ¶

Yes, PublicTransport@SG has good coverage of stops around NUS campus. 16061 used to be on my daily check list until last month :-)

said deepak at 3:05 p.m. on 27th May, 2009 ¶

Thanks for your efforts. I loved this site to show me the buses and their timing. Not so easy now...

said Brian at 2 p.m. on 5th August, 2009 ¶

Hey. It seems that the site's back up again. I'm testing random bus stops (without the LTA thingmajig) and it seems to be working.

said Richard at 12:31 p.m. on 20th August, 2009 ¶

Yeah, looks like it! What a pleasant surprise. Let's not get too optimistic though :-D

But if it stays up, it is incentive for me to put some time into switching the LTA stuff to use the new publictransport@sg mobile site - should be more reliable that way.

said antrix at 1:04 p.m. on 20th August, 2009 ¶

Hi! I just read this entry yesterday. It seems to be working now. May be you can update the status on the page.

Btw, I heard about your app from mrbrown and it is really a huge contrast from the official one.

Theses two versions give good illustrations of what is good and bad interface. Their version is probably by someone who doesn't take bus, or a Pri/Sec student who only takes one particular bus line, and hasn't mastered the loop structure in programming :)

said Back2Nature at 10:32 a.m. on 30th November, 2009 ¶

Yeah, I should update the status. Just been short of time - been developing some other new stuff :)

BTW, this is how I mastered loop structures when I was 8 years old, in Logo:

REPEAT 4 [FD 40 RT 90]

:-D

said deepak at 10:47 a.m. on 30th November, 2009 ¶