Harper Reed: “The power is not the mashup. It’s the data.”

Chicago Transit Authority | CTA Bus TrackerWhen most people go to the CTA Bus Tracker Web site and click on Estimated Arrival Times or the Bus Location Map, they see a way to find out when their next bus might get to the stop near them. Another nifty map mashup they can use to plot their trips daily.

When Harper Reed saw the same Web site last year, he saw every bus, everywhere, forevermore. And he wanted that data. “The power is not the mashup. It’s the data. The data is the answer,” says Harper.

So he set out to separate the data from the mashup. He used Firebug, the nifty Firefox add-on, to monitor that network traffic (the communication that goes between the browser window and the server) that was driving the data to the page.
This way, he found the URLs of the “endpoints” — a fancy word for URLs with special codes inside them — that control the entire Bus Tracker system.

Here’s a simple example:

http://ctabustracker.com/bustime/eta/eta.jsp?route=7&direction=West%20Bound&stop=5400%20W%20Harrison&showAllBusses=on

Breaking it down, element by element, this seemingly complicated URL is really just one machine (your computer) making a request of another machine (the CTA’s Bus Tracker server). It’s saying:

  • http://ctabustracker.com/: Hey, Bus Tracker
  • bustime/: I wanted to ask you about Bus Times
  • eta/: Specifically, some Estimated Times of Arrival for a specific
    route going in a specific direction for a specific bus stop, right
    about now
  • eta.jsp?: So anyway, here’s what I wanted to ask you:
  • route=7: On Route 7, the Harrison Bus
  • &direction=West%20Bound: For buses heading away from the lake
  • &stop=5400%20W%20Harrison: At the bus stop located at 5400 West Harrison
  • &showAllBusses=on: Can you please let me know when the next bus (any bus on that route) will get there?

CTA Bus Tracker "API"
Bingo. And he did that again and again, making all sorts of requests of the Bus Tracker system– just as a user, like you and me– and found all of the possible data types that are stored in the database and delivered to your browser, like a river that you can never step into twice. No matter what you do with Bus Tracker, no matter when or how you hit
it, all it’s doing is answering requests that are formed like this. It’s the data. The data is the answer.

After he found these endpoints, he wrote down what each of them are, what they mean, and what information they give you. The result is the CTA Bus Tracker “API”. Think of it  s a specification written after the fact. Like if someone came to your house, looked at your toaster real closely, and wrote down how it makes bread crispy and hot. And
it’s not the Toaster Engineer doing the writing, but some other guy.

Last Fall Harper and I met with some of the people at the Chicago Transit Authority
who are responsible for Bus Tracker and all other communication tools
there. They are open to developers working with their data. The woman
who escorted us up from the lobby had on the exact same purple color as
Harper, so I took a picture.

Harper Reed at the Chicago Transit Authority lobbyWe also attended a Coffee with Ron Huberman that my brother, Kevin O’Neil of CTA Tattler, put together. Ron showed us a prototype of the new Web site they were working on (and launched recently). One part of the Web site he showed us — it is not yet launched — is Developer Tools.

Speaking of developers, another thing Harper did was to set up a system for making regular queries to Bus Tracker and caching (storing) the results in a separate, stand-alone database over on a server that goes by the name of http://chicago.transitapi.com. This allows other developers to have a reliable database to ask questions of (see above) without having to ask the CTA all the time, and perhaps tax their servers or otherwise bother them. Like having a record of every instance of the river, stored forever.

The Chicago Bus Tracker Widget uses the Chicago Transit API to do cool things with this data. See also Chicago’s CTA Bus Tracker for iPhone and iPod Touch. I’d like to see more apps that use some of the more obscure data in the API. A “which bus should I take” app would be cool — just click to show where I am on a map and all the nearby bus stops with arriving buses (hat tip: Paul Smith).

One thing I noticed in looking at the documentation is that there are a lot of “stubs” for data in there that are not populated with useful information. Some examples:

  • Under “getRoutePoints”, there is a field for “serviceBulletins”. It would be nice to see info from the relevant RSS feed popped right in there
  • There is a field for “shortName”, but it always seems to be
    populated with the same number as “id”. It would be good to see the
    vernacular name for the bus here — the “Western Express” rather than “49X”
  • There is a field for “Length”, but always seems to be empty. Is
    this the length of the actual bus stop, in feet? The length of the
    route, in miles? Either way, that would be telling you something
  • Take a look at a getBusesForRoute example for the #8 Halsted.
    Great info on the bus — where it is (longitude/ latitude), who’s
    driving it (operator), where it’s going to end up (finalStop). I’d like
    to also know the last time it was cleaned, last time it was serviced,
    whether the bus was ever a part of an accident or a crime scene. Why
    not?
  • Under getTimeAndTemp (currently the coolest way to get this info), it would be nice to also pull whether there is currently any precipitation

There are other things not in the API that would be useful to developers and bus-taking decision makers.

  • Is there a canopy at that bus stop?
  • How about a seat?
  • What kind of seat?
  • When was this bus stop last cleaned?
  • When was graffitti last removed here?

As of January 26, eleven routes will be added to Bus Tracker: No. 3 King Drive, X3 King Drive Express, 4 Cottage Grove, X4 Cottage Grove Express, 22 Clark, 36 Broadway, 49B North Western, 82 Kimball/Homan, 93 California/Dodge, 96 Lunt and 97 Skokie. Huberman has said that one of the criteria for adding buses was stabilization of operations (no bus bunching and good on-time performance), so that’s good news for riders on these routes.

All hail the CTA Bus Tracker, the CTA, the CTA Bus Tracker “API”, and Harper Reed.

9 thoughts on “Harper Reed: “The power is not the mashup. It’s the data.”

  1. Great blog post! I’ve now bookmarked “Chicago’s CTA Bus Tracker for iPhone and iPod Touch.” I also heard ab/t an iPhone app that was about $.99 cents in the AppStore. I wondered who would pay when you could go to the online version for free. CTA’s mobile-ready version, while not iPhone-specific, was not that bad.
    🙂

  2. Oh, I forgot to answer zo.blue’s question about why someone would pay for this.
    I find the app faster to use than the website. It also has favorites so you can save any bus stop to the front page of the app. It’s slightly helpful when you don’t know which route goes to a bus stop, because the app shows you all the buses for that stop. That’s a decent feature, but I would like the reverse feature: Select the bus route, then select your bus stop because it takes a long time to find the bus stop you want in order to add to your favorites.

  3. I like it. One thing that would make the system more transparent is to bring back the bus productivity stat – passengers per platform hour. It’s interesting to note that CTA probably combines boarding information with the Tracker at some level. That would give planners and outsiders a way to know what routes and times are underserved, overserved, and other interesting things about the market and how to serve it.
    I was just reading about Winfield Mounds (the current subject of the Flickritis on this page). I wonder if they trail heading upward is going up one of the Indian Mounds.

  4. I know we’re in the infancy here, but I don’t see a point to replicating what the CTA’s doing — except as a means of learning, of course; that’s necessary, but there’s no need to release it to the public. And it would be nice if developers focused on web apps rather than downloadable apps restricted to a single platform.
    On my Sidekick I’ve bookmarked just about every stop I ever use, with unique identifiers — I just type “dcor” in its browser and the Sidekick knows I’m looking for the northbound Damen at Cornelia. “pmt” means I’m ready to start my evening commute looking for the 147 at the Tribune Tower. Most smartphone browsers are robust enough to handle this — it’s a waste to confine efforts to the iPhone and iPod touch.
    One extended feature I’d like to see: When you look at a list of stops on a route, the CTA displays them alphabetically; it would be great to see them in actual street order. Anytime someone’s on an unfamiliar route and not sure which stop is closest — or how far away their destination is — that would be a help.
    Cool writeup and great effort on the API.

  5. @ Sock Puppeteer: oops — fixed. Thanks!
    @ ryanwc: I’d love to see ridership stats in the feed — great idea. I don’t have any pictures of the acutal Winfield Mounds — the trail leading up goes to an abandoned bridge
    @ Bob S. — ya, the value is in explaining what’s available in the data, as well as trying get moar stuff from the CTA. Also, caching the data and publishing an API to it allows for developers to do stuff with it in a stable environment.
    Thanks for the comments!

  6. I am always lost but hope I can help a few folks find their way.

    It would be nice if we rebuilt a lot of things and made them do what we want them to do instead of what companies want them to do for us.

    Trying to work in that direction and it seems we may need to do some giant task such as getting wifi available quickly along all bus routes, building and maintaining a database of mac addresses latitudes and longitudes, building image repositories along all routes, building a system like the travel time and accident systems where folks riding buses or “L” give information such as escalator out, rapper in car three, red line goddess at belmont stop, crying baby bonanza on 6713 on the 80 and a communication between drivers and other ways to allow the flow between routes and “L” to minimize passenger wait time.

    My toe is barely in the water:

    https://github.com/randy7771026/Visual-CTA-Chicago

    and I hope a lot of other folks know how to swim in this tide.

Leave a Reply

Your email address will not be published. Required fields are marked *