Adding (some) actual route path waypoints data

[LOOHP]

I wonder whether this crawler is interested in adding support for actual route paths (waypoints data) into the crawling.

So far I've been able to scrap the waypoints of KMB routes via https://m4.kmb.hk:8012/api, with this data we'll be able to draw more beautiful routes on the map. (At least for routes that have waypoint data scrapable)

With that hkbus.app can gain another pretty feature as well, instead of only connecting all the stops with straight lines.

[chunlaw]

It is interesting, while there is a license concern for the data from KMB directly. Although the data is publicly available, it doesn't imply we could store/crawl/use it in our application. 🥹

[LOOHP]

I suppose license is the concern, although it is for non-commercial purposes. Not sure if there's any information on that.

Update: I don't think there's any information on KMB's website, the only way to know for sure might be by sending them an email. 😅

[chunlaw]

Not sure where the entry point is, would you have any related email address suggested?

[LOOHP]

For queries for KMB, their email would be [email protected]. (Their chat bot gives this email address for queries)

[chakflying]

Greetings! I just tried hkbus.app, was also disappointed by the lack of bus routes, opened mitm-proxy to find m4.kmb.hk:8012, and finally landed in this issue.

Has anyone asked and/or got any response? I really don't see any moral issue with doing this considering that

• this is literally public data, and
• it would have negligible cost to them, as this data is sent to every user everytime they query for a bus route.

[LOOHP]

There might not be any moral issues but there might be lots of potential copyright issues. The actuate route way-point data are not explicitly licensed for public use like the ETA APIs on data.gov.hk despite everyone can see the operators' websites and apps.
Moreover, running the risk of getting established apps that members of the public are already relying upon taken down isn't a good idea, it's not like a private project.

That said, I really hope the operators can open them up. Drawing pretty routes is great. If they could open up their "Solar Bus", "Electric Bus", and "Train/Bus Compartment Fullness" data that would be even better.

[chakflying]

Understood the concern.

I did a bit more digging and found that the same data is actually available from CSDI (data.gov.hk), which does have a free-to-redistrubute license.

The format is a bit more difficult to work with (natively in ESRI .gdb), and we would possibly need to build an API for query to make it useful, but would that be a more approachable path to start from?

[LOOHP]

Oh that's cool, and there seems to be API for it as well.

For example, KMB 1
We just have to plug in the gtfsId to this API endpoint, but we'll have to figure out which direction is which.

https://api.csdi.gov.hk/apim/dataquery/api/?id=td_rcd_1638844988873_41214&layer=fb_route_line&limit=10&offset=0&ROUTE_ID=<gtfsId>

There seem to be KMB, CTB & NLB routes, no MTR Bus, GMB & Trains tho. But that's a good start.

[chakflying]

Different direction uses a different ROUTE_SEQ. Looks like 1 is for outbound and 2 is for inbound, but I did not verify if it's true for all routes.

[chakflying]

Did a very quick POC and it worked fine! But I'm not sure how the app works and how would I go about obtaining the gtfsId from the RouteMap component.

[chunlaw]

PR is always welcome

[chunlaw]

@chakflying You may add the gtfsId as a props for RouteMap, and then in RouteEta.tsx, you may call it as

  const { route, stops, co, orig, dest, fares, gtfsId } = routeListEntry;
...
  <RouteMap
    stops={stopsExtracted}
    stopIdx={stopIdx}
    onMarkerClick={onMarkerClick}
    gtfsId={gtfsId}
  />
...

[chakflying]

PR open in hkbus/hk-independent-bus-eta#78.