Merge pull request #1 from TiberiuD/master

BodoMinea · web-flow · commit 2775556ebd0b · 2017-04-26T20:25:20.000+03:00
Added train information. Made some improvements.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,20 @@
+# Change Log
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/)
+and this project adheres to [Semantic Versioning](http://semver.org/).
+
+## [1.0.1] - 2017-04-26
+### Changed
+- The JSON format for the station information, for consistency. This format is not compatible with version 1.0.0.
+### Added
+- Better error management
+- Train information
+- This CHANGELOG file.
+### Fixed
+- Fixed a bug which was preventing the use of the node server from another directory
+- The server searches for PhantomJS and throws an error if it doesn't find it
+### Broke
+- Compatibility with the version 1.0.0 standard.
+
+## 1.0.0 - Initial release
diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@ CFR S.A., the Romanian national railway infrastructure administration company of
 
 This NodeJS & ExpressJS based API controls the PhantomJS headless webkit engine to extract data from the CFR webapp as requested in the URL API Endpoint and outputs reusable JSON data.
 
-### Installation & Requirments
+## Installation & Requirments
   - You need NodeJS and NPM installed on your system: [See their website for details](https://nodejs.org/en/download/). You may or may not have luck with portable versions.
   - Clone the repository and install the module dependencies:
 ```sh
@@ -18,23 +18,29 @@ $ npm install
 ```sh
 $ node .
 ```
-### Usage
+## Usage
+### Station information
 Now you can point your browser to http://localhost:9090/station/ID to see the magic. ID is the unique station-unit code, you have [a list of those in this very nice repo](https://github.com/vasile/data.gov.ro-gtfs-exporter/blob/master/cfr.webgis.ro/stops.geojson). For example, to get a JSON object with the current departure/arrival board & delay information for the Bucharest North railway station (the main & biggest one in our country), you would point your browser or the URL variable for whatever app you are consuming the data with to: http://localhost:9090/station/10017.
-### Ideas
+### Train information
+In the same way you can get the current trains in a certain railway station, you can get the current information for a certain train. CFR provides information such as delays, the last station the train has passed (with a 7-minute delay), the next station and other useful information.
+
+Just point your browser to http://localhost:9090/train/ID, where ID is the train's unique number. You can get these IDs from the station information feed. For example, you can retrieve the information for train IR 1651 from Bucharest North to Suceava North (valid as of April 2017) by accessing http://localhost:9090/train/1651.
+
+## Ideas
 While the official apps themselves work but may not look so great, romanian developers did their best to create some really cool open source projects and online services related to transportation and infrastructure. See [this live map](http://cfr.webgis.ro/), [this proprietary to GTFS converter](https://github.com/vasile/data.gov.ro-gtfs-exporter), [this trip planner](https://www.acceleratul.ro), etc.
 
 Using this API and other public resources, you may create your own style of station departure board, delay-notification service, cool looking mobile app, while learning how to program and work with structured data ?
-### License, disclaimer and known limitations
+## License, disclaimer and known limitations
 This is a completely open source project, built on open source modules and libraries and licensed under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).
 
 Also, you are completely responsible for what you do with it - keep in mind that CFR S.A. and InfoFer (the state-railway owned IT firm which builds their software) are not particularly transparent or third party developer friendly. If you do mass-scraping or develop some publicly accessible service that generates loads of traffic from the same server to them or clones their data to a database for various reasons, you may run into some trouble, [as this fellow enthusiast did while making a web trip-planner using CFR Calatori's timetable from their website](http://legi-internet.ro/blogs/index.php/cfr-crede-ca-are-monopol-pe-mersul-trenurilor-pe-internet).
 
 But for tinkering, playing around and working with real-time data that clearly can't confuse anyone if the error is not from CFR themselves, you should be fine and on the right side of the law, at least from my experience. Maybe they'll offer their own API with proper rules and licensing at some point.
 
-Known limitations:
-- Currently you can only get the station departure & arrival (times, originating & destination station) board with delays and lines (where applicable) through the /station/ID endpoint, while detailed information about individual trips and trips between stations is provided by the CFR webpage. I may work on additional features.
+#### Known limitations:
 - Requests are not authenticated and no rate limiting is implemented, so it's in no way ready to be exposed on the web.
 - This is not particularly fast, because the CFR Webpage isn't either. You'll probably want background requests and caching. After the initial request is made, it'll wait 8 seconds before parsing the data. If data hasn't been displayed on the webpage, it will wait an additional 20 seconds. After this, the API will output a blank object - this may mean that the scraped web service is down, it is really slow to respond or there are really no current trips stopping at that particular station (at night or at a small stop, for example).
 - This is scraping and parsing, so any structural update to the CFR webpage, while highly unlikely in the near future may break this.
+- The train information feed does not provide the details regarding the train's delays and other useful information that Infofer offers with their service. This will be updated in the future.
 
-Public information web-service provided by CFR S.A. through Informatica Feroviara: http://appiris.infofer.ro/SosPlcRO.aspx, appiris.infofer.ro/MyTrainRO.aspx, http://appiris.infofer.ro/MersTrenRo.aspx. This is information from infrastructure administration and not a specific passenger carrier. Official passenger timetables are found here: http://mersultrenurilorcfr.ro, http://infofer.ro/ and static data source with timetables updated at the end of each year: http://data.gov.ro/organization/sc-informatica-feroviara-sa
+Public information web-service provided by CFR S.A. through Informatica Feroviara: http://appiris.infofer.ro/SosPlcRO.aspx, appiris.infofer.ro/MyTrainRO.aspx, http://appiris.infofer.ro/MersTrenRo.aspx. This is information from infrastructure administration and not a specific passenger carrier. Official passenger timetables are found here: http://mersultrenurilorcfr.ro, http://infofer.ro/ and static data source with timetables updated at the end of each year: http://data.gov.ro/organization/sc-informatica-feroviara-sa
diff --git a/index.js b/index.js
@@ -1,31 +1,165 @@
 var exec = require('child_process').exec,
     cheerio = require('cheerio'),
     cheerioTableparser = require('cheerio-tableparser'),
-    data = [], output = [], newArray;
+	commandExistsSync = require('command-exists').sync,
+	path = require('path'),
     express = require('express')();
 
+// Check if PhantomJS can be ran on the system
+if (!commandExistsSync('phantomjs')) {
+	process.stderr.write(
+		"Could not find PhantomJS on this system. " +
+		"Please ensure that PhantomJS is installed and can be ran " +
+		"from the current path.\n" +
+		"http://phantomjs.org/\n"
+	);
+    process.exit(1);
+}
+
 express.get('/station/:id*', function(req, res, next) {
-  cmd = 'phantomjs station.js '+req.params['id'];
+	cmd = 'phantomjs ' + path.resolve(__dirname, 'station.js') + ' ' + req.params['id'];
+	
+	exec(cmd, function(error, stdout, stderr) {
+		if (error !== null) {
+			console.log('Execution error: ' + error);
+			
+			res.statusCode = 500;
+			res.send('Internal error.');
+			return;
+		}
+		
+		res.setHeader('Access-Control-Allow-Origin', '*');
+		var output = [];
+		
+		// Get the retrieved HTML table from PhantomJS
+		$ = cheerio.load(stdout);
+		cheerioTableparser($);
+		var data = $("table").parsetable(true, true, true);
+		
+		if (data.length != 0) {
+			var newArray = data[0].map(function(col, i) {
+				return data.map(function(row) {
+					return row[i];
+				});
+			});
+			newArray.shift();
+			
+			// Push each entry in the table into the output array
+			for (var i = 0; i < newArray.length; i++) {
+				// Split the origin and final stations from their respective times
+				if (newArray[i][3]) {
+					from = newArray[i][3].split(/[0-9]/)[0].replace(/\s\s*$/, '');
+					originatingdepart = newArray[i][3].match(/([0-9\:]+)/)[0]; 
+				} else { 
+					from = ''; originatingdepart = '';
+				}
+				
+				if (newArray[i][7]) {
+					to = newArray[i][7].split(/[0-9]/)[0].replace(/\s\s*$/, '');
+					finalarrive = newArray[i][7].match(/([0-9\:]+)/)[0]
+				} else {
+					to = ''; finalarrive = '';
+				}
+				
+				// Push the output
+				output.push({
+					ID:			newArray[i][1],
+					Rank:		newArray[i][0],
+					Operator:	newArray[i][2],
+					
+					Origin: from,
+					OriginDeparture: originatingdepart,
+					
+					Destination: to,
+					DestinationArrival: finalarrive,
+					
+					Arrival: newArray[i][5],
+					Departure: newArray[i][6],
+					
+					Line: newArray[i][8],
+					Delay: newArray[i][4]
+				});
+			} 
+		}
+		
+		// Send the output
+		res.send(JSON.stringify(output));
+	});
+});
 
-  exec(cmd, function(error, stdout, stderr) {
-    $ = cheerio.load(stdout);
-    cheerioTableparser($);
-    data = $("table").parsetable(true, true, true);
-    if (data.length != 0) { newArray = data[0].map(function(col, i) { 
-        return data.map(function(row) { 
-          return row[i] 
-        })
-      });
-    newArray.shift();
-    output = [];
-    for (var i = 0; i < newArray.length; i++) {
-    	if (newArray[i][3]) { from = newArray[i][3].split(/[0-9]/)[0].replace(/\s\s*$/, ''); originatingdepart = newArray[i][3].match(/([0-9\:]+)/)[0]; } else { from = ''; originatingdepart = ''; }
-    	if (newArray[i][7]) { to = newArray[i][7].split(/[0-9]/)[0].replace(/\s\s*$/, ''); finalarrive = newArray[i][7].match(/([0-9\:]+)/)[0] } else  { to = ''; finalarrive = ''; }
-      output.push({train: newArray[i][1], type: newArray[i][0], operator: newArray[i][2], from: from, originatingdepart: originatingdepart, to: to, finalarrive: finalarrive, arrive: newArray[i][5], depart: newArray[i][6], line: newArray[i][8], delay: newArray[i][4]});
-    } }
-  res.send(JSON.stringify(output));
-});  
+express.get('/train/:id*', function(req, res, next) {
+	cmd = 'phantomjs  ' + path.resolve(__dirname, 'train.js') + ' ' + req.params['id'];
+	
+	exec(cmd, function(error, stdout, stderr) {
+		if (error !== null) {
+			console.log('Execution error: ' + error);
+			
+			res.statusCode = 500;
+			res.send('Internal error.');
+			return;
+		}
+		
+		res.setHeader('Access-Control-Allow-Origin', '*');
+		var output = [];
+		
+		// Get the retrieved HTML table from PhantomJS
+		$ = cheerio.load(stdout);
+		cheerioTableparser($);
+		var data = $("table").parsetable(true, true, true);
+		
+		if (data.length != 0) {
+			/*
+			 * Remove the last two elements of the data array, 
+			 * since the table does contain other UI elements
+			 * we don't care about.
+			*/
+			data.splice(2, 2);
+			
+			TrainStatus = data[1][4];
+			TrainInCirculation = (TrainStatus === "In circulatie");
+			TrainArrived = (TrainStatus === "Sosit la destinatie");
+			TrainAwaitingDeparture = (TrainStatus === "Asteapta plecarea");
+
+			// Save the output
+			var TrainInfo = {
+				ID: 						data[1][1],
+				Rank:						data[1][0],
+				
+				Operator:					data[1][2],
+				
+				Route:						data[1][3],
+				Distance:					data[1][12],
+				RouteDuration:				data[1][13],
+				
+				TrainStatus:				TrainStatus,				
+				TrainInCirculation:			TrainInCirculation,
+				TrainArrived:				TrainArrived,
+				TrainAwaitingDeparture:		TrainAwaitingDeparture,
+				
+				LatestInfo:					data[1][5],
+				LatestInfoTime:				data[1][6],
+				
+				Delay:						data[1][7],
+				
+				Destination:				data[1][8],
+				DestinationArriveTime:		data[1][9],
+				
+				NextStop:					data[1][10],
+				NextStopTime:				data[1][11]
+			};
+			
+			output = {
+				Info:	TrainInfo,
+			};
+		}
+		
+		// Send the output
+		res.send(JSON.stringify(output));
+	});
+});
 
+process.on('uncaughtException', function (err) {
+	console.log('Caught unhandled exception: ' + err);
 });
 
 express.listen(9090);
diff --git a/package.json b/package.json
@@ -1,16 +1,24 @@
 {
   "name": "cfr-iris-scraper",
-  "version": "1.0.0",
-  "description": "Live train information: srape-powered API with JSON endpoints for the Romanian national railway infrastructure company CFR S.A's realtime information web interface IRIS (built by state-railway IT company InfoFer).",
+  "version": "1.0.1",
+  "description": "Live train information: scrape-powered API with JSON endpoints for the Romanian national railway infrastructure company CFR S.A's realtime information web interface IRIS (built by state-railway IT company InfoFer).",
   "main": "index.js",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"
   },
-  "author": "",
-  "license": "Apache License 2.0",
+  "contributors": [
+	{"name": "Bogdan Minea"},
+	{"name": "Tiberiu Danciu"}
+  ],
+  "repository": {
+	"type": "git",
+	"url": "https://github.com/BodoMinea/cfr-iris-scraper"
+  },
+  "license": "Apache-2.0",
   "dependencies": {
     "cheerio": "^0.22.0",
     "cheerio-tableparser": "^1.0.1",
-    "express": "^4.15.2"
+    "express": "^4.15.2",
+	"command-exists": "^1.2.2"
   }
 }
diff --git a/train.js b/train.js
