MEVN stack experiment with Jupyter
In this notebook I want to play with Jupyter and show the steps of how to create a MEVN application from a notebook. Normally I would do this in the normal Linux terminal and a text editor, but since we can combine code, explanation and shell commands, I want to create a story in this notebook which hopefully will be of any help of people experimenting with full-stack development. I will create the application, use some simple Linux tricks and use Selenium to test the application.
Please note development like this is far from optimal, but I think it is very cool what you can achieve without leaving your notebook, knowing a bit of Docker, Linux and Python.
The original notebook can be found here.
Objective¶
Setup an application with the following elements
using
Notebook settings¶
In [1]:
%%html
<style>
img.logo{
height: 100px;
}
img.screenshot{
max-width:500px;
-webkit-filter: drop-shadow(5px 5px 5px #222);
filter: drop-shadow(2px 5px 5px #222);
margin: 50px auto;
}
</style>
Clean up the images from earlier runs.
In [2]:
rm *.png
In [3]:
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
In [4]:
!jupyter --version
In [5]:
!jupyter notebook --version
Python version¶
In [6]:
import platform
platform.python_version()
Out[6]:
In [7]:
!docker version
In [8]:
!docker ps
pip3 install docker
In [9]:
import docker
docker_client = docker.from_env()
Available containers¶
In [10]:
for cntr in docker_client.containers.list():
print("name={} (id={})".format(cntr.name, cntr.id))
MongoDB¶
Check if the Mongo Docker container is running, otherwise, start the container.
In [11]:
mongo_running = False
for cntr in docker_client.containers.list():
if 'mongo' in cntr.attrs['Config']['Image']:
mongo_running = True
container = cntr
if mongo_running is False:
container = docker_client.containers.run("mongo:latest", name='mongo', ports={'27017': '27017'}, detach=True)
In [12]:
container
Out[12]:
Verification that the Mongo container is running:
In [13]:
!docker ps | grep mongo
See Documentation to install PyMongo.
$ pip install pymongo
pip3 install pymongo
In [14]:
from pymongo import MongoClient
mongo_client = MongoClient('localhost', 27017)
In [15]:
import pymongo
class MongoPipeline(object):
collection_name = 'games'
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
)
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def close_spider(self, spider):
self.client.close()
def process_item(self, item, spider):
self.db[self.collection_name].insert_one(dict(item))
return item
Retrieving data from https://www.nintendo.co.jp/ir/en/finance/software/index.html with the following markup:
<ul class="sales_layout">
<li class="sales_layout_list">
<div class="ta_l">
<p class="sales_title">The Legend of Zelda:<br> Breath of the Wild</p>
<p class="sales_value"><span>4.70</span> million pcs.</p>
</div>
<div class="ta_r">
<p>
<img src="./img/data_switch_001.png" alt="" width="110" height="" class="sales_product1">
</p>
</div>
</li>
</ul>
Using Scrapy we can gather the data in a convenient way.
pip3 install scrapy
In [16]:
import logging
import scrapy
from scrapy.crawler import CrawlerProcess
import re
class ConsoleSpider(scrapy.Spider):
name = "games"
start_urls = [
"https://www.nintendo.co.jp/ir/en/finance/software/index.html",
"https://www.nintendo.co.jp/ir/en/finance/software/wiiu.html",
"https://www.nintendo.co.jp/ir/en/finance/software/3ds.html",
"https://www.nintendo.co.jp/ir/en/finance/software/wii.html",
"https://www.nintendo.co.jp/ir/en/finance/software/ds.html"
]
custom_settings = {
'LOG_LEVEL': logging.CRITICAL,
'DOWNLOAD_DELAY': .25,
'RANDOMIZE_DOWNLOAD_DELAY': True,
'ITEM_PIPELINES': {
'__main__.MongoPipeline': 300,
},
'MONGO_URI': 'mongodb://localhost:27017',
'MONGO_DATABASE': 'nintendo'
}
def parse(self, response):
for cons in response.css('li.sales_layout_list'):
yield {
'console': response.css('div.tab div.tabInner span::text').extract_first(),
'name': cons.css('p.sales_title::text').extract_first().strip(),
'image': 'https://www.nintendo.co.jp/ir/en/finance/software' + cons.css('p img::attr(src)').extract_first()[1:],
'sales': cons.css('p.sales_value span::text').extract()[0].strip()
}
In [17]:
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(ConsoleSpider)
process.start()
Out[17]:
Verify the Nintendo database is created¶
In [18]:
if 'nintendo' in mongo_client.database_names():
print('Database found!')
Verify the games collection is created inside the Nintendo database¶
In [19]:
db = mongo_client['nintendo']
if 'games' in db.collection_names():
print('Collection found!')
Retrieve the games from the collection¶
In [20]:
games = list(db['games'].find({}))
In [21]:
print("Found {} games".format(len(games)))
In [22]:
!rm -rf images/*
In [23]:
import re
updated_games = []
for gameindex, game in enumerate(games):
image = game['image']
image_short = image.split('/')[-1]
!wget --no-check-certificate $image -P images/
game['image'] = 'images/' + image_short
game.pop('_id', None)
updated_games.append(game)
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Out[23]:
Verify the downloaded images¶
Run the ls images to show the available images and save the list to lsimages.
In [24]:
lsimages = !ls images
lsimages
Out[24]:
Show the first image¶
In [25]:
from IPython.display import Image
from IPython.core.display import HTML
Image(url='images/'+lsimages.list[0])
Out[25]:
Delete the old games from the database¶
In [26]:
result = db.games.delete_many({})
Insert the updated games¶
In [27]:
result = db.games.insert_many(updated_games)
In [28]:
!node --version
In [29]:
!npm --version
In [30]:
!rm -rf server
In [31]:
!mkdir server && cd server && npm init -y && npm install express-generator --save
Check ExpressJS version¶
In [32]:
!cd server && npm express --version
Create the scaffold for the server¶
Note that we use npx instead of npm because we run NPM from a local folder.
In [33]:
!cd server && npx express -y --force --git --view ejs .
In [34]:
ls server/
In [35]:
!cat server/package.json
Install dependencies¶
In [36]:
!cd server && npm install
Run the app¶
In [37]:
import subprocess
proc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
proc.pid
Out[37]:
Request the ExpressJS app¶
In [38]:
import requests
import time
time.sleep(5)
resp = requests.get('http://localhost:3000')
resp.status_code
Out[38]:
Stop the ExpressJS app¶
Use lsof to find the process that uses port 3000 and kill it.
In [39]:
process=!lsof -i:3000 -t
expressid = int(process[0])
expressid
Out[39]:
In [40]:
!kill -9 $expressid
Check if the server is down¶
In [41]:
try:
resp = requests.get('http://localhost:3000')
except:
print("Server is down!")
Install package¶
In [42]:
!cd server && npm install --save mongoose
Check installation¶
In [43]:
!cat server/package.json | grep mongoose
Add the Mongoose connection¶
Add the following to the top of server/app.js.
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/nintendo');
var db = mongoose.connection;
db.on("error", console.error.bind(console, "Connection error"));
db.once("open", function(callback){
console.log("Connection successful")
});
In [44]:
%%file server/app.js
var express = require('express');
var path = require('path');
var favicon = require('serve-favicon');
var logger = require('morgan');
var cookieParser = require('cookie-parser');
var bodyParser = require('body-parser');
// Mongoose connection
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/nintendo');
var db = mongoose.connection;
db.on("error", console.error.bind(console, "Connection error"));
db.once("open", function(callback){
console.log("Connection successful")
});
var index = require('./routes/index');
var users = require('./routes/users');
var app = express();
// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'ejs');
// uncomment after placing your favicon in /public
//app.use(favicon(path.join(__dirname, 'public', 'favicon.ico')));
app.use(logger('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use('/', index);
app.use('/users', users);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
var err = new Error('Not Found');
err.status = 404;
next(err);
});
// error handler
app.use(function(err, req, res, next) {
// set locals, only providing error in development
res.locals.message = err.message;
res.locals.error = req.app.get('env') === 'development' ? err : {};
// render the error page
res.status(err.status || 500);
res.render('error');
});
module.exports = app;
In [45]:
!head -n 20 server/app.js
Check if MongoDB initializes¶
In [46]:
import subprocess
proc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
In [47]:
import time
time.sleep(5)
for line in proc.stdout:
print(str(line))
if 'Connection successful' in str(line):
print("Success!")
break
In [48]:
process=!lsof -i:3000 -t
expressid = int(process[0])
!kill -9 $expressid
Create new model¶
Create the Mongoose model for the games that we gathered in the earlier step.
In [49]:
!mkdir server/models
In [50]:
%%file server/models/game.js
var mongoose = require("mongoose");
var Schema = mongoose.Schema;
var GameSchema = new Schema({
console: {
type: String
},
name: {
type: String
},
image: {
type: String
},
sales: {
type: String
}
});
module.exports = mongoose.model("Game", GameSchema);
Adding new route¶
Create the route to access the data of the games. In the scaffold we already have the index.js and users.js, so lets create the games.js to setup the routes for the new pages.
Existing routes¶
In [51]:
!cat server/routes/index.js
In [52]:
!cat server/routes/users.js
Add the route requirement¶
Add the following to server/app.js:
...
var consoles = require('./routes/games.js');
...
app.use('/games', games);
...
With sed we can insert text on a certain position in a file.
In [53]:
!sed -i "18i var games = require('./routes/games');" server/app.js
Enable the routes in the app:
In [54]:
!sed -i "36i app.use('/games', games);" server/app.js
and create the route file server/routes/games.js:
In [55]:
%%file server/routes/games.js
var express = require('express');
var router = express.Router();
var Game = require("../models/game");
router.get('/', (req, res) => {
Game.find({}, '', function (error, games){
if (error) { game.error(error); }
res.send({
games: games
})
}).sort({_id:-1})
})
module.exports = router;
Start the server and verify the new route which should return a JSON object.
In [56]:
import json
import requests
import time
proc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
time.sleep(5)
resp = requests.get('http://localhost:3000/games').json()
print(json.dumps(resp, indent=4))
Lets grab the ID of the first game of the data to verify the route in the next step.
In [57]:
game_id = resp['games'][0]['_id']
game_id
Out[57]:
Kill the application again.
In [58]:
process=!fuser 3000/tcp | awk '{print $1}'
expressid = int(process[1])
!kill -9 $expressid
Add another route to the games.js to get information for a single game.
router.get('/:id', (req, res) => {
var db = req.db;
Game.findById(req.params.id, '', function (error, game) {
if (error) { console.error(error); }
res.send(game)
})
})
In [59]:
%%file server/routes/games.js
var express = require('express');
var router = express.Router();
var Game = require("../models/game");
router.get('/', (req, res) => {
Game.find({}, '', function (error, games){
if (error) { game.error(error); }
res.send({
games: games
})
}).sort({_id:-1})
})
router.get('/:id', (req, res) => {
var db = req.db;
Game.findById(req.params.id, '', function (error, game) {
if (error) { console.error(error); }
res.send(game)
})
})
module.exports = router;
Verify the game detail route¶
As a final step to verify the API we check if we can get the detailed information for the game ID we saved in the previous step.
In [60]:
import time
proc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
time.sleep(5)
resp = requests.get('http://localhost:3000/games/'+game_id).json()
print(json.dumps(resp, indent=4))
In [61]:
process=!fuser 3000/tcp | awk '{print $1}'
expressid = int(process[1])
!kill -9 $expressid
Conclusion¶
For the back-end we have created an API using Mongoose and ExpressJS with the following two routes:
- All games
- Game detail
In [62]:
!rm -rf client
In [63]:
!mkdir client && cd client && npm init -y && npm install vue-cli --save
In [64]:
ls -la client
Check version of Vue¶
In [65]:
!cd client && npm vue --version
Create the scaffold¶
To create the scaffold we will use the vue-cli. However, this requires us to give direct input to the command line which is tricky because we need to answer different questions when creating the scaffold using the tool. The easiest way to automate this task is to use the Linux tool Expect. Make sure the tool is installed on your system.
$ sudo apt-get install expect -y
Note: if you do not want to use this trick, you can also use the external terminal and run
$ vue init webpack
in the client directory or simply clone the repository to your local file system.
The following script contains the answers for the prompts created by the vue init.
In [66]:
%%file client/init_expect_script.sh
#!/usr/bin/expect -f
spawn npx vue init webpack
expect "Generate project in current directory?" { send "Y\r" }
expect "Project name"
send "vueclient\r"
expect "Project description"
send "An experiment with Jupyter and MEVN\r"
expect "Author"
send "Jupyter\r\n"
expect "Vue build"
send "\r\n"
expect "Install vue-router?"
send "Y\r"
expect "Use ESLint to lint your code?"
send "Y\r"
expect "Pick an ESLint preset"
send "\r\n"
expect "Set up unit tests"
send "Y\r"
expect "Pick a test runner"
send "\r\n"
expect "Setup e2e tests with Nightwatch?"
send "Y\r"
expect "Should we run `npm install` for you after the project has been created? (recommended)"
send "\r\n"
interact
In [67]:
!chmod +x client/init_expect_script.sh
In [68]:
!cd client && ./init_expect_script.sh
In [69]:
ls client
In [70]:
!cd client && npm install [email protected] --save-dev
In [71]:
import time
proc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
Setup Selenium¶
First create a Docker instance that runs Selenium to avoid the cumbersome installation on the local Linux machine.
In [72]:
selenium_running = False
for cntr in docker_client.containers.list():
if 'selenium' in cntr.attrs['Config']['Image']:
selenium_running = True
container = cntr
if selenium_running is False:
container = docker_client.containers.run("selenium/standalone-chrome:latest", name='selenium', ports={'4444': '4444'}, detach=True)
In [73]:
time.sleep(10)
Install Selenium with pip
pip install selenium
In [74]:
!pip3 install selenium
In [75]:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver = webdriver.Remote("http://localhost:4444/wd/hub", DesiredCapabilities.CHROME)
driver.get('http://prod.jitsejan.com:8080')
driver.save_screenshot('vue_frontpage.png')
Out[75]:
In [76]:
from IPython.display import Image
Image("vue_frontpage.png")
Out[76]:
Kill the process on port 8080 after verifying the front page.
In [77]:
process=!fuser 8080/tcp | awk '{print $1}'
vueid = int(process[1])
!kill -9 $vueid
As we can see from the screenshot, we cannot access the page because webpack has changed some configuration. In order to circument this we need to add the host to the script in package.json and change
"dev": "webpack-dev-server --inline --progress --config build/webpack.dev.conf.js",
to
"dev": "webpack-dev-server --inline --progress --config build/webpack.dev.conf.js --host 0.0.0.0",
which can be done with sed:
In [78]:
!sed -i '/dev.*--inline/s/conf.js/conf.js --host 0.0.0.0/' client/package.json
In [79]:
cat client/package.json | grep dev
Additionally we need to disable the host check in build/webpack.dev.conf.js by adding
disableHostCheck: true
to the devServer.
In [80]:
!sed -i "26i\ \ \ \ disableHostCheck: true," client/build/webpack.dev.conf.js
In [81]:
!grep -C3 disableHostCheck client/build/webpack.dev.conf.js --color=auto
Retry¶
In [82]:
proc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
In [83]:
driver = webdriver.Remote("http://localhost:4444/wd/hub", DesiredCapabilities.CHROME)
driver.get('http://prod.jitsejan.com:8080')
driver.get_screenshot_as_file('vue_front.png')
Out[83]:
In [84]:
Image("vue_front.png")
Out[84]:
In [85]:
process=!fuser 8080/tcp | awk '{print $1}'
vueid = int(process[1])
!kill -9 $vueid
Create games page¶
In [86]:
ls client/src/components/
Create the component¶
Add the Vue template to the Games component.
<template>
<div class="games">
This page will list all the games.
</div>
</template>
<script>
export default {
name: 'Games',
data () {
return {}
}
}
</script>
In [87]:
%%file client/src/components/Games.vue
<template>
<div class="games">
This page will list all the games.
</div>
</template>
<script>
export default {
name: 'Games',
data () {
return {}
}
}
</script>
In [88]:
ls client/src/components/
Create the route¶
In [89]:
cat client/src/router/index.js
Add the import of the Games component to the router file.
import Games from '@/components/Games'
and add the path to the games page indicating to which component the page will link.
...
{
path: '/games',
name: 'Games',
component: Games
}
...
In [90]:
%%file client/src/router/index.js
import Vue from 'vue'
import Router from 'vue-router'
import HelloWorld from '@/components/HelloWorld'
import Games from '@/components/Games'
Vue.use(Router)
export default new Router({
routes: [
{
path: '/',
name: 'HelloWorld',
component: HelloWorld
},
{
path: '/games',
name: 'Games',
component: Games
}
]
})
Verify the new route¶
In [91]:
proc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
driver = webdriver.Remote("http://localhost:4444/wd/hub", DesiredCapabilities.CHROME)
driver.get('http://prod.jitsejan.com:8080/ui/#/games')
driver.get_screenshot_as_file('new_route.png')
process=!fuser 8080/tcp | awk '{print $1}'
vueid = int(process[1])
!kill -9 $vueid
Out[91]:
In [92]:
Image("new_route.png")
Out[92]:
In [93]:
!cd client && npm install --save axios
Setup connection with back-end¶
Create a services folder that will contain the API Javascript files.
In [94]:
mkdir -p client/src/services
In [95]:
%%file client/src/services/api.js
import axios from 'axios'
export default() => {
return axios.create({
baseURL: `http://prod.jitsejan.com:3000`
})
}
Create service to retrieve the games¶
In [96]:
%%file client/src/services/GamesService.js
import api from '@/services/api'
export default {
fetchGames () {
return api().get('games')
}
}
Add the service to the Games component¶
This component shows all the games available in the database. Note that this is the file that should be updated if you would like a more fancy layout, for example by adding Bootstrap or other frameworks.
<template>
<div class="games">
<div class="games-div" v-for="game in games" :key="game._id">
<p>
<span><b>{{ game.name }}</b></span><br />
<span>{{ game.console }}</span><br />
<span>{{ game.sales }}</span>
<a :href="'/ui/#/games/' + game._id">Details</a>
</p>
</div>
</div>
</template>
<script>
import GamesService from '@/services/GamesService'
export default {
name: 'Games',
data () {
return {
games: []
}
},
mounted () {
this.getGames()
},
methods: {
async getGames () {
const response = await GamesService.fetchGames()
this.games = response.data.games
}
}
}
</script>
In [97]:
%%file client/src/components/Games.vue
<template>
<div class="games">
<div class="games-div" v-for="game in games" :key="game._id">
<p>
<span><b>{{ game.name }}</b></span><br />
<span>{{ game.console }}</span><br />
<span>{{ game.sales }}</span>
<a :href="'/ui/#/games/' + game._id">Details</a>
</p>
</div>
</div>
</template>
<script>
import GamesService from '@/services/GamesService'
export default {
name: 'Games',
data () {
return {
games: []
}
},
mounted () {
this.getGames()
},
methods: {
async getGames () {
const response = await GamesService.fetchGames()
this.games = response.data.games
}
}
}
</script>
Start both the Express back-end and the Vue front-end to verify if the data from the API is retrieved with the updated component.
Small trick: by setting the logging preferences when starting the webdriver with Selenium, we can actually retrieve the logs you would normally see with the developer tools in the browser.
In [98]:
vueproc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
expressproc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
desired = DesiredCapabilities.CHROME
desired ['loggingPrefs'] = { 'browser':'ALL' }
driver = webdriver.Remote("http://localhost:4444/wd/hub", desired_capabilities=desired)
driver.get('http://prod.jitsejan.com:8080/ui/#/games')
driver.get_screenshot_as_file('new_route_with_data.png')
vueprocess=!fuser 8080/tcp | awk '{print $1}'
vueid = int(vueprocess[1])
!kill -9 $vueid
expressprocess=!fuser 3000/tcp | awk '{print $1}'
expressid = int(expressprocess[1])
!kill -9 $expressid
Out[98]:
In [99]:
logs = driver.get_log('browser')
logs
Out[99]:
In [100]:
Image("new_route_with_data.png")
Out[100]:
As we can see in the logs, we have an issue accessing the back-end because of the Access-Control-Allow-Origin limitation. To circumvent this, we need to make sure that we are allowed to access the API; hece install the package Cors and enable it in the Express application.
In [101]:
!cd server && npm install --save cors
Add the requirement for the cors package and enable it in the Express app by adding the following to server/app.js.
...
var cors = require('cors')
...
app.use(cors())
...
In [102]:
%%file server/app.js
var express = require('express');
var path = require('path');
var favicon = require('serve-favicon');
var logger = require('morgan');
var cookieParser = require('cookie-parser');
var bodyParser = require('body-parser');
// Mongoose connection
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/nintendo');
var db = mongoose.connection;
db.on("error", console.error.bind(console, "Connection error"));
db.once("openUri", function(callback){
console.log("Connection successful")
});
var cors = require("cors")
var index = require('./routes/index');
var users = require('./routes/users');
var games = require('./routes/games');
var app = express();
app.use(cors());
// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'ejs');
// uncomment after placing your favicon in /public
//app.use(favicon(path.join(__dirname, 'public', 'favicon.ico')));
app.use(logger('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use('/', index);
app.use('/users', users);
app.use('/games', games);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
var err = new Error('Not Found');
err.status = 404;
next(err);
});
// error handler
app.use(function(err, req, res, next) {
// set locals, only providing error in development
res.locals.message = err.message;
res.locals.error = req.app.get('env') === 'development' ? err : {};
// render the error page
res.status(err.status || 500);
res.render('error');
});
module.exports = app;
Now we are connected and allowed to retrieve the data. We use Selenium and wait for the games-div to appear.
In [103]:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
vueproc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
expressproc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
desired = DesiredCapabilities.CHROME
desired ['loggingPrefs'] = { 'browser':'ALL' }
driver = webdriver.Remote("http://localhost:4444/wd/hub", desired_capabilities=desired)
driver.get('http://prod.jitsejan.com:8080/ui/#/games')
WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CLASS_NAME, 'games-div')))
driver.refresh()
driver.save_screenshot('allgames.png')
Out[103]:
Out[103]:
In [104]:
Image("allgames.png")
Out[104]:
Clean up¶
In [105]:
driver.quit()
vueprocess=!fuser 8080/tcp | awk '{print $1}'
vueid = int(vueprocess[1])
!kill -9 $vueid
expressprocess=!fuser 3000/tcp | awk '{print $1}'
expressid = int(expressprocess[1])
!kill -9 $expressid
Create detail page for the game¶
Create the component and connect to the API by importing the service and execute the promise. Retrieving the data for a game can also be solved by extending the GamesServices.js we created earlier by adding another fetch function, but since every problem can be solved in different ways, I chose to use the method as shown in the following file.
<template>
<div class="games-detail">
<h1>Details for {{ data.name }}</h1>
<img :src="'/static/' + data.image" /><br/>
Console: {{ data.console }}<br/>
Sales: {{ data.sales}} million<br/>
</div>
</template>
<script>
import api from '@/services/api'
export default {
name: 'GamesDetail',
data () {
return {
data: ''
}
},
created () {
api().get('games/' + this.$route.params.id)
.then(response => {
this.data = response.data
})
.catch(e => {
this.errors.push(e)
})
}
}
</script>
In [106]:
%%file client/src/components/GamesDetail.vue
<template>
<div class="games-detail">
<h1>Details for {{ data.name }}</h1>
<img :src="'/static/' + data.image" /><br/>
Console: {{ data.console }}<br/>
Sales: {{ data.sales}} million<br/>
</div>
</template>
<script>
import api from '@/services/api'
export default {
name: 'GamesDetail',
data () {
return {
data: ''
}
},
created () {
api().get('games/' + this.$route.params.id)
.then(response => {
this.data = response.data
})
.catch(e => {
this.errors.push(e)
})
}
}
</script>
Add the route for the detail page by adding the following to client/src/router/index.js:
...
import GamesDetail from '@/components/GamesDetail'
...
{
path: '/games/:id',
name: 'GamesDetail',
component: GamesDetail
}
In [107]:
%%file client/src/router/index.js
import Vue from 'vue'
import Router from 'vue-router'
import HelloWorld from '@/components/HelloWorld'
import Games from '@/components/Games'
import GamesDetail from '@/components/GamesDetail'
Vue.use(Router)
export default new Router({
routes: [
{
path: '/',
name: 'HelloWorld',
component: HelloWorld
},
{
path: '/games',
name: 'Games',
component: Games
},
{
path: '/games/:id',
name: 'GamesDetail',
component: GamesDetail
}
]
})
Start the processes
In [108]:
vueproc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
expressproc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
Verify the detail page for a given ID.
In [109]:
desired = DesiredCapabilities.CHROME
desired ['loggingPrefs'] = { 'browser':'ALL' }
driver = webdriver.Remote("http://localhost:4444/wd/hub", desired_capabilities=desired)
driver.get('http://prod.jitsejan.com:8080/ui/#/games/'+game_id)
driver.save_screenshot('gamedetail.png')
Out[109]:
In [110]:
Image("gamedetail.png")
Out[110]:
Note: not all images from the source are working properly, but that could easily be fixed by improving the webscraper and verifying the images.
In [111]:
vueprocess=!fuser 8080/tcp | awk '{print $1}'
vueid = int(vueprocess[1])
!kill -9 $vueid
expressprocess=!fuser 3000/tcp | awk '{print $1}'
expressid = int(expressprocess[1])
!kill -9 $expressid
Finalize¶
To wrap this notebook up, lets update the generic App.vue to make it easier to navigate between the pages by adding a menu to client/src/App.vue. Copy the following text to the client/src/App.vue.
<template>
<div id="app">
<router-link :to="{ name: 'HelloWorld' }">Home</router-link>
<router-link :to="{ name: 'Games'}">Games</router-link>
<router-view/>
</div>
</template>
<script>
export default {
name: 'app'
}
</script>
<style>
#app {
font-family: 'Avenir', Helvetica, Arial, sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
text-align: center;
color: #2c3e50;
margin-top: 60px;
}
</style>
In [112]:
%%file client/src/App.vue
<template>
<div id="app">
<router-link :to="{ name: 'HelloWorld' }">Home</router-link>
<router-link :to="{ name: 'Games'}">Games</router-link>
<router-view/>
</div>
</template>
<script>
export default {
name: 'app'
}
</script>
<style>
#app {
font-family: 'Avenir', Helvetica, Arial, sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
text-align: center;
color: #2c3e50;
margin-top: 60px;
}
</style>
In [113]:
vueproc = subprocess.Popen("cd client && npm run dev &", shell=True, stdout=subprocess.PIPE)
expressproc = subprocess.Popen("cd server && npm start &", shell=True, stdout=subprocess.PIPE)
time.sleep(10)
In [114]:
desired = DesiredCapabilities.CHROME
desired ['loggingPrefs'] = { 'browser':'ALL' }
driver = webdriver.Remote("http://localhost:4444/wd/hub", desired_capabilities=desired)
driver.get('http://prod.jitsejan.com:8080')
driver.save_screenshot('menuadded.png')
Out[114]:
In [115]:
Image("menuadded.png")
Out[115]:
Clean up¶
In [116]:
%%bash
rm -rf client
rm -rf server
rm -rf iamges
Future work¶
In this example application I did not use any fancy styling or advanced HTML structures to make the website more appealing. Additionally, the API is only used to retrieve data, while obviously it could also used to create data, update it or delete it (CRUD), but for the scope of this notebook I kept it to a minimum. As a third improvement you could choose to put everything of this notebook in one Docker compose file, where MongoDB, Selenium, ExpressJS and VueJS are all containerized, but since I am not planning to port this application to another server, it is fine to simply install the files locally and remove them again after the notebook has finished.