Go back

Basic search with Elasticsearch

Inspired by this tutorial I tried to continue investigating Elasticsearch since I would like to use a fast indexing tool for the data I am gathering and the applications I am developing.

Install the Python library for Elasticsearch

https://elasticsearch-py.readthedocs.io/en/master/

$ pip install elasticsearch

Note: on my Mac I installed Elasticsearch through Brew

$ brew install elasticsearch
$ brew services start elasticsearch

Creating the data

Read the CSV files

Read the character data

In [1]:
import pandas as pd
character_df = pd.read_csv('data/nintendo_characters.csv')
character_df
Out[1]:
id name description color occupation picture
0 2 Luigi This is Luigi green plumber https://upload.wikimedia.org/wikipedia/en/f/f1...
1 1 Mario This is Mario red plumber https://upload.wikimedia.org/wikipedia/en/9/99...
2 3 Peach My name is Peach pink princess https://s-media-cache-ak0.pinimg.com/originals...
3 4 Toad I like funghi red NaN https://upload.wikimedia.org/wikipedia/en/d/d1...

Remove the NaN

In [2]:
character_df.occupation = character_df.occupation.fillna('')

Read the world data

In [3]:
world_df = pd.read_csv('data/super_mario_3_worlds.csv', sep=';')
world_df
Out[3]:
id world name image description picture
0 1 World 1 Grass Land Grass Land.PNG Grass Land is the first world of the game. It ... https://www.mariowiki.com/images/thumb/f/fa/Gr...
1 2 World 2 Desert Land World2SMB3.PNG Desert Land is the second world of the game. I... https://www.mariowiki.com/images/thumb/d/d1/Wo...
2 3 World 3 Water Land Sea Side.PNG Water Land is a water-themed region that was r... https://www.mariowiki.com/images/thumb/b/b7/Se...
3 4 World 4 Giant Land SMAS-Big Island Map.PNG Giant Land is mainly composed of an island in ... https://www.mariowiki.com/images/thumb/9/9c/SM...
4 5 World 5 Sky Land Sky world.PNG Sky Land is the world that has been conquered ... https://www.mariowiki.com/images/thumb/6/69/Sk...
5 6 World 6 Ice Land SMB36.PNG Ice Land is an area covered in snow and ice. T... https://www.mariowiki.com/images/thumb/4/40/SM...
6 7 World 7 Pipe Land Pipe maze.PNG Pipe Land is a series of small islands in a ne... https://www.mariowiki.com/images/thumb/a/aa/Pi...
7 8 World 8 Dark Land Dark land2.PNG The eighth and final world is ruled by King Bo... https://www.mariowiki.com/images/thumb/0/01/Da...
8 9 World 9 Warp Zone World 9.PNG World 9 is only accessible by a Warp Whistle. ... https://www.mariowiki.com/images/thumb/0/09/Wo...

Setup Elasticsearch

Create the parameters

In [4]:
ES_HOST = {"host" : "localhost", "port" : 9200}
INDEX_NAME = 'nintendo'
TYPE_NAME = 'character'
ID_FIELD = 'id'

Setup the Elasticsearch connector

In [5]:
from elasticsearch import Elasticsearch
es = Elasticsearch(hosts = [ES_HOST])

Create the index

Create the index for nintendo if it does not exists, otherwise first delete it.

In [6]:
if es.indices.exists(INDEX_NAME):
    print("Deleting the '%s' index" % (INDEX_NAME))
    res = es.indices.delete(index = INDEX_NAME)
    print("Acknowledged: '%s'" % (res['acknowledged']))

request_body = {
    "settings" : {
        "number_of_shards": 1,
        "number_of_replicas": 0
    }
}
print("Creating the '%s' index!" % (INDEX_NAME))
res = es.indices.create(index = INDEX_NAME, body = request_body)
print("Acknowledged: '%s'" % (res['acknowledged']))

Deleting the 'nintendo' index
Acknowledged: 'True'
Creating the 'nintendo' index!
Acknowledged: 'True'

Create the bulk data

Loop through the dataframe and create the data to insert into the index.

In [7]:
bulk_data = []
In [8]:
for index, row in character_df.iterrows():
    data_dict = {}
    for i in range(len(row)):
        data_dict[character_df.columns[i]] = row[i]
    op_dict = {
        "index": {
            "_index": 'nintendo',
            "_type": 'character',
            "_id": data_dict['id']
        }
    }
    bulk_data.append(op_dict)
    bulk_data.append(data_dict)
In [9]:
for index, row in world_df.iterrows():
    data_dict = {}
    for i in range(len(row)):
        data_dict[world_df.columns[i]] = row[i]
    op_dict = {
        "index": {
            "_index": 'nintendo',
            "_type": 'world',
            "_id": data_dict['id']
        }
    }
    bulk_data.append(op_dict)
    bulk_data.append(data_dict)

Insert the data into the index

In [10]:
import json
print("Bulk indexing...")
res = es.bulk(index = INDEX_NAME, body = bulk_data, refresh = True)

Bulk indexing...

Query using CURL

In [11]:
!curl -XGET 'http://localhost:9200/_search?pretty'

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "hits" : {
    "total" : 3295,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "nintendo",
        "_type" : "character",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "id" : 2,
          "name" : "Luigi",
          "description" : "This is Luigi",
          "color" : "green",
          "occupation" : "plumber",
          "picture" : "https://upload.wikimedia.org/wikipedia/en/f/f1/LuigiNSMBW.png"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "character",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "id" : 1,
          "name" : "Mario",
          "description" : "This is Mario",
          "color" : "red",
          "occupation" : "plumber",
          "picture" : "https://upload.wikimedia.org/wikipedia/en/9/99/MarioSMBW.png"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "character",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "id" : 3,
          "name" : "Peach",
          "description" : "My name is Peach",
          "color" : "pink",
          "occupation" : "princess",
          "picture" : "https://s-media-cache-ak0.pinimg.com/originals/d2/4d/77/d24d77cfbba789256c9c1afa1f69b385.png"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "character",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "id" : 4,
          "name" : "Toad",
          "description" : "I like funghi",
          "color" : "red",
          "occupation" : "",
          "picture" : "https://upload.wikimedia.org/wikipedia/en/d/d1/Toad_3D_Land.png"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "world",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "id" : 1,
          "world" : "World 1",
          "name" : "Grass Land",
          "image" : "Grass Land.PNG",
          "description" : "Grass Land is the first world of the game. It was attacked by Larry Koopa, who stole the wand of the Grass Land king and turned him into a dog (or a Cobrat from Super Mario Bros. 2 in the remake). The landscape itself is mainly composed of plains, surrounded by hills and even some cliffs in the south. A fortress can be found in the middle of Grass Land, and the king's castle lies to the southeast, surrounded by a circular moat. The enemies Mario encounters here are regular ones, like Goombas, Koopa Troopas and Piranha Plants. The world features a Spade Panel, two Toad Houses and six levels, of which four have to be cleared to reach the king's castle.",
          "picture" : "https://www.mariowiki.com/images/thumb/f/fa/Grass_Land.PNG/200px-Grass_Land.PNG"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "world",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "id" : 2,
          "world" : "World 2",
          "name" : "Desert Land",
          "image" : "World2SMB3.PNG",
          "description" : "Desert Land is the second world of the game. It is a region within a vast desert, filled with sand, palm trees and some pyramids. A fortress is located in the west part of the desert, and a quicksand field can also be found, as well as a great pyramid that the player needs to traverse in order to reach the king's castle. The king was attacked by Morton Koopa Jr., who turned him into a spider (or a Hoopster from Super Mario Bros. 2 in the remake). The world features two Spade Panels and three Toad Houses, of which one lies in a secret area behind a rock that needs to be crushed by a Hammer. The boulder also hides two Fire Brothers which stole the last Warp Whistle. Four of the five levels need to be cleared to get to the great pyramid and the castle. Desert Land houses many desert-related creatures like Fire Snakes and the extremely rare Angry Sun.",
          "picture" : "https://www.mariowiki.com/images/thumb/d/d1/World2SMB3.PNG/200px-World2SMB3.PNG"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "world",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "id" : 3,
          "world" : "World 3",
          "name" : "Water Land",
          "image" : "Sea Side.PNG",
          "description" : "Water Land is a water-themed region that was raided by Wendy O. Koopa, who turned the king into a kappa (or a Dino-Rhino from Super Mario World in the remake). While some levels take place on solid ground, most of the levels and even one of the worlds two fortresses involve water in a certain way. At the northern part of the world map, Mario will encounter drawbridges that open and close in a set pattern. The world's castle is located far to the east on a small, remote island that is only accessible through a Warp Pipe. A boat can be unlocked by using a Hammer on a rock in the south. Through it, the player can reach some bonus Spade Panels and Toad Houses. Water Land contains nine levels in total, of which one can be skipped if a certain drawbridge is closed, and houses several water creatures like Bloopers, Cheep Cheeps, and Big Berthas. The world also introduces a very rare Boo known as a Stretch.",
          "picture" : "https://www.mariowiki.com/images/thumb/b/b7/Sea_Side.PNG/200px-Sea_Side.PNG"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "world",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "id" : 4,
          "world" : "World 4",
          "name" : "Giant Land",
          "image" : "SMAS-Big Island Map.PNG",
          "description" : "Giant Land is mainly composed of an island in the vague shape of a Koopa. It is a relatively green island with plants growing on it that resemble Fire Flowers. The castle at the west coast of the island was attacked by Iggy Koopa, who transformed the king into an orange dinosaur (or Donkey Kong Jr. in the remake). The world has two fortresses, one on the east side and one on a small island in a lake in the world's center. The most prominent feature of Giant Land, which gives this world its name, is the fact that many enlarged versions of regular enemies, blocks, and environmental features can be found here. The world features four Toad Houses, two Spade Panels and six levels, of which five need to be cleared to reach the king's castle.",
          "picture" : "https://www.mariowiki.com/images/thumb/9/9c/SMAS-Big_Island_Map.PNG/200px-SMAS-Big_Island_Map.PNG"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "world",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "id" : 5,
          "world" : "World 5",
          "name" : "Sky Land",
          "image" : "Sky world.PNG",
          "description" : "Sky Land is the world that has been conquered by Roy Koopa, who has turned its king into a condor (Albatoss in the remake). It is divided into two parts: a ground part and a sky part. The player begins on the ground. The most notable feature of this area is the possibility to gain the Goomba's Shoe, an item that can be obtained in level 5-3. After clearing the levels on the ground, the player can reach a spiraling tower that reaches up to the sky. The main part of the level is located here, and there are also some creatures exclusively to this realm, namely the Para-Beetle. After clearing the tower that serves as a link between the two areas, the player can go back to the ground, but they will have to clear the tower again on their way up. If the Koopaling isn't defeated at the first try, his Airship will be able to move freely between sky and ground. There are nine levels in total, three Spade Panels, three Toad Houses and two fortresses. The castle is on the southwest part of the sky part.",
          "picture" : "https://www.mariowiki.com/images/thumb/6/69/Sky_world.PNG/200px-Sky_world.PNG"
        }
      },
      {
        "_index" : "nintendo",
        "_type" : "world",
        "_id" : "6",
        "_score" : 1.0,
        "_source" : {
          "id" : 6,
          "world" : "World 6",
          "name" : "Ice Land",
          "image" : "SMB36.PNG",
          "description" : "Ice Land is an area covered in snow and ice. The castle was attacked by Lemmy Koopa - who has turned its king into a fur seal (Monty Mole in the remake) and Mario has to venture there and reclaim the magic wand just like in the previous worlds. Before he can reach the castle however, the player has to navigate Mario through the levels of Ice Land. These levels feature frozen ground which makes movement more difficult, as Mario has poor footing on them and is likely to slip off into a bottomless pit. In some levels, the player can find ice blocks that contain coins or enemies. These blocks can only be melted with one of Fire Mario's fireballs. There are ten levels in total, three Spade Panels, two Toad Houses, and three fortresses. The castle is far to the east near the sea.",
          "picture" : "https://www.mariowiki.com/images/thumb/4/40/SMB36.PNG/200px-SMB36.PNG"
        }
      }
    ]
  }
}

Search all worlds:

curl -XGET 'http://localhost:9200/nintendo/world/_search?pretty'

Pagination:

curl -XGET 'http://localhost:9200/nintendo/world/_search?size=2&from=2&pretty'

Specify the fields you want to be returned:

curl -XGET 'http://localhost:9200/nintendo/character/_search?pretty&q=name:Luigi&fields=name,occupation'

Search for the word 'pipe':

curl -XGET 'http://localhost:9200/nintendo/world/_search?pretty&q=pipe'
notebook elasticsearch Python Jupyter
Go back