← 返回列表

Elasticsearch Lua II

高级反反爬 Dhaval Kapil 发布 2016-08-16 入库 2026-11-18T00:36:06

This post is about my GSoC project, that I worked on during summer, 2016. I worked under the LabLua organization on adding a test suite and improving documentation for elasticsearch-lua. elasticsearch-lua.

Introduction

Elasticsearch is a distributed, scalable and full-text search engine based on Lucene. It provides an HTTP web interface and handles JSON documents. It is presently ranked 1 in the category of ‘Search engines’.

elasticsearch-lua is a client for Elasticsearch that provides a wrapper over the REST interface for the Lua Programming Language. I developed it as part of GSoC 2015 with my mentor Pablo Musa.

My GSoC project this year was entitled ‘Improve elasticsearch-lua tests and builds’ and was a continuation of the work that I had done last year. Apart from adding a test suite for elasticsearch-lua and making it robust, I also decided to work on the documentation of the code.

Test suite for elasticsearch-lua

The tests are divided into unit, integration and stress tests. Note that all these tests run for Lua 5.1, 5.2, 5.3 and LuaJIT 2.0. Code coverage is measured for unit tests and integration tests. Coveralls was chosen to measure and maintain code coverage. As of now, around 91% of the code is covered with tests.

Unit Tests

There are many different modules within elasticsearch-lua. For every such module, there is a corresponding unit test written. Unit tests can be found in tests/ directory. Care was taken to test extensively all the endpoints. Some key points to note:

The diff of changes due to unit tests can be seen here.

Integration Tests

Apart from the test of every component individually, it is equally important that they work together while interacting with each other. To make elasticsearch-lua robust, it was necessary to add some integration tests.

The diff of changes due to integration tests can be seen here.

Stress Tests

Stress tests involve testing elasticsearch-lua limits. By having these tests, the client will be able to prove its stability in an effective manner.

The diff of changes due to stress tests can be seen here.

Documentation

Having a good documentation is very important for any library. It helps developers to understand functionalities without having to investigate the code. Moreover, it helps the library adoption as new developers can use it as a guide to get started. Although this was initially not a task for the GSOC project, after realizing its importance, I opted to invest a lot of time in the documentation and added it to the GSoC project timeline.

Guides

The guides consist of documents and tutorials that help developers to install, use and customize elasticsearch-lua. The guides explain the most frequently used functionalities along with some internals. These pages are hosted here.

API Documentation

The API Documentation lists all possible functions provided by the elasticsearch-lua. Each function name is accompanied by the parameters that it accepts. The API documentation is published here.

The diff of changes pertaining to documentation can be seen here.

Additional tasks (Not part of GSoC)

Apart from the tasks mentioned above, I worked on the following as well:

Luaver

While working with elasticsearch-lua, I had to frequently switch between different versions of lua while developing the test suite. Switching is not simple and I faced the following issues often:

  1. Building different lua versions required some effort such as downloading the version source, unzipping, installing and managing any dependency faced. Also, the previous version had to be deleted completely in order to avoid any ambiguity.

  2. Luarocks installation depends on the Lua version. Switching lua versions can mess up the installed rocks.

  3. To solve these issues I used workaround methods, such as editing the source code of some existing rocks.

  4. Sometimes, these code changes broke the entire rock. In such cases, I had to remove all existing rocks, rebuild luarocks and then reinstall the needed rocks.

As I was already familiar with NodeJS and Ruby and understood how such problems were addressed by nvm and rvm, I decided to create a similar tool for lua, and that is how luaver was born.

I also wrote a separate blog post about luaver and you can support the project here. Initially, I didn’t expect to spend much time on it and figured that I could manage both GSoC and develop luaver simultaneously. However, at some point in time, I got too involved in luaver which resulted in me getting one week behind the timeline that I had proposed for GSoC. Nevertheless, I covered it up soon.

Updating elasticsearch-lua

It is important that the client implements all the features provided by Elasticsearch. Also, Elasticsearch is evolving a lot and releasing in a fast pace, so it is important that clients are also up-to-date. Some features were missing and the client version was 1.6 while Elasticsearch is in 2.3. Therefore, I decided to update existing features and implement some missing features.

Benefits of working on the same project for two consecutive years

  1. I myself had written the client. The codebase was already at my finger-tips. I could spend more time working than understanding and getting comfortable with the code.

  2. I wanted to further consolidate my client and make it stable. I couldn’t get much time during the rest of the year to work full-fledged on the development. Google Summer of Code offered a nice incentive.

  3. I had already worked with the Lua community. Being in familiar environment, I was able to work and think freely. luaver was created to benefit the open source Lua community. If this was my first time I wouldn’t even have thought about developing it.

原文链接:https://dhavalkapil.com/blogs/Elasticsearch-Lua-GSOC-2016