algoTrade

This documentation is made for Debian based systems, also it looks best on a wide screen
repo:https://github.com/CacheMeIfYouCan1/algoTrade

Table of Contents

1. Introduction
2. Setup
3. Documentation
   1. Structure
   2. Dictionaries
   3. Logic
4. Usage
5. License
6. Addendum

Introduction

Please note that this project is not a complete trading bot! The provided scripts are only a base foundation, which gathers all necessary data, to build a profitable trading bot. The strategy which was built on top of this foundation will not be disclosed.

Setup

Setting up the development environment

first we install virtualvenv

$ pip install virtualenv

create a directory for the algoTrading bot

$ mkdir algoTrading $ cd algoTrading

next we need to create a virtual environment and source it

$ python3 -m venv algoVenv $ source algoVenv/bin/activate

clone the repository and install the required components

$ git clone https://github.com/CacheMeIfYouCan1/algoTrade/ $ cd algoTrade$ cd project $ pip install -r requirements.txt

now the project is set up

Documentation

Structure

algoTrade/project/
|- algoTrade.py
|- requirements.txt
|- shared/
| |- sharedDict.py
| |- __init.py__
|- getData/
| |- getData.py
| |- __init.py__

Dictionaries

shared/sharedDict.py:

This contains the dirctionaries which contains all variables that are needed within the different classes. It is structured as following:

market data dictionary :

Variable:	Description
market_data_dict['market']:	used to determine which market is being analyzed
market_data_dict['oracle_price']:	stores oracle price as displayed by exchange
market_data_dict['old_price']:	stores the last oracle price before the most recent price change
market_data_dict['base_price']:	stores the last oracle price fetched
market_data_dict['change_factor']:	factor determining how much the price has changed
market_data_dict['acquired']:	used to keep track of manual lock/release
market_data_dict['lock']:	used for locking

order book data dictionary:

variable:	Description
order_book_dict['market']:	used to determine which market is being analyzed
order_book_dict['current_ask_price']:	last fetched ask price
order_book_dict['current_ask_size']:	size of the last fetched ask order
order_book_dict['current_bid_price']:	last bid price
order_book_dict['current_bid_size']:	size of the last bid order
order_book_dict['best_ask_price']:	best ask price of the last x orders
order_book_dict['best_ask_size']:	size of the last best ask order
order_book_dict['best_bid_price']:	best bid price of the last x orders
order_book_dict['best_bid_size']:	size of the last best bid order
order_book_dict['asks_list']:	list containing ask price and size
order_book_dict['bids_list']:	list containing bid price and size
order_book_dict['acquired']:	used to keep track of manual lock/release
order_book_dict['lock']:	used for locking

dictionary to keep track of relations between values:

variable:	Description
value_relations_dict['total_size_asks']:	sum of the last x ask sizes
value_relations_dict['total_size_bids]:	sum of the last x bid sizes
value_relations_dict['calculated_spread']:	calculated spread between best ask and bid
value_relations_dict['calculated_price']:	price, calculated with bids and asks
value_relations_dict['oracle_calculated_price_difference']:	difference between the calculated and the oracle price
value_relations_dict['ask_bid_size_factor']:	factor how much difference is between bids and asks sizes
value_relations_dict['acquired']:	used to keep track of manual lock/release
value_relations_dict['lock']:	used for locking

---

Logic

getData/getData.py:

This class contains all functions which retrieve data and determine the relations between the retrieved data. Most functions are running within an infinite loop and a recursion is implemented in case of error, through a try-catch block. This is to ensure that the processes are always running.

value_relations():

def value_relations(self, market_data_dict, order_book_dict, value_relations_dict):
        while True:
            try:
                with value_relations_dict= Decimal(order_book_dict['best_ask_price'])
                    best_bid = Decimal(order_book_dict['best_bid_price'])
                    base_price = Decimal(market_data_dict['base_price'])

                    value_relations_dict['calculated_price'] = (best_ask + best_bid) / 2
                    value_relations_dict['oracle_calculated_price_difference'] = value_relations_dict['calculated_price'] / base_price
                    value_relations_dict['calculated_spread'] = (best_ask / best_bid) * 100 - 100

                if len(order_book_dict['asks_list']) >= 50:
                    value_relations_dict['total_size_asks'] = sum(
                        Decimal(x) for x in order_book_dict['asks\_list'][:50]
                    )
                    order_book_dict['asks_list'].pop(0)

                if len(order_book_dict['bids_list']) >= 50:
                    value_relations_dict['total_size_bids'] = sum(
                        Decimal(x) for x in order_book_dict['bids\_list'][:50]
                    )
                    order_book_dict['bids_list'].pop(0)

                total_asks = value_relations_dict.get('total_size_asks', 0)
                total_bids = value_relations_dict.get('total_size_bids', 0)

                if total_asks and total_bids:
                    value_relations_dict['ask_bid_size_factor'] = Decimal(total_bids) / Decimal(total_asks)

            except KeyboardInterrupt:
                sys.exit("Keyboard interrupt")
            except Exception as error:
                print(error)
                print("value relations failed, continuing...")

this function takes the folllowing three arguments:

1. market_data_dict
2. order_book_dict
3. value_relations_dict

and determines the relation between the fetched data, to store it in the value_relations_dict. Therefore market_data_dict and order_book_dict are accessed read-only and not written to. Both dictionaries should not be locked, because the contained data is simultaneously fetched and written in getData.py while being read by value_relations.py. Locking these dictionaries will therefore cause deadlocks.

This has one minor flaw: the data which is used to calculate the relations between the values can lag behind. This flaw an be neglected, because the relations express speculative tendencies instead of absolute states, since the fetched data is also speculative and will fluctuated based on the exchange used as a data source.

value_relations executes three tasks:

estimating the relations of fetched data:

In the first part of the function, we estimate the oracle_calculated_price_difference, calculated_spread and the calculated_price and propagate the corresponding variables in the dictionary, while it is locked, to avoid deadlocks.

the values are estimated as following:

value:	estimation
oracle_calculated_price_difference:	calculated_price / base_price
calculated_spread:	best_ask_price / best_bid_price * 100 - 100
calculated_price:	best_ask_price + best_bid_price / 2

estimating total size of asks and bids:

the task which is done, is simply to iterate through all content of our asks_list and bids_list at the size dimension and summing the values up. This is limited to the last 50 entries, since we only store 50 value pairs in the given lists.

estimating the relations between asks and bids:

finally we estimate the relation between the ask sizes and the bid sizes by dividing them. This will give us an overview if there are generally more aks or bid orders open in the orderbook, and how big this difference is.

update_best_bid_ask():

def update_best_bid_ask(self, order_book_dict, bids, asks):

    if bids:
        best_bid = max(bids, key=lambda x: [0])
        order_book_dict['best_bid_price'] = best_bid[0]
        order_book_dict['best_bid_size'] = best_bid[1]

    if asks:
        best_ask = min(asks, key=lambda x: x[0])
        order_book_dict['best_ask_price'] = best_ask[0]
        order_book_dict['best_ask_size'] = best_ask[1]

The function takes three arguments:

1. order_book_dict
2. bids
3. asks

It propagates order_book_dict with the highes bid and the lowest ask, together with their corresponding size. It is not necessary to lock the dictionary, because these variables are set nowhere else. Locking the dictionary at this point without checking if its locked already could cause deadlocks.

get_order_data():

async def get_order_data(self, order_book_dict):
        subscription = {
            "type": "subscribe",
            "channel": "v4_orderbook",
            "id": order_book_dict['market'],
        }

        try:
            websocket_order = create_connection('wss://indexer.v4testnet.dydx.exchange/v4/ws')
            websocket_order.send(json.dumps(subscription))

            while True:
                try:
                    with order_book_dict= websocket_order.recv()
                        data = json.loads(message)

                        if data.get('channel') != 'v4_orderbook':
                            continue

                        contents = data.get('contents', {})
                        bids = contents.get('bids', [])
                        asks = contents.get('asks', [])

                        # Process bids
                        if bids and isinstance(bids, list):
                            current_bid = bids[0]
                            if isinstance(current_bid, list) and len(current_bid) >= 2:
                                order_book_dict['current_bid_price'] = current_bid[0]
                                order_book_dict['current_bid_size'] = current_bid[1]
                                if Decimal(order_book_dict['current_bid_size']) >= 0.000001:
                                    order_book_dict['bids_list'].append(Decimal(order_book_dict['current_bid_size']))
                                    self.update_best_bid_ask(order_book_dict, bids, asks)

                        # Process asks
                        if asks and isinstance(asks, list):
                            current_ask = asks[0]
                            if isinstance(current_ask, list) and len(current_ask) >= 2:
                                order_book_dict['current_ask_price'] = current_ask[0]
                                order_book_dict['current_ask_size'] = current_ask[1]
                                if Decimal(order_book_dict['current_ask_size']) >= 0.000001:
                                    order_book_dict['asks_list'].append(Decimal(order_book_dict['current_ask_size']))
                                    self.update_best_bid_ask(order_book_dict, bids, asks)

                except KeyboardInterrupt:
                    websocket_order.close()
                    sys.exit("Keyboard interrupt")

        except WebSocketConnectionClosedException as e:
            print(e)
            print("order book socket connection failed, restarting...")
            await self.get_order_data(order_book_dict)

get_order_data takes only the order_book_dict as an argument. The function is designed to fetch the order book data from the exchange and propagate the dictionary with all relevant data, while cutting out the noise.

key considerations:

Its important to consider that the data is fetched as a stream, which is why the socket connection needs to be established before the infinite while loop. This is to ensure that there is only one socket connection open, which fetches the data stream. Its also important to lock the order_book_dict at this point, to avoid race-conditions.

The data is fetched in json format and the relevant fields are stored as a two dimensional array, which consists of the bid/ask value and the corresponding size of the order. Datasets can be either ask or bid data, every dataset represents one open order.

Unfortunately the fetched data seems to be corrupted sometimes, We can validate that by checking if the dataset contains 'price'.

All fetched orders are open at the time when they are fetched. Some orders contain the size '0', they are filtered out as they would interfere with the estimation of the value relations.

functionality:

the logic is simple and can be summarized as following:

If a bid or ask order is present, the values are stored in the the dictionary. If their size is bigger than 0, their value will be appended to the corresponding list and the best bid/ask price is updated.

limitations:

The fetched orders are open at the time when they are received, we do not know when they get filled, cancelled or when they do expire.

Orders which are immediately filled, do not appear in the orderbook.

get_market_data()

    async def get_market_data(self, market_data_dict):
        subscription = {
            "type": "subscribe",
            "channel": "v4_markets",
            "id": market_data_dict['market'],
        }

        try:
            websocket_market = create_connection('wss://indexer.v4testnet.dydx.exchange/v4/ws')
            websocket_market.send(json.dumps(subscription))

            for _ in range(2):  # Read 2 messages
                message = websocket_market.recv()
                data = json.loads(message)

                contents = data.get('contents')
                if not contents:
                    continue

                markets = contents.get('markets')
                if not markets:
                    continue

                market = market_data_dict['market']
                if market not in markets:
                    print(f"\nNO MATCH FOR MARKET: {market}\n")
                    continue

                oracle_price = markets[market].get('oraclePrice')
                if oracle_price is None:
                    continue

                old_price = market_data_dict.get('base_price')
                if old_price != oracle_price:
                    market_data_dict['old_price'] = old_price
                    market_data_dict['base_price'] = oracle_price

                    # avoid first initialization factor calculation
                    if old_price != Decimal('0.1'):
                        if Decimal(old_price) != 0:
                            change_factor = (Decimal(oracle_price) - Decimal(old_price)) / Decimal(old_price)
                            market_data_dict['change_factor'] = change_factor

            websocket_market.close()

        except WebSocketConnectionClosedException as e:
            print(e)
            print("market order socket connection failed, restarting...")
            await self.get_market_data(market_data_dict)

        except KeyboardInterrupt:
            websocket_market.close()
            sys.exit("Keyboard interrupt")

get_market_data takes only the market_data_dict as an argument. This function is built to fetch the market data from the exchange and propagate the dictionary with all relevant data, while cutting out the noise.

key_considerations:

unlike the order book data, the market data is not fetched as a stream. This is why we want to open and close the websocket within the infinite while-loop. It is important to lock the market data, to avoid race conditions.

The market data is fetched in json format. For the currently used trading style, the only relevant field within the market data is the oracle price.

Unfortunately the fetched data seems to be corrupted sometimes, We can validate that by checking if all necessary party are present in the json. The fetched data comes in two responses and its necessary to iterate through both.

functionality:

The logical functionality consists of three tasks:

1. store the oracle price in the dictionary
2. determine if the price has changed and
3. estimate the factor of the recent price change

limitations:

See definition of Oracle price in context of crypto-trading.

---

/algoTrade.py

this script is used to start the algoTraging program and contains following functions:

run_async_task():

def run_async_task(async_func, shared_dict):

    loop = asyncio.get_event_loop()
    loop.run_until_complete(async_func(shared_dict))

run_async_task takes the task and the corresponding dictionary as arguments. It is used to start asynchronous tasks within a process. This is to merge and asynchronous approach with multiprocessing.

algo_trade():

def algo_trade(market_data_dict, order_book_dict, value_relations_dict):

    get_data_instance = GetData()

    process_get_market_data = multiprocessing.Process(target=run_async_task, args=(get_data_instance.get_market_data, market_data_dict))
    process_get_order_data = multiprocessing.Process(target=run_async_task, args=(get_data_instance.get_order_data, order_book_dict))
    process_value_relations = multiprocessing.Process(target=get_data_instance.value_relations, args=(market_data_dict, order_book_dict, value_relations_dict))

    process_get_market_data.start()
    process_get_order_data.start()
    process_value_relations.start()

    while True:
        try:
            print(" ")
            print(" ")
            print("#####################")
            print("#    new Dataset    #")
            print("#####################")

            print("current ask price:", order_book_dict['current_ask_price'])
            print("current ask size", order_book_dict['current_ask_size'])

            print("current bid price:", order_book_dict['current_bid_price'])
            print("current bid size", order_book_dict['current_bid_size'])

            print("Best bid price:", order_book_dict['best_bid_price'], "Best bid size:", order_book_dict['best_bid_size'])
            print("Best ask price:", order_book_dict['best_ask_price'], "Best ask size:", order_book_dict['best_ask_size'])

            print("base price: ", market_data_dict['base_price'])
            print("last change factor: ", market_data_dict['change_factor'])

            print("calculated price: ", value_relations_dict['calculated_price'])
            print("calculated spread: ", value_relations_dict['calculated_spread'])

            print("sum of bids: ", value_relations_dict['total_size_bids'] )
            print("sum of asks: ", value_relations_dict['total_size_asks'] )

            print("difference between oracle and calculated price: ", value_relations_dict['oracle_calculated_price_difference'])
            print("difference between ask and bid sizes: ", value_relations_dict['ask_bid_size_factor'])

            time.sleep(2.5) # sleep 2.5 secs for better readability of output


        except KeyboardInterrupt:
            websocket_order.close()
            websocket_market.close()

            process_get_market_data.terminate()
            process_get_order_data.terminate()
            process_value_relations.terminate()

            process_get_market_data.join()
            process_get_order_data.join()
            process_value_relations.join()

            sys.exit("Keyboard interrupt")


        except Exception as error:
            websocket_order.close()
            websocket_market.close()

            process_get_market_data.terminate()
            process_get_order_data.terminate()
            process_value_relations.terminate()

            process_get_market_data.join()
            process_get_order_data.join()
            process_value_relations.join()

            print("error: ", error)
            print("continuing")
            algo_trade(market_data_dict, order_book_dict, value_relations_dict, order_management_dict)

Core function, this is used to start the programm. This is done by defining the processes and starting them, then afterwards running an output which prints the necessary data from the different dictionaries inside an infinite loop. The processes are also gracefully shut down in case of error and keyboard interrupt.

In case of error, a recursion is implemented after the graceful shutdown. This is to restart all processes and avoid zombies eating up all resources.

---

Usage:

run this script within the previously created virtual environment with the following command:

python3 algoTrade.py <MARKET_TICKER>

example:

python3 algoTrade.py BTC-USD

---

License:

This file is part of algoTrade.

algoTrade is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation.

algoTrade is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

Please see https://www.gnu.org/licenses/.

---

Addendum:

what is missing?

As previously mentioned this is not a complete trading-bot. This is because its never a good idea to give away working trading strategies by making them publicly available. So what would be needed to make it a complete trading-bot?

On the technical side there could be many approaches. Personally I have added following functionality:

monitoring market and orders

two monitoring systems:

one cosystem exists to monitor the given market conditions and screen them for certain patterns. As soon as the patterns are present, orders are opened.

the other monitors the open orders and adjusts or cancels them if necessary. This is getting signals from the market monitoring system

execution of orders:

there are 2 key functionalities when it comes to execution of orders, opening and cancelling. But to be trading succesfully an algorithm needs to be a little more versatile on how to open or close orders. Key questions have been for me:

• what type of order am I opening (Maker, Taker, Fill or Kill) in which situations?
• how long should the order stay open?
• what should the closing value be?
• in which cases does the closing value needs to be altered? (trailing stop)
• how should my stop loss strategy be?

All these functions need to be implemented in the execution part of the trading-bot.