Logo TU Ilmenau




Q-CE: Self-Organized Cognitive Engine based on Q-Learning

Dr.-Ing. Ali Haider Mahdi
Zeeshan Ansar
Dr.-Ing. Stephen Mwanje
Dr.-Ing. Oleksandr Artemenko
Prof. Dr.-Ing. habil. Andreas Mitschele-Thiel
Date of publication
One of the main challenges in Cognitive Radio Networks
(CRNs) is that the dynamic radio environment affects the
Quality of Service (QoS) requirements of Secondary Users (SUs).
Another challenge is how to predict Primary Users (PUs) activities
over licensed channels, to avoid interfering with PUs. So, there is
a need to implement a Cognitive Engine (CE) as a self-organized
entity in a CR that overcomes those challenges. In our previous
studies, an algorithm called Adaptive Discrete Particle Swarm
Optimization (ADPSO) combined with Case Based Reasoning
(CBR) has been proposed for CE. ADPSO selects the optimal
configurations when an unknown environment is countered while
CBR allows usage of previous knowledge in an environment
that has been previously observed. CBR however depends on
a single observation of the previous state and gives inaccurate
results where an individual states’ performance changes. Another
problem is how to find the best action when the environment is
changing dynamically. In this paper, we propose a self-organized
Q-Learning based CE (Q-CE) which: 1) autonomously adapts
the link configuration; 2) applies the previous action under
similar environments and 3) where the environment changes, QCE
learns from radio environment behavior and PU activities
the best action to apply, in order to achieve QoS requirements
and avoid interfering with PUs. The proposed CE combines
following methods: ADPSO for link configuration; CBR for fast
reasoning under similar environment; and Q-Learning to learn
the environmental behavior. The results show improvements of
about 67% in the achieved throughput, about 50% in signaling
overhead when compared with the previous solutions that use
only ADPSO and CBR